Legacy OCR Texts Make Language Models Learn at Only 30 Percent Efficiency – the FineBooks Project Aims to Change That

The FineBooks project by Hugging Face and EleutherAI tested 14 open-source OCR models on 2,165 historical book pages, publishing results as a leaderboard. The motivation is that OCR texts from libraries are often flawed, and a language model trained on such texts learns at only 30% efficiency compared to human transcriptions, as quantified by the Talkie project. The leading model, dots.mocr, has 3 billion parameters and reaches 97.6% accuracy at $1.94 per thousand pages, while OvisOCR2, with only 0.9 billion parameters, ranks second at $0.46 per thousand pages, showing that model size and OCR quality hardly correlate for historical documents. The evaluation covers only Antiqua scripts in English, French, German, and Latin, and is limited to book-like documents with single-column flowing text. As a next step, the team plans to reprocess about 200,000 public-domain BHL documents with one of the leading models and publish the text as an open dataset. While the models are good enough for AI training, they replace archaic characters like long s with modern equivalents, which is a problem for scholarly transcription; targeted fine-tuning could fix this according to the authors.