FineBooks Benchmarks OCR Models for Better LLM Training

Training AI models on public-domain literature is notoriously difficult because older optical character recognition (OCR) systems produce text riddled with errors. According to the Talkie project, training an LLM on poorly digitized OCR text is only 30 percent as efficient as training on clean, human-transcribed text. To solve this, Hugging Face and EleutherAI introduced FineBooks, a project that evaluated 14 open-weight OCR models on 2,165 historical pages from the Biodiversity Heritage Library (BHL).

The benchmark revealed that smaller, specialized models often outperform larger ones. The top-performing model, dots.mocr, uses only 3 billion parameters, while the runner-up, OvisOCR2, has just 0.9 billion parameters and costs a mere 46 cents per thousand pages. Both outperformed the larger Qwen3.5-9B model. The leading models achieved character accuracy rates exceeding 97 percent, with processing costs staying under two dollars per thousand pages. The evaluation measured Character Error Rate (CER) across two categories: a "diplomatic" variant that penalizes the modernization of archaic characters like the long "s", and a "reading" variant that permits such updates.

For AI practitioners, these findings mean that reprocessing massive public-domain archives is now highly cost-effective. The FineBooks team plans to use a top-performing model to reprocess approximately 200,000 public-domain documents from the BHL, which holds over 64 million pages. However, the current evaluation is limited to single-column pages in English, French, German, and Latin using Antiqua typefaces. It does not yet support Fraktur, handwriting, or non-Latin scripts.

While the output is clean enough for AI training, the researchers caution that these models are not yet suitable for academic or library use. The models output plain text or Markdown rather than the ALTO XML format required by library databases, and they tend to silently modernize historical ligatures. Nonetheless, the open-source benchmark will continue to add new models on a rolling basis.