Images and documents: where machine reading fails

The Beginner level establishes that a multimodal model accepts photos and documents, and that figures need checking. Here we look precisely at where and why it fails, because that is what decides whether you can build a process on top of it.

Two very different ways of reading an image

Classic character recognition spots letter shapes and transcribes them. It understands nothing, it is highly reliable on clean text, and it tells you where it has doubts.

A vision model interprets the whole image. It understands that a table is a table and that a column is called "Amount". But it does not transcribe: it reconstructs what is plausible. On a half-erased digit, classic recognition will flag uncertainty; the vision model will propose a plausible figure, without flagging anything.

That is the difference that matters: one doubts, the other fills in.

The four documented pitfalls

Tables with merged cells. The reading order becomes ambiguous and values shift across a column.

Multi-column documents. It can read across, mixing two columns into one sentence.

Visually similar digits. A 3 and an 8, a 1 and a 7 in tight handwriting or a compressed photo.

Handwritten annotations. A manual correction on a printed document is often simply ignored, and that is usually the most important piece of information on the page.

The combination that works

Classic recognition for exact transcription, vision model for structure and meaning. Then you cross-check: if both give the same amount, you move on; if they diverge, a human looks at it. That is the architecture of serious document pipelines, and it costs little extra.

The rate to measure before you start

Do not ask "does it work". Take fifty representative documents, including the ugly ones, and count the errors. A 2% rate on invoices is one error in every fifty: acceptable with a review step, unacceptable in full automation.

That figure, no one can give you. It depends on your documents, not on the model.