Getting Line Items Out of a Thousand Supplier PDFs

The demo worked. Five invoices from your main supplier, clean PDFs generated by their accounting software, every line item pulled correctly into a spreadsheet in about four seconds. So you point it at the folder with the other nine hundred, and the folder contains: a photo of a delivery note taken at an angle on a warehouse floor, a scan where the top third is a fax header, a supplier whose line items run across a page break, and forty documents from a company that sends everything as one image inside a PDF wrapper.

This is the normal shape of the problem, and the reason it is worth understanding what your tool is doing rather than trying a different one.

What are the three jobs people call OCR?

Separating them is most of the battle, because choosing a tool for the wrong one is the usual mistake.

Recognition turns pixels into characters. It answers what letters are on this page. This is the oldest of the three and the most solved.

Layout understanding works out structure. Which characters belong to the same table cell, which pair of strings is a key and its value, where a row starts and ends. A page of correctly recognised characters with no layout is a bag of words.

Extraction to a schema maps that structure onto the fields you actually named. Which of the seven numbers on this page is the total. This is the only one that produces something your accounting system can consume, and it is where the handover matters, because the categorisation waiting on the other side guesses the most common label rather than the correct one.

Recent research puts the second and third jobs at the centre rather than treating them as post processing. The Spatial ModernBERT work on financial documents casts extraction as token classification across three simultaneous heads, one predicting a label such as PO number or item description, one predicting a column index, and one distinguishing item rows from header rows. Reading that design tells you what the hard part is: not the characters, but knowing that this string is a quantity, in the third column, on the second item row.

Which approach handles what?

Three families, and they fail in different places.

Those first four figures come from Google's own Document AI price list, which is unusually legible for this category. The spread is the interesting part: twenty times between recognising a page and extracting named fields from it. That gap is the price of the two jobs people assume are included.

The classical option is not a poor relation, it is a different trade. The Tesseract project's guidance on improving quality is candid about what it needs: at least 300 dpi, dark text on a light background for version 4 and later, deskewed pages, and a small border, because line segmentation degrades badly on a skewed page. Given clean input it is excellent and costs nothing per page. Given a photograph taken in a warehouse it is the wrong tool, and no amount of configuration fixes that.

Why is character accuracy the wrong number?

Because you do not consume characters, you consume documents. A page can be 99 percent character accurate and still be useless if the one percent landed in the invoice total.

Work the arithmetic once and the shape becomes obvious. Suppose your extractor gets each field right 98 percent of the time, which is a respectable number. Suppose you need twelve fields off each invoice: supplier, invoice number, date, due date, currency, subtotal, tax, total, and four line item columns. The chance that all twelve are right on the same document is 0.98 to the twelfth power, or about 78 percent.

So roughly one document in five has something wrong in it. Push per field accuracy to 99.5 percent and that falls to about six percent of documents. At 99.9 percent it is a bit over one percent. The relationship is unforgiving in exactly the region where vendors quote numbers, which is why a headline accuracy figure without a field count attached tells you nothing you can plan with.

Measure the thing you care about instead. Take fifty real documents from your own suppliers, type the correct answers by hand once, and count how many documents came out entirely correct. That number is your real rate, and it will be lower than any published one because your documents are worse than a benchmark set.

A pipeline that survives real documents

Six steps, and the two that get skipped are the first and the fifth.

Classify first. Before reading anything, decide what the document is. An invoice, a credit note, a delivery note, a statement, something else. This is skipped because everything in the folder is called an invoice, and it is the cheapest step to add. Extracting invoice fields from a credit note produces numbers with the wrong sign, which is the error type that survives every downstream check.

Recognise with the input fixed. Deskew, upscale anything under 300 dpi, and split multi page PDFs before the model sees them. Most accuracy complaints are input problems.

Locate, then extract. Keep these as separate outputs even if one API does both, because when a value is wrong you need to know whether the text was misread or the right text was assigned to the wrong field. Those have different fixes.

Validate with arithmetic, not confidence. This is the step that earns its keep. Line items must sum to the subtotal. Subtotal plus tax must equal the total. Quantity times unit price must equal line total. The invoice date must not be in the future. A document that fails any of these is wrong no matter what score came back with it, and a document that passes all of them is very likely right even if the model was unsure.

Route on the combination. Send a document to a person when it fails validation, or when a required field is missing, or when confidence is low on a field that arithmetic cannot check, such as the supplier name. Everything else goes straight through.

That last distinction matters more than the threshold you pick. Numeric fields are checkable by arithmetic, so their confidence scores are nearly redundant. Text fields are not, so their scores are the only signal you have. Treating both the same is why review queues fill up with documents that were fine.

The two documents that break every pipeline

Across a few dozen suppliers, the failures concentrate on two document types, and both are worth handling explicitly rather than hoping the model copes.

The first is the multi document PDF. A supplier emails one file that contains an invoice, a delivery note and a statement, because that is what their printer driver produced. Every step downstream assumes one document per file, so the extractor reads a total from page one and a date from page five and reports a confident, coherent, entirely fictional invoice. The fix is a splitting step ahead of classification, and the signal it looks for is usually a page whose header differs from the previous page's rather than anything clever.

The second is the credit note that looks like an invoice. Same layout, same supplier template, same field names, and every number should carry the opposite sign. No confidence score will flag this, no arithmetic check will fail, and the error lands directly in your ledger. The only defence is classification on a real signal: the document title, the presence of a negative total, or a supplier specific rule you write once after it happens the first time.

Both of these share a property worth naming. They are not accuracy failures. The model read the page correctly and the pipeline drew the wrong conclusion, which is why chasing a better extractor never fixes them.

Where the confidence score actually comes from

It is worth knowing what you are thresholding on, because the number means different things in the three jobs.

In recognition, a confidence score is roughly how sure the engine is about a character or a word, and it is well calibrated in the sense that low scores really do correlate with misreads. Blurry text scores low, and that is useful.

In extraction, the score is about the assignment: how sure the model is that this string is the field you asked for. A crisply printed number that the model put in the wrong field can carry a high score, because the reading was easy even though the decision was wrong. That is precisely the case the arithmetic check catches and the threshold does not.

So use recognition confidence to decide whether to rescan, and use validation to decide whether to trust. Using extraction confidence as your only gate produces a review queue full of legible documents and a ledger with confident mistakes in it.

What does a vision model change?

It removes the setup and moves the failure. A general vision model handed a page and a schema will return something shaped like your schema for any layout, including one it has never seen, with no training step at all. For a small business with forty suppliers each using a different template, that is genuinely the fastest route to something working.

What it does not remove is the need to check. A model that cannot read the tax line will not usually say so; it will produce a number that looks like a tax line. This is the same failure discussed in our piece on forcing a model to hold a fixed output shape, and the conclusion is identical: a schema guarantees the shape and says nothing about whether the contents are true. Validation is what closes that gap, which is why the arithmetic step above is not optional in this design.

Cost behaves differently too. Managed document APIs charge per page, so a thousand pages costs the same whether the pages are dense or sparse. A vision model charges by tokens, and a high resolution page of a dense table is a lot of tokens. Our comparison of what a real feature costs across the model APIs covers how to estimate that before committing. For the specific question of what an image contributes to the request, what vision actually adds to a language model is the shorter version.

Questions people ask

Can I skip OCR and use a vision model directly?

For born digital PDFs, you can often skip both and read the embedded text layer, which is free and exact. Check for it first, because a surprising share of supplier PDFs have perfectly good text sitting inside them and everyone runs image recognition over the top out of habit. For scans and photos you still need recognition, whether that is a separate step or something the vision model does internally.

How do I handle tables that run across a page break?

Stitch before you extract, not after. Concatenate the pages of one document into a single input where the tool supports it, and keep the header row from the first page in scope. Extracting each page separately and merging afterwards means the second page has no column headers, and the model will invent them.

What about handwriting?

Managed recognition APIs handle it far better than classical engines, and delivery notes signed at the door are the common case. Accept a lower per field rate on anything handwritten and route those documents to review by default rather than by threshold. The volume is usually small enough that this costs nothing.

How long does the first working version take?

A weekend for a general model with a schema and three validation rules, running over a folder and writing a spreadsheet. Longer for anything that writes into an accounting system, and the extra time is almost never the extraction. It is deciding what happens when a supplier is not in your list, what happens to a duplicate invoice number, and who fixes a document the pipeline rejected.

Is it worth training a custom extractor?

Only when you have volume from a stable set of layouts. Training pays for itself when the same twenty suppliers send the same templates every month, because a trained extractor on a known layout beats everything else and the per page price stops mattering next to the review time it saves. For a long tail of one off suppliers it does not, and the general model with validation is the better fit.

What this costs in practice

Work it from review minutes rather than API prices, because at small volumes the API is the cheap part. A thousand invoices a month at Google's structured extraction price is thirty dollars. A thousand invoices a month with one in five going to review, at two minutes each, is nearly seven hours of somebody's time.

That comparison decides everything. Moving per document accuracy from 78 percent to 94 percent is worth far more than moving from a thirty dollar tool to a ten dollar one, and the thing that moves it is usually the validation rules and the input quality rather than the model. Fix the scanner settings, add the arithmetic checks, classify the document type first, and the same model gets you a different result.

If you are wiring this into a shop's own systems rather than a spreadsheet, the connection layer matters as much as the extractor. Our MCP server page covers how a tool gets structured access to your catalogue and orders, which is where an extracted invoice has to land to be worth extracting.