I ended the last article (Linked: Part 2) on a single root cause, structured, quantitative data forced through a pipe built for unstructured language, producing the linearization defect during extraction and the tokenization disconnect during embedding. The natural conclusion is that you need a better extractor.

That conclusion is half right, and the missing half is what this article is about. Flattening is the default at every step along the way, because every component in a retrieval-augmented generation pipeline expects the one before it to hand over plain running text, with nothing marking which part was a header and which was a figure. The parser turns a two-dimensional page into a one-dimensional line, the embedding model reduces a whole table to one location in a vector space, and the retriever treats every question, conceptual or arithmetic, as the same nearest neighbour search. That is the flattening reflex, and fixing it at one step buys you nothing if the next component does it again.

Someone has measured this. FinanceBench, published by Patronus AI, Contextual AI and Stanford in November 2023, asked deliberately straightforward questions about SEC filings, the kind an analyst answers without breaking stride. Start with its cleanest result, the setting its authors called Oracle. Hand the model the exact pages holding the answer, no retrieval to get wrong and no format to mangle, and GPT-4-Turbo still got 15% of them wrong, the best any configuration in the study's main comparison managed.

Now the realistic setting. One shared vector store over PDFs of all 360 documents, which is what an enterprise actually deploys, and the same model answered 81% of the questions wrong or refused them outright, leaving 19% correct. Give it one store per document instead, a separate index for each filing rather than one index holding all of them, and it answers half of them correctly, though that setting assumes you already know which filing to open, which in a real corpus is a good part of the question. Same model throughout, so what moved was the shape of the search. Hold onto the word PDFs, because the next section argues those were the wrong starting file, and that choice is part of what the 81% is measuring.

Models have moved on since 2023, and so has the stack around them. That 19% came from OpenAI's ada-002 embeddings inside a 2023 pipeline, and Anderson and colleagues, in a paper on embeddings trained for finance, rebuilt the same shared-store setting the following year, reaching 47% correct and then 55% by swapping in embeddings trained on financial text.

Those gains came from a rewritten query, a newer model, an embedder trained on the right domain, and one cheaper step worth naming on its own, prepending each passage with the filename of the document it came from, which carries the company name and the filing date. The last eight points came from the embedder alone. After all of it, the setting an enterprise actually deploys still returns close to half its answers wrong or refused. Benchmarks built since have not been kinder. FinanceQA, published by Mateega and colleagues in January 2025, set current models against the analyses a junior analyst does on the job and found they failed about sixty percent of them, mostly on accounting convention and multi-step reasoning rather than on finding the page. The failure it captures is architectural, and a hard-tuned pipeline moves the score without changing what the score is measuring.

Every number above measures the problem and not the fix. I am not aware of a published ablation running these three against that benchmark, and I have not run one either, so what follows is an argument about where structure gets lost rather than a scored result.

Preserve the Grid Before You Chunk It: Layout-Aware Parsing

A PDF extractor pulls the text out of the file, and text is text. What sits inside the file is something else entirely. A PDF is a stage director, telling the renderer to place this string at these coordinates and that string forty millimetres to the right, with no instruction anywhere about what belongs to what.

A naive extractor sweeps those strings up in whatever order it finds them and hands you one flat line. The grid you read as a table was never in the file. It was an illusion produced by position, and position is the first thing the extractor discards.

One check before any of this. If the document is an SEC filing, start somewhere better than a PDF. The copy on EDGAR, the SEC's public filing database, is HTML, so its tables are marked up as tables, rows and cells a parser can walk directly. Inline XBRL tagging is mandatory on top of that, which means the financial statements and their footnotes arrive already mapped to a standard taxonomy, tagged down to the individual amount.

Fetching a PDF of that same filing and running a layout model over it is choosing the harder problem on purpose. The benchmark from the opening did exactly that, and researchers building on it have gone as far as converting EDGAR HTML into PDF to stay comparable with it, which tells you how settled the wrong default has become. The jurisdiction changes and the lesson does not, since the European single electronic format carries the same tagging and India files XBRL with the Ministry of Corporate Affairs. Find the regulator's structured copy before you parse a print of it.

What follows is for everything else, the investor deck, the credit agreement, the scanned document, the management discussion and analysis that the tagging does not reach, the filing that arrived as an email attachment. That tail is most of what sits in a bank's own systems, where clean tagged filings are the exception and not the corpus.

Read the coordinates instead of guessing at reading order. Two kinds of tools do this, and they are worth keeping apart, because people blur them together. Layout-aware models like LayoutLM are trained on text together with its position on the page, so a number arrives with its address attached, fourth column, third row, under a header. That is a base architecture, and it needs its own fine-tuning data, meaning labeled examples someone has to produce, before it does anything useful for you.

Dedicated parsers like LlamaParse and Amazon Textract are closer to turnkey, no fine-tuning data and one API call. Whether they survive a 10-K, a company's annual filing and a few hundred pages of it, is a separate question. Point one at a PDF and it does the practical work, detecting a table, lifting the grid out of coordinate space, and handing back either Markdown directly or structured output you convert into it, all before a single chunk is cut. In those formats, a table is still a table. Headers stay welded to their columns, and 14,200,000 remains tied to the Total Liabilities row instead of drifting into a wall of prose.

The scanned document is the harder version of this. There is no text layer to pull, so the parser runs optical character recognition first, and everything downstream inherits whatever it read. That step does hand you something the rest of the extraction path never offers, since an OCR engine reports how confident it was on each block. Keep those scores and route the low ones to a person. Discard them, and a misread digit enters the index looking exactly like a good one.

None of that is free. A layout parser bills by the page, though the bill lands once at ingestion rather than on every query, which is the same shape as the embedding pass you are already paying for. Where a tagged HTML copy exists, you skip the charge entirely, which is the second reason to check EDGAR before reaching for a PDF.

Here is the trap, and it catches the people who think the work ends at conversion. Chunks have size limits, so a fifty-row table has to be split before it is stored.

Now watch the splitter work. It cuts cleanly at the size boundary. The second fragment arrives with no header row, and the bottom half of your carefully preserved table is naked figures with no idea what they measure. You have rebuilt the linearization defect inside the system built to remove it.

The remedy is small and easy to forget, repeat the header row at the top of every chunk, or rewrite each row as an explicit label-value pair, so every number carries its own identity no matter where it lands. That remedy assumes LlamaParse or Textract got the header row right to begin with. Let one misread a multi-line header as a row of data, and repeating it stamps that mistake onto every chunk, formatted exactly like a header that got it right. Wrong and confident is worse than missing.

Even when the parser reads the header correctly, real tables still complicate the remedy without breaking it, a units line stated once above the table, a footnote asterisk two rows down, a header that spans three merged cells. Carry all of that over, or the number survives and its fine print does not. Nesting is the one that catches people, since repeating the top row of a statement tells a chunk it is looking at Assets without telling it the row sits under Current Assets, and the intermediate level is where the meaning lives.

A table that spans a page break is a different failure than a bad chunk boundary, and it happens earlier. Nothing in a PDF says that the table continuing on page twelve is the same table that started on page eleven. A parser that does not stitch the two together reads page twelve as an orphaned block of numbers with no header of its own, or welds it onto whichever table sits nearest on that page instead.

One approach sidesteps parsing altogether and deserves its own article. ColPali and the vision-based retrievers that followed it encode the page as an image and match against it directly, no parsing step at all, which removes every extraction failure above by never extracting. It solves retrieval. It does not solve the tokenization disconnect, because whatever page it surfaces still arrives at a model that reads the numbers on it as tokens. It also moves the risk downstream rather than removing it, since a vision model reading a dense, low-contrast table is doing harder work than a text model reading the same table cleanly parsed.

Two remedies now point at the same table, so choose between them by density rather than size. A long sparse table, a schedule of properties or a list of subsidiaries, splits fine with the header carried across. A number-dense one earns the multi-vector treatment that comes next, stored whole and indexed by its description, where the header never has to survive a split it never goes through.

Search the Description, Reason on the Grid: Multi-Vector Retrieval

Preserving the grid solves structure. It does nothing for the tokenization disconnect, because a perfectly formatted table full of numbers still embeds badly. Neat arrangement on the page does not change what happens inside the embedding model, where the digits are chopped into tokens that carry no sense of magnitude and land in positions with nothing to do with size. Do not search the table at all.

That is the whole idea behind multi-vector retrieval, separate what you search on from what you send. The plural is the point, since one table can carry more than one key, a summary of what it shows or the questions it answers, each a different way into the same payload, and the summary is the one to build first. The name is overloaded, since ColBERT-style late interaction stores a passage as one vector per token rather than one for the whole thing and gets called multi-vector too, which is why this one is sometimes called parent-document retrieval instead. None of it is new, it has shipped as a named retriever in the LangChain and LlamaIndex documentation since 2023.

A raw table embeds poorly, while a plain language summary of it embeds well. So you have a model read the table and write a short description of it like "Q3 regulatory capital disclosure showing a Tier 1 capital ratio of 12.4% against total risk-weighted assets", and you embed the description as the searchable key.

The Tier 1 ratio is a bank's core capital divided by its risk-weighted assets, which is a formal way of asking how much loss this institution can absorb before it is in serious trouble. The threshold the compliance question turns on is 8.5%, the Basel III baseline once the capital conservation buffer is counted and before national regulators build on it, and a bank that slips under it faces restrictions on dividends and buybacks.

When a query matches the description, the pipeline does not hand the model the description. It reaches back and hands over the original table, intact. You search on prose, which is what embeddings are built for. You reason over the grid, which is what a correct answer requires. The disconnect never fires, because the numbers were never asked to be searchable.

Do the cheap thing first, though. If the parser recorded which period and which entity a table belongs to, filter on those before the vector search runs at all, and a Q2 table stops being a candidate for a Q3 question. That is a hard constraint rather than a similarity score, it costs nothing at query time, and it is the same principle as everything else in this piece, use the structure you already extracted instead of asking an embedding to infer it. Multi-vector retrieval earns its keep on what a filter cannot express, since which quarter is one question and which of the forty tables inside that quarter answers you is another.

Here is where the harder version lives, and it is the retrieval-side face of what the last article called the tokenization disconnect. Digits that carry no magnitude also carry little to tell one grid from another. Filter down to Q3 and the compliance agent asking whether the Tier 1 ratio fell below threshold still faces every table that quarter filed, a capital disclosure, a segment breakdown, a maturity schedule. To an embedding model reading raw grids, those are fields of digits punctuated by short labels, and the labels that would separate them, Tier 1 capital against three to five years, are diluted by everything around them rather than absent. So the retriever returns whichever grid landed nearest, the model reasons carefully over the wrong table, and the answer comes back as a real number from a real filing for the right quarter, which is exactly the kind of wrong that survives review.

The description is what breaks that tie. A wall of similar numbers becomes one short sentence where regulatory capital and Tier 1 carry most of the weight, which gives the query something to match on that the grid itself never offered.

The description is doing real work now, which means a bad description does real damage. A summary compresses by choosing what to leave out, and a fifty-line balance sheet reduced to one sentence about the Tier 1 ratio will not mention amortization of intangible assets unless someone decided that metric belonged in the sentence.

Ask about it, and the query has nothing to match against. The number is still sitting in the table. The table just never gets called. The cheap insurance is to index the row labels alongside the description and search them lexically, BM25 or anything else that matches on the words themselves, so a query naming a line item the summary skipped still has a literal string to land on. Pairing that description with a lexical key is hybrid retrieval in the ordinary sense of that word, the description catching what a question means and the row label catching what it names, and how heavily to weigh one against the other is its own tuning problem that no default setting solves for you.

One honest note on cost, having a model generate a summary for every page of every document is slow and expensive, and you would not run it blindly across tens of millions of files. You reserve the treatment for the number-dense tables where it earns its keep, and you handle ordinary prose the ordinary way. Finding those tables does not need another model, since the parser already knows which blocks it read as tables, and the ratio of digits to characters sorts the dense ones from the rest for the price of a regex.

Why not just paste the whole document into the context window?

It is the right question, and it became the obvious one the moment context windows crossed a million tokens. If the model can hold an entire filing, retrieval starts to look like an artifact of a constraint that no longer exists.

Three things break it. Attention dilutes across long contexts on most models, so a figure buried at the sixty percent mark of a two-hundred-page filing competes with everything around it and is silently missed. Liu and colleagues measured the shape of this in 2023, in a paper called Lost in the Middle, and found a curve like a U, accuracy highest when the answer sits at the very start or the very end of the context and worst when it sits in the middle. The newest long-context models have closed much of that gap for a single buried fact. Ask one to weigh that fact against three others scattered through the same filing, and the gap reopens.

The same FinanceBench run cuts the other way here. Long context beat retrieval badly in that study. Whole filings pasted into the window, no vector store at all, and GPT-4-Turbo answered 79% correctly against the shared store's 19%, which is the 81% from the opening stated the other way round.

Two things sit under that number. It fails 21% of the time on questions the authors built to be a minimum standard, straightforward enough that an analyst would not slow down for them, and one in five is not a rate compliance can carry. And the filings kept outgrowing the window, so the longest ones were cut down to their first 95,000 to 100,000 tokens.

The reason they gave for cutting there is the interesting part, nearly all the questions relate to the earlier sections of a filing, which means the answer usually sat near the front of the context, exactly where the U-shaped curve says models do best. That 79% was scored with the middle-of-the-document problem largely absent from the test. The authors' verdict was still that the approach does not belong in production, too slow, too expensive, and unable to hold documents of the size analysts work with.

Cost is the second, and it inverts too, retrieval charges you once at ingestion and pays a small toll per query afterward, a structure that holds no matter when the next question arrives. Full context can get close to that with prompt caching, but only while the cache stays warm, and a filing nobody has asked about in a few minutes goes back to full price on the next question.

The deciding problem is the third one, a larger window does not touch the tokenization disconnect at all. Whether that figure arrives inside a two-thousand-token chunk or a two-million-token one, the model still reads 14,200,000 as a string of tokens with no reliable ordinal relationship to 14,100,000. How a tokenizer groups digits does change how well a model computes on them, and none of the embedding models you would actually deploy hand the retrieval layer a sense of size.

Notice where that leaves you. To make full context work on a balance sheet you would still have to preserve the table structure and still have to route the arithmetic elsewhere, which is the same two fixes with a much larger bill attached.

Send Math to Something That Can Count: Hybrid Routing

The first two fixes get the correct table in front of the model, which is necessary and is not sufficient. The Oracle condition from the opening is the closest the study comes to testing that, the evidence pages handed over outright with retrieval taken out of the problem, and it still left 15% of answers wrong, every one of them a wrong answer rather than a refusal.

Be careful what that 15% is asked to prove. Oracle removes retrieval and leaves extraction where it was, since those evidence pages come out of the same PDFs the last section argued against, so an unknown share of that 15% is the first fix failing rather than the model reasoning badly. Read it as a ceiling on what survives clean retrieval rather than a measurement of reasoning. The reflex reaches into the benchmark too.

What is left is a failure living outside retrieval, split between extraction and what the model does with information it already has, and the study separates neither. The authors read their own results as reasoning weakness. Arithmetic is the part you can hand off cleanly, and it is the one thing you should never ask a vector search or a language model to do. Vectors encode direction, direction carries meaning, and the cosine similarity you search with divides out whatever length they had. "Calculate the quarter-over-quarter change in liabilities" is a math question wearing the clothes of a search query.

That is not a corner case. Across the portion of the corpus they labeled by type, 66% of questions turn on numerical reasoning against 28% that are pure extraction, with the small remainder logical reasoning. Most of those were built around financial metrics, because computing metrics and reasoning about them is a core part of what an analyst does all day, which is why a pipeline that treats math as the exception has the ratio backwards.

So you put a router at the front of the pipeline whose only job is to read the incoming question and decide which machine it belongs to. A conceptual question, such as what your policy on distressed assets actually says, goes to the vector database, where semantic proximity is the right instrument and does the job well. Numbers work differently, the liquidity shortfall between Q2 and Q3 needs a structured database instead, where the extracted figures sit in typed columns and arithmetic is exact and checkable. One path is probabilistic and tolerant of ambiguity. The other is exact and refuses to improvise.

The arithmetic is exact once the query is written, and writing the query is still a model reading a schema and picking columns, which fails the way everything else here fails, a plausible query against the wrong column returning a number that looks entirely reasonable. The deterministic half sits downstream of a probabilistic one, so the review belongs on the generated query rather than on the answer it produced.

"Typed columns" is doing quiet work in that sentence. A number does not become a decimal in a schema just because a parser found it. Someone still has to design the schema, decide whether a dash means zero or not reported this period, which are different facts and produce different quarter-over-quarter answers, and catch it when one company's filing calls a line item Total Liabilities and another calls the same thing something close but not quite the same. That is the same extraction problem this piece opened with, paid once at ingestion instead of paid badly on every query.

Some of that bill is pre-paid where the tagging from the first section reaches. Filers can mint their own tags when nothing in the taxonomy fits, and they do it at a rate worth taking seriously, roughly one financial item in five in a 10-K. The SEC's Division of Economic and Risk Analysis publishes that rate by filer category each year, and its most recent analysis, running through fiscal 2024, has it falling everywhere but the largest filers, where it has held flat.

A company's bespoke label for a restructuring charge joins up with nothing on its own, though the filing often says where it belongs, since the calculation relationships in the XBRL exhibit record which standard totals that custom element rolls into. Follow those and the mapping comes from the filer rather than from a guess.

Check the arithmetic before you trust the link. A calculation relationship is an assertion that a set of children sums to a parent, and every value it names is already sitting in the same instance document, so you recompute the roll-up and reject any relationship that does not reconcile. Rejecting the link leaves the figure where it was and withdraws only the filer's claim about where it belongs. That is deterministic, and it needs no model and no reviewer. What it does not catch is the custom element that sits in no calculation relationship at all, which arrives with nothing to check it against and no rule that will place it.

One column the schema needs that nobody remembers to add, the date a figure was reported alongside the period it describes. Restate a prior period's liabilities in a later filing and that figure has two true values, what the company said then and what it says now. Store only the second and the first is gone, and for an audit that gap is the whole question. Two dates on every row where most schemas carry one is the bitemporal model, one date for when the figure was true and one for when the company said it, and finance was building point-in-time systems on it long before anyone asked a language model about a filing. Storing both is the easy half, since most questions will name neither date, so the pipeline answers from the most recent filing by default and puts that reporting date in the answer, where whoever asked can see which of the two figures they got.

A numerical question does not have to go to a structured database. Handing the retrieved table to a model with a code execution tool works too, which lets it write and run the arithmetic in Python instead of guessing at it token by token.

Two limits on that. Reading a number off a page and doing arithmetic on it are different tasks, and code execution only makes the second one reliable. The model still has to get 14,200,000 off the page correctly before any script can run on it. And a script written fresh each time suits ad-hoc arithmetic. A metric with a prescribed definition belongs elsewhere, since a regulatory capital ratio carries specific deductions that vary by jurisdiction and needs vetted code that runs the same way every quarter.

That split is clean when the question is. It is less clean for the question this piece keeps coming back to, whether the Tier 1 ratio fell below threshold in Q3, which is really two questions inside one sentence, a compliance question layered over a numerical one. Break it in two, what is the ratio and then is it below threshold, or hand the model both tools and let it fire each itself, a pattern usually called agentic RAG. The second costs a round trip before the answer even begins, so save it for questions that are actually layered and let a plain classifier take the rest.

Get that split wrong and the router is no better than a coin flip, and it is the one component here that fails the way this whole piece is about. A misrouted arithmetic question comes back from the vector store fluent and confident, with nothing in the answer indicating that a routing decision was made at all. So instrument it, log which path fired and on what basis, and let ambiguous questions run down both, so a disagreement surfaces instead of resolving itself in silence.

Three Fixes, One Refusal: Where the Work Actually Lives

None of these moves is exotic, and none of them requires a better model. Layout-aware parsing refuses to flatten the page, multi-vector retrieval keeps what a table means separate from what it contains instead of flattening the two together, and hybrid routing will not let every question collapse into the same kind of lookup. They are three refusals of one reflex, taken at three different seams, and each of them amounts to the same decision, keep the structure the default pipeline was built to discard.

Three systems instead of one is a real cost, a parsing layer, a structured database with its own schema to maintain, and a router deciding between them. A single vector database asked to do all three jobs does not fail three times in three visible places. It fails once, quietly, and hands back an answer that sounds as confident as a correct one. What the three buy is failure that stays legible, provided you instrument the router, which is the one seam that will not announce itself.

The heavier cost is judgment. This piece has quietly handed a person four decisions no fix here makes for them, what a blank dash in a filer's table actually means, which unmapped tag belongs to which concept, whether a restated figure answers the question that was actually asked, and which questions are layered enough to need both paths at once. Every fix gives that judgment somewhere to live. None of them supply it.

That is where the effort belongs. When RAG fails in a serious domain, the instinct is to reach for a smarter model, and the smarter model fails in exactly the same place, more fluently and more convincingly.

Nothing downstream can recover what ingestion threw away.