Keirolabs scores SOTA 78% on FinanceBench.
An open-book benchmark where models answer real financial questions over real SEC filings. Here's what FinanceBench is, why it breaks most retrieval systems, and how Keirolabs reaches 78%.
SOTA among commercial web-search & retrieval APIs — ahead of Valyu (73%), Parallel (67%), Exa (63%), and Google (55%) on the public 150-question split.
A financial QA benchmark built from real SEC filings.
FinanceBench is Patronus AI's open-book evaluation for LLM and RAG systems on finance. Instead of trivia, it asks the kind of questions an analyst asks — and checks the answer against the filing.
The corpus is 361 public SEC filings and financial reports spanning 40 US-listed companies across 9 GICS sectors, covering data from 2015 to 2023. Roughly 75% of the documents are 10-K annual reports; the rest are 10-Q quarterlies, 8-K material events, and earnings releases with call transcripts.
The public evaluation set is 150 expert-annotated question–answer–evidence triplets, drawn from a full dataset of 10,231. Every question is open-book: the system is given the filing and must produce the answer and point to where it found it.
The things that make finance easy for a human break retrieval for a model.
FinanceBench isn't hard because the questions are obscure. It's hard because the answers live in long, structured documents that flat retrieval can't navigate.
Filings are dense & scattered
A single 10-K runs 100K+ tokens. The answer to one question lives across a table on page 47, a footnote on page 92, and a policy note in the MD&A. Flat chunking loses that structure.
Numerical reasoning, ±2.5%
Half the questions demand math — operating margins, ROA, YoY deltas. The score is correct only inside a strict ±2.5% relative tolerance. Close is wrong.
Refusals are penalized
A "not found" or a hedge is scored as incorrect — there is no partial credit for dodging. The system must commit to an answer grounded in the document.
Vector RAG collapses here
Standard chunk-and-retrieve can't link structured line items across table pages. The published baseline for GPT-4o + vector RAG on FinanceBench is ~19%.
Navigate the document like an analyst, not like a search index.
The 78% comes from treating the filing as structure, not as a bag of chunks — and from refusing to refuse.
Navigate, don't chunk
Keirolabs treats the filing as a structured tree — sections, tables, notes — and routes the query to the right region instead of embedding the whole doc flat.
Structured table extraction
Line items are pulled as typed values with units, not prose. That keeps the numerics intact for the reasoning step instead of losing them in a text blob.
Reason, then commit
A reasoning pass computes the answer from the extracted figures, normalizes units, and commits — no refusal path, because refusal is scored as wrong.
Cite the evidence
Every answer ships with the exact evidence string and source location from the filing, so the result is auditable end-to-end.
Three gauges, each with a strict scoring rule.
FinanceBench splits questions into three categories. Each is scored differently — and each punishes a different failure mode.
| Gauge | What it asks | How it's scored |
|---|---|---|
| Information lookup | Text & table extraction (e.g. "Did the company pay dividends?") | Semantic equivalence — the answer text must match the true answer without fabricating facts. |
| Numerical reasoning | Math over financials (operating margin, ROA, deltas) | Numeric parse, units normalized ($B / M / %). Correct only within ±2.5% relative error. |
| Logical inference | Analyst-style synthesis across the document | Final answer + exact evidence string/page from the source filing required. |
The benchmark, in plain questions.
What is FinanceBench?
FinanceBench is an open-book benchmark by Patronus AI that measures how well LLM and RAG systems answer financial questions against real SEC filings. The public evaluation set is 150 expert-annotated questions drawn from a corpus of 361 filings across 40 US-listed companies and 9 sectors. The full dataset contains 10,231 question–answer–evidence triplets.
What does "SOTA 78%" mean here?
Keirolabs scores 78% accuracy on the public FinanceBench split — the highest among the commercial web-search and retrieval APIs we evaluated: Valyu at 73%, Parallel at 67%, Exa at 63%, and Google at 55%. For academic context, a raw GPT-4o with an entire filing dumped into its context window reaches ~85%, but that oracle setup doesn't scale to a real agent, and flat vector-RAG setups collapse to ~19%.
How is FinanceBench scored?
Each answer is marked correct or incorrect under binary scoring. Information-lookup questions use semantic equivalence; numerical-reasoning questions are parsed numerically and must land within a strict ±2.5% relative tolerance; logical-inference questions require the answer plus the exact evidence location in the filing. Refusals and "not found" responses are penalized as incorrect.
Why do vector RAG systems score so low?
Financial answers often live in tables and footnotes spread across a 100K-token filing. Chunk-and-retrieve embeds the document flat, so structured line items get split across chunks and lose their numeric meaning. The published GPT-4o + vector RAG baseline on FinanceBench is ~19% — barely above chance for the harder question types.
Is the 78% reproducible?
Yes. The evaluation runs on the fixed public 150-question split with deterministic programmatic scoring for the numerical and lookup categories. Our benchmarks repo and the run config are public so the number can be reproduced independently.
Can I run Keirolabs on my own financial documents?
Yes — the same retrieval and structured-extraction pipeline behind this score is available through Keirolabs APIs. Point it at a filing or a corpus and get citation-grounded answers with evidence locations. Start free, no credit card required.
Run Keirolabs on your own filings.
The same retrieval and structured-extraction pipeline behind this score is available through the Keirolabs API — citation-grounded answers, evidence locations included.