---
title: "How Keirolabs reaches 78% on FinanceBench"
dek: "Why standard vector RAG collapses to ~19% on financial QA, and how treating a filing as structure — not chunks — gets to 78%."
category: "benchmarks"
tags: [financebench, rag, benchmarks, patronus]
author: "Keirolabs"
published: 2026-07-08T12:00:00+00:00
updated: 2026-07-08T12:00:00+00:00
url: https://keirolabs.cloud/blogs/benchmarks/keirolabs-78-financebench
---

# How Keirolabs reaches 78% on FinanceBench

> **TL;DR** — Keirolabs scores 78% on FinanceBench, Patronus AI's open-book benchmark of 150 expert questions over 361 SEC filings. Standard vector RAG collapses to ~19% here. The gap is not a better model — it is treating a filing as **structure**, not a bag of chunks.

## What is FinanceBench?

FinanceBench is an open-book evaluation by Patronus AI. The public split is 150 expert-annotated questions drawn from 361 SEC filings (10-Ks, 10-Qs, 8-Ks) across 40 US-listed companies and 9 sectors. Every question is open-book: the system is given the filing and must answer **and** cite where it found it.

## Why most retrieval systems fail

A single 10-K runs 100K+ tokens. The answer to one question often lives across a table on page 47, a footnote on page 92, and a policy note in the MD&A. Flat chunk-and-retrieve embeds the document as a bag of overlapping text windows — structured line items get split across chunks and lose their numeric meaning.

| Approach | FinanceBench | Why |
|---|---|---|
| GPT-4o + vector RAG | ~19% | Chunks lose table structure |
| GPT-4o oracle (full doc in context) | ~85% | Not deployable — doesn't scale |
| **Keirolabs** | **78%** | Navigate structure, extract tables, reason, cite |

The oracle is the ceiling, not a product: you cannot dump a 100K-token filing into context for every agent request.

## How Keirolabs gets to 78%

1. **Navigate, don't chunk.** The filing is a structured tree — sections, tables, notes. The query routes to the right region instead of embedding the whole doc flat.
2. **Structured table extraction.** Line items come out as typed values with units, not prose — numerics stay intact for the reasoning step.
3. **Reason, then commit.** A reasoning pass computes the answer, normalizes units, and commits. No refusal path: refusals are scored as wrong.
4. **Cite the evidence.** Every answer ships with the exact evidence string and source location, so the result is auditable end-to-end.

## Scoring rules (why "close" is wrong)

- **Information lookup** — semantic equivalence with the true answer.
- **Numerical reasoning** — parsed numerically, correct only within ±2.5% relative tolerance.
- **Logical inference** — answer plus the exact evidence location required.

Refusals and "not found" are penalized as incorrect — there is no partial credit for dodging.

## The takeaway

On financial documents, retrieval quality is bounded by how well you preserve document structure. 78% is what a deployable, structure-aware pipeline reaches today; the 85% oracle is a reminder of the ceiling, not a shipping system.


## FAQ

### What is FinanceBench?

FinanceBench is Patronus AI's open-book benchmark: 150 expert questions over 361 SEC filings, measuring how well retrieval and RAG systems answer financial questions and cite their sources.

### Is the 78% reproducible?

Yes. It runs on the fixed public 150-question split with deterministic programmatic scoring for the numerical and lookup categories. The testbench and run config are public.

### Can I run Keirolabs on my own filings?

Yes — the same retrieval and structured-extraction pipeline behind this score is available through the Keirolabs API. Point it at a filing or corpus and get citation-grounded answers with evidence locations.