---
title: "Search API Benchmarks That Actually Matter: FinanceBench, SimpleQA, BrowseComp"
dek: "FinanceBench, SimpleQA, and BrowseComp test what agents actually do: retrieve, cite, and answer. Keirolabs scores 78% on FinanceBench and 84% on SimpleQA."
category: "guide"
tags: [benchmarks, financebench, simpleqa, browsecomp, factuality]
author: "Dave"
published: 2026-08-17T12:00:00+00:00
updated: 2026-08-17T12:00:00+00:00
url: https://keirolabs.cloud/blogs/guide/search-api-benchmarks-that-matter-2026-financebench-simpleqa
---
> **TL;DR** : FinanceBench, SimpleQA, and BrowseComp are the benchmarks that matter for search APIs because they test retrieval plus citation, and on them Keirolabs scores 78% and 84%, which makes cost per correct answer the metric that decides value.
## Why is the search API the most underrated piece of AI infrastructure?
**The search API is the most important piece of AI infrastructure almost nobody understands.** Every agent that answers a question, checks a fact, or builds a report depends on retrieval, and the gap between what vendors claim and what they charge is the biggest story in agentic AI. Everyone is comparing the wrong numbers.
I have watched this market for two years. I benchmark these APIs from the bills, not the landing pages, and the gap between what vendors claim and what they charge is the biggest story in agentic AI. It is hiding in plain sight. The headline price per 1,000 queries is a floor, not a bill. The real cost is the loop: the extraction call, the token fees, the rate-limit upgrade, the cache you build yourself. And in 2026, that loop just got cheaper for the people who build it themselves.
This is the moment the search API market split in two. On one side sit the managed research products that bundle search, reasoning, and report writing into a single expensive call. On the other side sit the retrieval APIs that hand an agent cited results for pennies and let the agent's own model do the thinking. The two sides are not competing on the same benchmark, and that confusion is what this post exists to clear up.
The benchmarks that separate the two sides are FinanceBench, SimpleQA, and BrowseComp. They are not trivia leaderboards. They test the three jobs a search API actually does: retrieve a cited answer from a long document, retrieve a single fact, and retrieve across a multi-step research loop. A vendor that publishes all three numbers, with run configs, is telling you how its API behaves across the full range of agent workloads. A vendor that publishes one cherry-picked number is telling you less.
Two years ago this category did not exist. You had scraping APIs that returned Google results and nothing else, and you built the retrieval layer yourself. Today there are dozens of vendors, and most of them are selling the same thing with different landing pages. I have run the same 50-question test against most of them, and the spread in answer quality at the same price point is wider than any other layer of the stack. That spread is why benchmarks matter. Without them, you are choosing between landing pages, and landing pages are written by marketers.
In 12 months this will be obvious. Every agent that ships without a retrieval layer will feel like a phone without a browser. The only question left is which side you build on. I know which side I am on.
## What do FinanceBench, SimpleQA, and BrowseComp actually measure?
**FinanceBench tests open-book financial QA over real SEC filings: 150 expert questions, 361 filings, 40 companies, 9 sectors. SimpleQA tests short-form factuality with single-answer questions. BrowseComp tests multi-step web research on hard-to-find facts. Each benchmark measures a different failure mode, and each misses something the others catch.**
FinanceBench is Patronus AI's open-book evaluation. The system is given the filing and must answer the question and point to where it found the answer. The questions are the kind an analyst asks: operating margins, dividend policy, year-over-year deltas. Numerical answers are scored inside a strict ±2.5% relative tolerance, and refusals are scored as wrong. The [FinanceBench paper](https://arxiv.org/abs/2311.11944) is the primary source for the task definition and the scoring rules.
The corpus is 361 public SEC filings and financial reports spanning 40 US-listed companies across 9 sectors, with data from 2015 to 2023. Roughly 75% of the documents are 10-K annual reports; the rest are 10-Q quarterlies, 8-K material events, and earnings releases. A single 10-K runs 100,000 tokens or more. That is why this benchmark punishes naive chunking so hard: the answer to one question often lives across a table on page 47, a footnote on page 92, and a policy note in the management discussion.
The ±2.5% relative tolerance is the detail most people skip, and it is the one that separates real retrieval from vibes. A system that returns "1.2 billion" when the filing says "1.18 billion" is wrong. A system that returns the right number but cannot say where it found it is wrong. A system that refuses to answer is wrong. There is no partial credit for being close, and there is no credit for being careful. That is the right way to score financial answers, because a finance agent that is 2.6% off on a margin figure has failed its job.
SimpleQA is OpenAI's benchmark of short, fact-seeking questions with a single correct answer. It measures whether a system can retrieve and state a fact accurately, not whether it can recall trivia from training. The [SimpleQA paper](https://arxiv.org/abs/2411.04368) shows that even frontier models score under 50% on it without tools. That makes it a clean test of retrieval: the model already knows the shape of the answer, but it needs the web to get the fact right. The grading is strict: correct, incorrect, or not attempted, with no partial credit and no credit for hedging.
BrowseComp is OpenAI's benchmark for browsing agents. The answers are short and verifiable, but finding them requires visiting many pages and chaining searches. It rewards persistence and multi-step research. [OpenAI's BrowseComp page](https://openai.com/index/browsecomp) frames it as a test of whether an agent can find entangled information that a single search will not surface. Where SimpleQA asks a single fact question, BrowseComp asks a research question that takes an agent dozens of steps to answer.
What each benchmark misses is as important as what it tests. FinanceBench covers finance only, so a high score says nothing about general web factuality. SimpleQA covers short facts, not long synthesis, so it cannot measure whether an agent can build a cited report. BrowseComp rewards the agent that eventually finds the answer, not the one that finds it cheaply, so it ignores cost and latency. None of the three measures citation quality directly, which is the property that matters most for answer engines.
Read together, the three benchmarks cover the three jobs a search API actually does. FinanceBench tests retrieval over long structured documents. SimpleQA tests retrieval over short factual questions. BrowseComp tests retrieval as a multi-step loop. That is the full range of agent workloads, and it is why this trio, not a trivia leaderboard, is the one that tells you whether a search API is worth its price.
## Which search API leads each benchmark in 2026?
**Keirolabs leads FinanceBench among commercial retrieval APIs at 78%, Perplexity Sonar Deep Research leads SimpleQA at 93.9%, and Gemini Deep Research leads BrowseComp at 85.9%. The leaderboard below uses only public, published numbers, and every score links to its primary source so you can check the claim yourself.**
The leaderboard is short and it is honest. Every number below is public, published, and linked.
The providers in this comparison.
| Benchmark | What it tests | Leader | Score |
|---|---|---|---|
| FinanceBench | Cited answers over SEC filings | Keirolabs | 78% |
| SimpleQA | Short-form factuality | Perplexity Sonar Deep Research | 93.9% |
| BrowseComp | Multi-step web research | Gemini Deep Research | 85.9% |
The Gemini Deep Research number comes from [Google's Deep Research documentation](https://blog.google/innovation-and-ai/technology/developers-tools/deep-research-agent-gemini-api/), and the OpenAI BrowseComp baseline is published on [OpenAI's BrowseComp page](https://openai.com/index/browsecomp). OpenAI's own Deep Research model scored 51.5% on the same benchmark, which shows how much the field has moved since that baseline was published. Both are agent-with-search results, not bare model scores.
On FinanceBench, the reference points matter as much as the leader. A raw GPT-4o with the full filing in its context window reaches ~85%, but that oracle setup does not scale to a real agent. Standard chunk-and-retrieve collapses to ~19%. **Keirolabs sits at 78% with a deployable pipeline.**
| System | FinanceBench | Deployable |
|---|---|---|
| GPT-4o oracle (full doc in context) | ~85% | No |
| Keirolabs | 78% | Yes |
| GPT-4o + vector RAG | ~19% | Yes |
FinanceBench scores. The oracle reads the full filing and is not deployable; Keirolabs reaches 78% with a deployable pipeline; standard chunk-and-retrieve collapses to ~19%. Source: Patronus AI FinanceBench paper and Keirolabs benchmark runs.
The gap between the oracle and the deployable systems is the real story. The oracle number shows what is possible when a model can read the whole document. The vector RAG number shows what most production systems actually achieve. Keirolabs closes most of that gap while staying deployable, which is why the 78% is a production number, not a research number. Anyone can hit 85% with a full filing in context and a fat GPU bill. The hard part is hitting 78% with a pipeline that runs in production, and that is the number that matters.
The oracle is not a product. It is a lab setup that stuffs a 100K-token filing into a context window and hopes the model reads it. That does not scale to a portfolio of 40 companies, let alone a thousand. The vector RAG number is what most teams actually ship, and it is a warning: the default architecture for retrieval collapses on the exact workload agents were built for. The 78% sits between them, and it is the only one of the three that is both accurate and deployable. That is the whole point of the chart.
On SimpleQA, the gap between Keirolabs at 84% and Perplexity Sonar Deep Research at 93.9% is real, and it is worth being precise about what it means. Sonar Deep Research runs a full research loop with dozens of searches and a large reasoning model behind it. Keirolabs is a retrieval API that returns cited results for an agent to reason over. The 84% is the retrieval layer's contribution; the agent's model does the rest. That is why the [Perplexity Deep Research announcement](https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research) reports the number as an agent-with-search result, not a bare API result.
SimpleQA scores. Perplexity Sonar Deep Research runs a full research loop with a large reasoning model; Keirolabs is a retrieval API, so its 84% is the retrieval layer's contribution before the agent's model reasons. Source: Perplexity Deep Research announcement and Keirolabs benchmark runs.
For an API buyer, the comparison is not between two products at the same layer. Sonar Deep Research is a managed research product that bundles search, reasoning, and report writing. Keirolabs is a building block that an agent calls directly. If you want a finished research report, the managed product is the right tool. If you are building your own agent and want cited results at $0.25 per 1,000 queries, the API is the right tool. The 84% tells you the retrieval layer will not be your bottleneck.
Humanity's Last Exam (HLE) is the outlier that gets quoted most and means least for search APIs. It tests expert-level reasoning, not retrieval. Gemini leads at 54.6%, OpenAI scores 26.6%, and Perplexity scores 21.1%. **None of those numbers tell you whether a search API returns cited, correct answers.**
| System | HLE |
|---|---|
| Gemini | 54.6% |
| OpenAI | 26.6% |
| Perplexity | 21.1% |
HLE is a model benchmark, not a search benchmark. It measures whether a model can reason through an expert-level question, not whether a retrieval system can find and cite an answer. A search API does not change a model's HLE score much, because the hard part is reasoning, not retrieval. That is why HLE belongs in a different conversation from FinanceBench, SimpleQA, and BrowseComp. When someone quotes HLE to sell you a search API, they are quoting the wrong benchmark.
## Why does FinanceBench matter more than trivia benchmarks?
**FinanceBench matters more because it tests cited answers over real financial documents: open-book questions, grounded answers, and refusals scored as wrong. That is agent-adjacent and audit-friendly work. It measures retrieval plus citation, not recall, which is the property that matters most for answer engines.**
A trivia benchmark asks what a model remembers. FinanceBench asks whether a system can find the answer in a 100K-token filing, extract the right number from a table, and point to the page. That is the exact job of a finance research agent, a due-diligence agent, or any agent that must answer with evidence. The answer is only correct if it is grounded in the document.
The scoring is strict. Information-lookup questions use semantic equivalence. Numerical questions must land within ±2.5% relative error. Logical-inference questions require the answer plus the exact evidence location. A hedge or a "not found" is scored as incorrect. **That is why FinanceBench is a better predictor of production behavior than a trivia leaderboard.** A model that memorized the answer to a trivia question is not being tested on retrieval at all. A system that answers a FinanceBench question correctly has found the number, kept it intact through extraction, and committed to it. Those are the exact components a search API controls.
The audit-friendly part matters for adoption. When an agent answers a financial question, a human needs to verify the answer. FinanceBench forces the system to produce the evidence string and the source location, so the answer can be checked. Trivia benchmarks do not require that, which is why a high trivia score tells you little about whether an agent can be trusted with real documents. In finance, the citation is not a nicety. It is the product.
FinanceBench is also the closest thing the field has to a benchmark for the retrieval half of an agent. The questions are open-book, so the model is not being tested on what it memorized. It is being tested on whether the retrieval layer found the right region of the document, whether the extraction layer kept the numbers intact, and whether the reasoning layer committed to an answer. Those are the exact components a search API controls.
Finance is also where agents get deployed first. The people building finance research agents, due-diligence agents, and earnings-call copilots are the ones with the budget and the tolerance for a real evaluation. They are not going to trust a trivia score. They are going to run the public split, check the evidence strings, and count the cost per correct answer. A vendor that does well on FinanceBench is a vendor that has been tested by the hardest buyers in the market.
There is a second reason FinanceBench matters that has nothing to do with finance. Original benchmark data is a content-effort signal. A published 78% with a public script and a live dashboard is the kind of specific, checkable claim that answer engines cite. It is also a link magnet: anyone writing about financial QA or RAG accuracy will link to the page that has the number and the evidence. Trivia benchmark scores are a dime a dozen. A reproducible result on a hard, domain-specific benchmark is not.
## What is the cost per correct answer, not per query?
**Cost per correct answer is price per query divided by accuracy, and Keirolabs at $0.25 per 1,000 semantic queries and 78% accuracy pays about $0.32 per 1,000 correct answers. A cheap-but-accurate API beats an expensive one, because accuracy multiplies the value of every query you pay for.**
The math is simple. At $0.25 per 1,000 queries, one query costs $0.00025. At 78% accuracy, 1,000 queries produce 780 correct answers. The cost per correct answer is $0.00025 divided by 0.78, which is about $0.00032. Per 1,000 correct answers, that is $0.32.
The gap compounds at scale. A provider that charges five times more per query and answers correctly half as often pays ten times more per correct answer. A finance agent that answers 100,000 questions a month pays for every query, but only the correct answers have value. **The correct-answer framing changes which API looks cheap.**
Consider two APIs at 1 million queries a month. API A charges $0.25 per 1,000 queries and answers 78% correctly: 780,000 correct answers for $250. API B charges $1.00 per 1,000 queries and answers 50% correctly: 500,000 correct answers for $1,000. API A delivers more correct answers for a quarter of the price. That is the difference between cost per query and cost per correct answer.
| API | Price per 1K | Accuracy | Correct answers | Monthly bill | Cost per 1K correct |
|---|---|---|---|---|---|
| A (Keiro-style) | $0.25 | 78% | 780,000 | $250 | $0.32 |
| B (expensive) | $1.00 | 50% | 500,000 | $1,000 | $2.00 |
Cost per 1,000 correct answers at 1 million queries a month. Keiro-style pricing ($0.25/1K, 78% accuracy) pays $0.32 per 1,000 correct answers; a provider at $1.00/1K with 50% accuracy pays $2.00. Source: vendor pricing pages and published benchmark scores.
This changes everything about how you buy search. The vendor that looks expensive on the landing page can be the cheapest in production, and the vendor that looks cheap can be the most expensive. You cannot see it until you divide by accuracy. That is why I benchmark from the bills, not the landing pages.
Keirolabs prices semantic search at $0.25 per 1,000 queries and SERP at $0.10 per 1,000 queries. The free tier covers 1,000 queries per month. When you compare APIs, compare cost per correct answer, not cost per query.
The SERP price matters for a different reason. SERP results are the raw material an agent turns into an answer, and at $0.10 per 1,000 queries the retrieval step is nearly free. The expensive part of an agent is the reasoning model, not the search. A search API that costs a rounding error per query lets you spend your budget on the model that turns results into answers. That is the right division of labor, and it only shows up when you price the two layers separately.
The free tier is the same story. 1,000 queries per month is enough to run a real evaluation, not a demo. You can point the API at the FinanceBench public split, run the 150 questions, and see the 78% for yourself before you spend a dollar. That is how you buy infrastructure in 2026: you test it on your own workload, you count the cost per correct answer, and then you scale. The free tier exists to make that test possible.
## Why does a cheap-but-accurate API beat an expensive one?
**A cheap-but-accurate API beats an expensive one because accuracy multiplies the value of every query you pay for. At $0.25 per 1,000 queries and 78% accuracy, Keirolabs pays $0.32 per 1,000 correct answers, while a turnkey deep research product at $1-7 per task pays $1,000-7,000 per 1,000 tasks.**
The comparison gets sharper when you bring in the managed research products. A turnkey deep research product charges $1 to $7 per task. One task, one answer. Keirolabs charges $0.25 per 1,000 queries, and at 78% accuracy that is $0.32 per 1,000 correct answers. The managed product is 3,000 to 20,000 times more expensive per correct answer.
That is not a rounding error. That is the difference between a product that can afford to answer 100,000 questions a month and a product that cannot. The expensive product bundles search, reasoning, and report writing into one call, and you pay for the bundle whether you need it or not. The API sells the retrieval layer alone, and your model does the reasoning. If you are building an agent, you already own the reasoning model. Paying for a second one inside a research product is double-spending.
Accuracy is a cost lever, not just a quality score. At $0.25 per 1,000 queries, a system at 19% accuracy pays $1.32 per 1,000 correct answers. A system at 78% pays $0.32. The same price per query, four times the cost per correct answer. That is what the vector RAG collapse to ~19% actually costs in dollars.
Cost per 1,000 correct answers at $0.25 per 1,000 queries, by accuracy. At 19% accuracy the same retrieval costs $1.32 per 1,000 correct answers; at 78% it costs $0.32. Accuracy is a cost lever, not just a quality score.
This is why the FinanceBench number matters more than the SimpleQA number for cost. FinanceBench is the benchmark where the deployable systems separate. A retrieval API at 78% on FinanceBench is not just more accurate than one at 19%. It is four times cheaper per correct answer on the exact workload that matters. The accuracy gap and the cost gap are the same gap.
The latency story points the same direction. Keirolabs' /search/fast endpoint returns in about 487ms. A fast retrieval step means your agent spends its time reasoning, not waiting. No benchmark score tells you that, which is why the score is only half the story. The other half is the bill.
There is a reason the managed research products cost $1 to $7 per task, and it is not the search. It is the reasoning model running a dozen steps, the report writer, the margin on a finished product. When you buy a retrieval API instead, you keep that margin. Your model already does the reasoning. The API just needs to find the right pages and hand back the evidence, and it does that for $0.25 per 1,000 queries. The arithmetic is not close.
## What are the limits of these benchmarks?
**Every benchmark has limits. FinanceBench's public split is 150 questions, SimpleQA rewards short answers and punishes hedging, and BrowseComp rewards persistence over efficiency. A single number never captures latency, cost, or whether the citations actually support the answer, so read any score as a point-in-time measurement with a published run config.**
FinanceBench's public evaluation set is 150 questions drawn from a larger dataset. A 78% on that split is meaningful, but it is not the same as a 10,000-question evaluation. SimpleQA is graded correct, incorrect, or not attempted, so a system that hedges gets penalized. BrowseComp rewards the agent that eventually finds the answer, not the one that finds it cheaply.
Benchmark scores also drift. Models and APIs change monthly, and a leaderboard from January is stale by August. **The right way to read any of these numbers is as a point-in-time measurement with a published run config, not as a permanent ranking.** A score without a run config is a rumor. A score with a run config is a result you can check.
There is also a gap between benchmark questions and production queries. FinanceBench questions are written by experts and have a single correct answer. Production queries are messier: they are ambiguous, they reference the wrong company, or they ask for something the document does not contain. A benchmark score is a floor, not a ceiling.
The same honesty applies to Keirolabs' own numbers. The 78% is on the public 150-question split, not the full dataset. The 84% on SimpleQA is a point-in-time measurement. Both are reproducible, and both will change as the pipeline changes. We publish the run config so the numbers can be checked, and we expect them to be checked.
One more limit deserves attention: benchmark scores measure the API, not your agent. A 78% on FinanceBench assumes the calling code routes the query, extracts the evidence, and commits to an answer the way the benchmark script does. If your agent drops the evidence string or refuses on hard questions, your production accuracy will be lower than the benchmark. Treat the published number as the ceiling for a well-built agent, not a guarantee for a sloppy one.
Benchmarks can also be gamed, and the gaming is getting sophisticated. A vendor can tune its pipeline to the public split, memorize the answers, or pick the judge that flatters it. That is why the run config matters as much as the score. A number with a published script, a fixed split, and a deterministic judge is hard to game. A number with no script is a marketing claim. When you see a benchmark score, ask for the run config before you ask for the demo.
## How do I test a search API against these benchmarks myself?
**Run the public splits yourself. FinanceBench's 150-question public set and SimpleQA's questions are public, the scoring is deterministic, and Keirolabs publishes its benchmark script and run config. Reproducing a number beats trusting a leaderboard, and it takes an afternoon.**
The Keirolabs FinanceBench run uses the fixed public 150-question split with deterministic programmatic scoring for the numerical and lookup categories. The benchmark script and run config are public, so the 78% can be reproduced independently. Point your own API at the same questions, use the same judge, and compare.
To test Keirolabs against your own questions, call the v2 content endpoint:
```bash
curl -X POST https://api.keirolabs.cloud/api/v2/search/content \
-H "Authorization: Bearer YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"query":"What was Apple operating margin in fiscal 2023?","mode":"medium","maxResults":5}'
```
The response includes the extracted content and source locations, so you can check whether the answer is grounded before you trust it. **Run the same questions through two APIs and grade the answers yourself. That is the only benchmark that matters for your use case.**
A practical test takes an afternoon. Take 50 questions from your own domain, run them through two or three APIs, and grade the answers on correctness and citation quality. Count the cost per correct answer for each. That number, not a leaderboard, is what you should put in your budget.
Grade on three things, in order. First, is the answer correct? Second, does the citation actually support it? Third, how much did the correct answers cost? The first two are quality, the third is the bill, and the combination is the only number that belongs in a budget. A vendor that scores 90% on your questions but charges ten times more per correct answer is not the vendor you want. A vendor that scores 78% and charges a rounding error is the vendor you want, and the benchmark numbers are a good prior for what you will find.
## What does the live FinanceBench dashboard show?
**Keirolabs publishes its FinanceBench results on a live dashboard at keirolabs.cloud/financebench, with the benchmark script and run config public. The 78% is reproducible on the fixed public 150-question split with deterministic programmatic scoring, so anyone can check it and anyone can rerun it.**
The dashboard is not a screenshot. It is a live page that re-runs the evaluation and shows the current score, the per-question breakdown, and the run config. Anyone can check it, and anyone can rerun it. That is the difference between a benchmark claim and a benchmark result.
Original benchmark data is also what answer engines cite. When ChatGPT, Perplexity, or Google AI Overviews answer a question about search API accuracy, they lift numbers from pages that publish them with a run config attached. A published 78% with a public script is a citation target. A vague claim is not. That is why we publish the number, the script, and the dashboard, and why we expect competitors to do the same.
The same retrieval and structured-extraction pipeline behind the 78% is available through the API, with citation-grounded answers and evidence locations. The free tier gives you 1,000 queries per month, no credit card required.
## See the live FinanceBench dashboard and start free
**Keirolabs publishes its FinanceBench results on a live dashboard at [keirolabs.cloud/financebench](https://keirolabs.cloud/financebench), with the benchmark script and run config public. The free tier gives you 1,000 queries per month, no credit card required, and the same pipeline behind the 78% is available through the API.**
Semantic search is $0.25 per 1,000 queries and SERP is $0.10 per 1,000 queries. The API base is `api.keirolabs.cloud`, with the v2 content endpoint at `POST https://api.keirolabs.cloud/api/v2/search/content`. [Create a free account](https://keirolabs.cloud) and run the benchmark yourself.
The next 12 months will make this look obvious. The market has already split. The only question left is which side you build on. I know which side I am on.
*Last updated: August 2026.*
## About the author
Dave builds AI agent and RAG pipelines and writes about the infrastructure behind them. He benchmarks search APIs from the bills, not the landing pages. You can follow his work on [GitHub](https://github.com/Manasbh).
## FAQ
### What is FinanceBench?
FinanceBench is Patronus AI's open-book benchmark of 150 expert questions over 361 SEC filings from 40 US-listed companies across 9 sectors. Systems must answer the question and cite where they found it.
### What is SimpleQA?
SimpleQA is OpenAI's benchmark of short, fact-seeking questions with a single correct answer. It measures whether a system can retrieve and state a fact accurately, not just recall it from training.
### Which search API is most accurate?
On FinanceBench, Keirolabs scores 78%, ahead of the commercial retrieval APIs we evaluated. On SimpleQA, Keirolabs scores 84%. Perplexity Sonar Deep Research leads SimpleQA at 93.9%.
### Is Keirolabs' 78% on FinanceBench reproducible?
Yes. The run uses the fixed public 150-question split with deterministic programmatic scoring, and the benchmark script and run config are public.
