Deep search on SimpleQA

The result

The headline number is 95.3%: 953 of 1,000 seeded SimpleQA questions had the gold answer present in the data the pipeline returned. The raw normalized substring score is 87.9%; a hand audit of every raw miss — date and comma variants, abbreviations, transliterations, split compounds — rescues 7.6 points. 43 misses stay misses.

Every question in the sample is deterministic: seed 42, a Fisher–Yates shuffle over OpenAI's 4,326-question SimpleQA set.6 Every returned payload — snippets and full page text — is preserved in results.json in the repo, so any judged call can be re-audited by hand.1

The reverse framework

Search was built to find documents. Ask a search engine a question and it returns ten places the answer might live — the reading is left to you.

Keiro resolves facts. The question is compiled into the fact it asks for; the fact is located on the pages carrying it right now; every candidate is read end-to-end until the fact is proven; the response returns the answer with its evidence ranked. Search runs forward, from words to documents. Resolution runs in reverse, from the fact back to the web.

On this benchmark the reversal is worth 22.9 points: 72.4% when the gold answer has to appear in SERP snippets alone, 95.3% when every candidate is read in full and ranked by the evidence it carries.

The resolution engine

Most infrastructure answers one question: which URLs match. Ours answers the reverse question: where does the fact live right now. One POST to /v1/search in, one proven answer out — no index to build, no corpus to maintain, nothing cached. The database is the live web itself. Four stages take a question to its fact:

What counts as a hit

The raw check is mechanical: a normalized substring test over everything returned. The judged pass adds a hand audit of every miss, and it is deliberately stingy about rescues.

Benchmarks

Each provider's strongest published SimpleQA result we could find, with the harness that produced it. The harnesses differ — answer models, judges, question counts — so treat the gaps as directional. The cost column is each provider's published list price per 1,000 requests (refs 7–16; detail in the next section). The structural difference that favors Keiro: every other score gets a reader LLM to repair retrieval mistakes; ours does not.

Provider	SimpleQA (%)	Δ vs Keiro	Cost / 1k	Harness	Ref
Keiro deep search	95.30	—	$4.447	Own harness · retrieval-only judged check · n=1,000, seed 42	[1]
Firecrawl	94.7	−0.6	$10.008	GPT-5.4 agent, high reasoning, ≤20 search/extraction calls, official grader	[2]
Tavily	93.3	−2.0	$8.009	GPT-4.1 answers from retrieved docs, official SimpleQA classifier · full 4,326-question set · self-published	[3]
you.com	92.09	−3.2	$5.0010	GPT-5.4 nano synthesis + GPT-5.4 mini judge · open-source harness	[4]
Exa	91.9	−3.4	$12.0011	GPT-5.4 agent (also 90.06% on you.com's harness)	[2]
Parallel	91.0	−4.3	$1.0012	GPT-5.4 agent (also 89.78% on you.com's harness)	[2]
Claude Search	90.5	−4.8	$10.0013	GPT-5.4 agent, native search tool	[2]
Perplexity	85.92	−9.4	$5.0014	GPT-4.1 answers, official SimpleQA classifier	[5]
Google	82.15	−13.2	$5.0015	SERP via Serper, GPT-4.1 answers (80.17% on you.com's harness)	[5]
Brave Search	76.05	−19.3	$5.0016	GPT-4.1 answers, official SimpleQA classifier	[5]

What a request costs

Deep search is $4.44 per 1,000 requests, list price — one flat meter that covers the entire pipeline this page describes: live-web resolution, full-page reading (an average of 7.9 pages per query in this benchmark), proof scoring and re-rank.

That sits between the cheapest raw-search tier and the $5–10 band where most search APIs return ranked links and excerpts. The difference is what the meter covers: the full-page layer behind +15.3 of the 95.3 points is, for most of the field, a second product with its own metering — scrape credits, contents endpoints, or LLM tokens.

The 43 true misses

After judging, 43 of 1,000 questions were true misses. They cluster into four failure classes — in three of them the fact is not on the reachable open web at all.

Reproduce this

testbench.js is a standalone, zero-dependency Node ≥ 18 script — no npm install. It seeds a deterministic Fisher–Yates sample of SimpleQA, runs each question through the pipeline, and writes every record — including the full returned payload — to results.json.

$ SAMPLE_SIZE=1000 SAMPLE_SEED=42 node testbench.js
# SimpleQA retrieval eval: 1000 questions (of 4326), seed=42
# per-question records (payload included, for judging) -> results_simpleqa.json
# { hit_rate: 0.8773, content_hit_rate: 0.7932, top1: 0.737, retries: 267 }--- testbench.js (core) ---
// SimpleQA retrieval eval for the Keiro search API.
// The API's one job: surface the gold answer in the returned snippets/content.
// No reader model, no answer extraction - retrieval, ranked.

// seeded RNG (mulberry32) + Fisher-Yates sample: deterministic per seed
function mulberry32(seed) {
  return function () {
    seed |= 0; seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

// normalize: lowercase, strip articles, strip non-alnum, collapse whitespace
function normalize2(s) {
  return String(s || "")
    .toLowerCase()
    .replace(/\b(a|an|the)\b/g, " ")
    .replace(/[^a-z0-9 ]/g, " ")
    .replace(/\s+/g, " ")
    .trim();
}
const goldIn = (text, gold) =>
  !!text && normalize2(gold) !== "" && normalize2(text).includes(normalize2(gold));

async function runOne(q) {
  const res = await fetch(API_URL, {
    method: "POST",
    headers: { "Authorization": "Bearer " + KEY, "Content-Type": "application/json" },
    body: JSON.stringify({ query: q.question, max_results: 10 }),
  });
  const payload = await res.json();
  const results = payload.results || [];
  const fetched = results.filter((r) => r.content);

  const rec = {
    snippet_hit: goldIn(results.map((r) => r.snippet || "").join(" "), q.gold),
    content_hit: goldIn(fetched.map((r) => r.content || "").join(" "), q.gold),
    gold_rank: firstRankHit(results, q.gold),   // top-1 tracking
    latency_ms: payload.meta ? payload.meta.latency_ms : 0,
    // full returned payload kept, so every judged call is auditable
    snippets: results.map((r) => ({ title: r.title, snippet: r.snippet })),
    content: fetched.map((r) => ({ title: r.title, text: r.content.slice(0, 8000) })),
  };
  rec.hit = rec.snippet_hit || rec.content_hit;
  return rec;
}

• SAMPLE_SIZEquestions per run — default 100, this report used 1,000
• SAMPLE_SEEDseed for the Fisher–Yates sample (default 42)
• SAMPLE_OFFSETslice deeper into the 4,326-question set
• BENCH_OUTresults file (default results_simpleqa.json)

Each question costs ~1 Keiro credit and ~2.2 s of pacing at the starter tier's 30 req/min. The full 1,000-question results file — every snippet and resolved page — is in the repo.1