Deep search on SimpleQA

The result

The headline number is 95.3%: 953 of 1,000 seeded SimpleQA questions had the gold answer present in the data the pipeline returned. The raw normalized substring score is 87.9%; a hand audit of every raw miss — date and comma variants, abbreviations, transliterations, split compounds — rescues 7.6 points. 43 misses stay misses.

Every question in the sample is deterministic: seed 42, a Fisher–Yates shuffle over OpenAI's 4,326-question SimpleQA set.6 Every returned payload — snippets and crawled page text — is preserved in results.json in the repo, so any judged call can be re-audited by hand.1

The inversion

A normal search API returns the same artifact every time: ranked links with two-line snippets. Crawling, parsing and answer-finding stay your problem. Keiro's deep search moves that work server-side — the response carries the answer's context, not a list of places to look for it.

On this benchmark the inversion is worth 22.9 points: 72.4% when the gold answer has to appear in SERP snippets alone, 95.3% with the fetch and re-rank stages behind it.

Four stages between query and answer

One POST to /v1/search in, one ranked payload out. Everything between — fanout, fetch, scoring, re-rank — runs server-side. No reader model anywhere in the path.

What counts as a hit

The raw check is mechanical: a normalized substring test over everything returned. The judged pass adds a hand audit of every miss, and it is deliberately stingy about rescues.

Benchmarks

Each provider's strongest published SimpleQA result we could find, with the harness that produced it. The harnesses differ — answer models, judges, question counts — so treat the gaps as directional. The structural difference that favors Keiro: every other score gets a reader LLM to repair retrieval mistakes; ours does not.

Provider	SimpleQA (%)	Δ vs Keiro	Harness	Ref
Keiro deep search	95.30	—	Own harness · retrieval-only judged check · n=1,000, seed 42	[1]
Firecrawl	94.7	−0.6	GPT-5.4 agent, high reasoning, ≤20 search/extraction calls, official grader	[2]
Tavily	93.3	−2.0	GPT-4.1 answers from retrieved docs, official SimpleQA classifier · full 4,326-question set · self-published	[3]
you.com	92.09	−3.2	GPT-5.4 nano synthesis + GPT-5.4 mini judge · open-source harness	[4]
Exa	91.9	−3.4	GPT-5.4 agent (also 90.06% on you.com's harness)	[2]
Parallel	91.0	−4.3	GPT-5.4 agent (also 89.78% on you.com's harness)	[2]
Claude Search	90.5	−4.8	GPT-5.4 agent, native search tool	[2]
Perplexity	85.92	−9.4	GPT-4.1 answers, official SimpleQA classifier	[5]
Google	82.15	−13.2	SERP via Serper, GPT-4.1 answers (80.17% on you.com's harness)	[5]
Brave Search	76.05	−19.3	GPT-4.1 answers, official SimpleQA classifier	[5]

The 43 true misses

After judging, 43 of 1,000 questions were true misses. They cluster into four failure classes — in three of them the fact is not on the reachable open web at all.

Reproduce this

testbench.js is a standalone, zero-dependency Node ≥ 18 script — no npm install. It seeds a deterministic Fisher–Yates sample of SimpleQA, runs each question through the pipeline, and writes every record — including the full returned payload — to results.json.

$ SAMPLE_SIZE=1000 SAMPLE_SEED=42 node testbench.js
# SimpleQA retrieval eval: 1000 questions (of 4326), seed=42
# per-question records (payload included, for judging) -> results_simpleqa.json
# { hit_rate: 0.8773, content_hit_rate: 0.7932, top1: 0.737, retries: 267 }--- testbench.js (core) ---
// SimpleQA retrieval eval for the Keiro search API.
// The API's one job: surface the gold answer in the returned snippets/content.
// No reader model, no answer extraction - retrieval, ranked.

// seeded RNG (mulberry32) + Fisher-Yates sample: deterministic per seed
function mulberry32(seed) {
  return function () {
    seed |= 0; seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

// normalize: lowercase, strip articles, strip non-alnum, collapse whitespace
function normalize2(s) {
  return String(s || "")
    .toLowerCase()
    .replace(/\b(a|an|the)\b/g, " ")
    .replace(/[^a-z0-9 ]/g, " ")
    .replace(/\s+/g, " ")
    .trim();
}
const goldIn = (text, gold) =>
  !!text && normalize2(gold) !== "" && normalize2(text).includes(normalize2(gold));

async function runOne(q) {
  const res = await fetch(API_URL, {
    method: "POST",
    headers: { "Authorization": "Bearer " + KEY, "Content-Type": "application/json" },
    body: JSON.stringify({ query: q.question, max_results: 10 }),
  });
  const payload = await res.json();
  const results = payload.results || [];
  const fetched = results.filter((r) => r.content);

  const rec = {
    snippet_hit: goldIn(results.map((r) => r.snippet || "").join(" "), q.gold),
    content_hit: goldIn(fetched.map((r) => r.content || "").join(" "), q.gold),
    gold_rank: firstRankHit(results, q.gold),   // top-1 tracking
    latency_ms: payload.meta ? payload.meta.latency_ms : 0,
    // full returned payload kept, so every judged call is auditable
    snippets: results.map((r) => ({ title: r.title, snippet: r.snippet })),
    content: fetched.map((r) => ({ title: r.title, text: r.content.slice(0, 8000) })),
  };
  rec.hit = rec.snippet_hit || rec.content_hit;
  return rec;
}

• SAMPLE_SIZEquestions per run — default 100, this report used 1,000
• SAMPLE_SEEDseed for the Fisher–Yates sample (default 42)
• SAMPLE_OFFSETslice deeper into the 4,326-question set
• BENCH_OUTresults file (default results_simpleqa.json)

Each question costs ~1 Keiro credit and ~2.2 s of pacing at the starter tier's 30 req/min. The full 1,000-question results file — every snippet and crawled page — is in the repo.1