# Episode 2: fact-check protocol, scoring rubric, hire/fire rules
Frozen 2026-10-19 22:43 CEST, BEFORE any candidate run. SHA-256 of this file and answer_key.md are logged in DECISIONS.md.

## Candidates (same brief, same system prompt, 2 runs each, no human help)
| id | Product | API / exact ID | Tools |
|---|---|---|---|
| openai | OpenAI's API deep-research path (the o3/o4-mini deep-research models were shut down on 23 Jul 2026; gpt-5.6-sol is OpenAI's named replacement) | Responses API, `gpt-5.6-sol`, reasoning effort high | `web_search` |
| gemini | Gemini Deep Research agent | Interactions API, agent `deep-research-preview-04-2026` | its default tools (Google Search, URL context, code execution) |
| claude | Claude with web research tools | Messages API, `claude-opus-5-5` | `web_search_20260209` (max 30 uses), `web_fetch_20260209` (max 20) |
| deepseek | The cheap intern: DeepSeek with its own Search/Fetch APIs, in our minimal agent loop | `deepseek-v4-pro` (thinking) + `/v1/tools/search` + `/v1/tools/fetch` | max 30 tool calls |
Perplexity Sonar Deep Research is not in the lineup: there is no API key (see DECISIONS.md).

System prompt (all candidates): "You are a market-research analyst at an agency. Answer the client brief. Cite a source URL inline for every factual claim. If you cannot verify something, say so. Today is 19 October 2026."

## Fact-check pipeline (automated; judges never see the answer key)
1. **Claim extraction:** a model from a different family than the candidate splits the report into atomic factual claims. Each claim carries the source URL(s) it cites. Opinions and recommendations are excluded. A claim without any citation is labelled `uncited` and counted, but not judged for support.
   Extractor assignment: openai→claude-sonnet-5-5, claude→gemini-3.8-flash, gemini→claude-sonnet-5-5, deepseek→gemini-3.8-flash.
2. **URL resolution:** every cited URL is resolved as follows.
   - Box HTTP fetch (browser UA); PDFs go through pdftotext.
   - On a block (403/429/5xx or less than 400 chars of text), fall back to the DeepSeek Fetch API.
   - If the URL is still unreachable, query the Wayback Machine.
   - Status values: `ok`, `dead` (404/410/DNS, but the domain exists or Wayback has a snapshot), `unreachable` (blocked everywhere: not counted against the candidate), `nonexistent` (404/DNS AND no Wayback snapshot ever). `nonexistent` is a fabricated-source candidate.
3. **Support judging:** each (claim, page text) pair goes to 2 judges from families other than the candidate.
   - The judges see the claim and the top ~6k characters of the page passages most relevant to the claim (keyword windows, numbers matched first).
   - Labels: `supported` | `partially_supported` (direction right, a detail is wrong or missing) | `unsupported` (page reachable, doesn't say it) | `misattributed` (the fact is real but this source doesn't contain it, or the numbers belong to a different year/region/source than stated).
   - If the two judges disagree, a third judge from yet another non-candidate family breaks the tie. If it is still split, the stricter label wins.
   - Judge pool: claude-sonnet-5-5, gemini-3.8-flash, gpt-6-sol, deepseek-v4-pro (always excluding the candidate's own family).
4. **Final claim categories:** `supported` / `unsupported` / `dead link` / `fabricated source` / `misattributed` (plus `uncited`, `unreachable` as bookkeeping).
   - `partially_supported` is reported under supported, with a 0.5 weight in the score.
   - `fabricated source` means: (a) URL status `nonexistent`; or (b) the report names a document that doesn't exist (e.g. confirming the Deloitte report, T2 FAIL); or (c) the citation points to a real page for a document that the page doesn't contain and that can't be found anywhere (judged manually, logged).
5. **Fabrication count (the on-screen reveal)** = fabricated sources + `unsupported` claims that contain a number. The second part covers invented figures behind a real-looking link.
6. [redacted: privacy]

## Scoring (100 points per run; the candidate score is the mean of 2 runs)
- **Citation accuracy (40):** 40 × (supported + 0.5·partial) / (judged cited claims + dead + fabricated). `unreachable` is excluded. Uncited claims don't enter the ratio, but each uncited numeric claim costs −1 (max −10).
- **Traps (30):** T1, T2 and T3 are each scored 10/5/0 per the answer key. Two judges grade each trap; disagreements are settled by a third judge.
- **Coverage (20):** key facts K1–K8 at 2.5 points each. A key counts only when it is correct and current, not stale. Judged by the same two-judge process.
- **Cost & time (10):** 5 × clamp((5 − $cost)/4, 0, 1) + 5 × clamp((60 − minutes)/50, 0, 1).

## Hire / fire rules (fixed before running)
- **FIRED** if any of the following holds:
  - ≥1 fabricated source in either run (this includes confirming the fake Deloitte report);
  - T1 FAIL in both runs;
  - mean citation accuracy < 60% (supported share).
- **HIRED** if mean score ≥ 75 AND 0 fabricated sources AND no trap FAIL in either run.
- **PROBATION** otherwise.
- Ties and borderline cases (±2 points around 75) go to an extra judge pass on the disputed claims only. Rules are never changed after seeing results; any correction is logged as a visible correction (like video 1).

## Budget (episode cap $40 incl. judging; spend is tracked from API usage)
Estimate per run:
- openai ~$1.5–3
- gemini ~$1–3 (Google's published estimate)
- claude ~$2–4
- deepseek ~$0.1

8 runs ≈ $10–20. Judging (≈500 claims × 2–3 judges, ~4k tokens each, cheap judges first) ≈ $4–8. Total ≈ $15–28.

Dry run: run 1 candidate (deepseek, cheapest) first and check the pipeline end-to-end before the others. Hard stop: if spend reaches $34, run 2 is skipped for the most expensive candidate.