Best-of-3 with a judge: the judge never picks a winner
Three models rewrote the same transcript chunk in parallel and a 99-line judge merged them, anchored to the original. Seven weeks of audited production later I fired the jury for one reasoning model, and parked the machinery one HTTP call from revenge.
14 LLM calls to enhance one recording. That was my transcription pipeline for its first seven weeks. Today it makes 8. The six I cut belonged to the best architecture in the repo: best-of-3 with a judge. Three models rewrite the same transcript chunk in parallel, and a fourth call resolves them.
And here is the first thing everyone gets wrong: the judge never picked a winner. Not once. It was never asked to.
The second thing: best-of-3 ran on exactly 2 of 8 pipeline stages, and on 2026-05-11 I killed it for a single model. The machinery stayed on main, tests green, one dashboard PUT from revenge. This post is the mechanism, the real numbers, and why "killed" and "deleted" are different words.
Where a jury earns its cost
transcript-optimizer is my personal transcription service. One user: me. Zoom and Drive recordings go in. Chirp 3 (GCP Speech-to-Text v2) transcribes 15-minute chunks. Then an 8-stage LLM pipeline enhances the raw transcript: fusion, classify, speakers, cleanup, domain, structure, translate, summarize. Session-typed summaries land in Google Drive and Slack. Python, FastAPI, arq workers on a GCE VM, PostgreSQL with pgvector. 175 commits, built from a 43-task plan. A playground, not a product. But an instrumented one, and the instrumentation is the point of this story.
Six of the eight stages have roughly one right answer. Classify picks a session type out of seven. Speakers assigns labels. Structure formats. You don't need three opinions for that.
Two stages are different: cleanup and domain. Cleanup strips filler and stutters and repairs punctuation. Domain fixes the jargon the ASR mangled. Both are subjective rewrites. They're where models disagree most, and where a rewrite can quietly invent a sentence nobody said. I already run code reviews through Claude, Codex, and Gemini in parallel, because the union of three blind spots is smaller than any one of them. Same bet here. So exactly these two stages got a jury:
# config/pipeline.yaml (production, until the 2026-05-11 kill)
cleanup:
models: ["claude", "gemini", "openai"]
mode: "best_of_3"
temperature: 0.3
max_tokens: 16384
judge_model: "claude"
enabled: true
domain:
models: ["claude", "gemini", "openai"]
mode: "best_of_3"
temperature: 0.2
max_tokens: 16384
judge_model: "claude"
enabled: true
The arithmetic is blunt. Three parallel candidates plus one judge is 4 LLM calls per stage. Six single-Claude stages plus two juried stages: 14 enhancement calls per recording. The jury also wasn't three clones. Working defaults: claude on sonnet-4, gemini on 2.5-flash, openai on gpt-5.4-mini. The repo's hardcoded price table carried them at $3/$15, $0.15/$0.60, and $0.40/$1.60 per million tokens in and out. (That table is itself stale: Google's live 2.5-flash rate is $0.30/$2.50 today.) One mid, two cheap, three different labs. Disagreement between different models is signal; disagreement between clones is noise. The expensive model enters this story later, and not as a juror.
The mechanism
Fan-out with a failure ladder
_run_best_of_3 in src/enhancement/pipeline.py (206 lines total) dispatches each provider on a copy of the pipeline context via asyncio.gather. A failed model logs best_of_3_model_failed and gets dropped. Then the ladder:
- All three fail:
RuntimeError. The stage fails loudly. No silent degradation. - Exactly one survives: its output is used directly. No judge call. Arbitrating one opinion is theater, and theater costs tokens.
- Two or more survive: the judge merges.
The judge itself (src/enhancement/judge.py, 99 lines) has one more trick. Its prompt is shaped for three versions, so two survivors get padded:
# src/enhancement/judge.py
if len(versions) < 2:
raise ValueError("judge_best_of_3 requires at least 2 versions")
# Pad to 3 if we got partial failures (2 of 3 succeeded)
while len(versions) < 3:
versions.append(versions[-1])
Duplicate the last survivor. Dumb, explicit, correct. Eight tests in tests/test_enhancement/test_judge.py pin all of it: the merge, the padding, the rejection of single and empty inputs, that the prompt carries the model names, and that every judge call lands in the audit table.
The anchored merge
The default mental model of an LLM judge is a courtroom. Candidates argue, judge scores, one wins. My judge doesn't score and nothing wins. The whole 44-line prompt (src/enhancement/prompts/judge.md) is built around synthesis:
Your task is to produce a single **merged** transcript that takes the best
elements from each version.
...
3. Do NOT add content that doesn't exist in the original.
4. Do NOT remove content that exists in the original.
...
Return ONLY the merged transcript. No commentary, no explanations, no metadata.
The judge receives the original transcript plus versions A, B, and C, each labeled with the model that produced it. It weighs four criteria (accuracy, readability, completeness, consistency) and emits one merged transcript. Call this the anchored merge: synthesis referenced against the source, under hard no-add and no-remove rules.
Why merge instead of pick? Three reasons, and they generalize past transcripts.
Picking discards two-thirds of what you just paid for. Claude nails the paragraph breaks. Gemini catches the mangled product name. OpenAI keeps a sentence the other two dropped. A picker keeps one of those wins. A merger keeps all three.
Scoring prose is fake precision. There is no scalar for "better transcript." Ask a model for a 1-10 grade and you get a confident number carrying no information. Ask it to merge and it must actually reconcile the disagreements, span by span. The work is the output.
The anchor is the hallucination guard. A rewrite stage is exactly where an LLM invents content. Candidates can drift. The merge cannot, because it is checked against the original, not against the candidates. A hallucinated sentence in version B loses the vote against the source. This is why the judge gets the raw transcript at all: not as a fourth opinion, but as the ground the other three stand on.
One client, every call audited
All of it runs through one async client (src/enhancement/llm_client.py, 395 lines) wrapping the anthropic, openai, and google-genai SDKs. Per-provider semaphores cap concurrency (claude 5, gemini 10, openai 5). A per-provider circuit breaker trips after 5 consecutive failures, cools down for 60 seconds, and is exposed as a Prometheus gauge. A hardcoded price table turns token counts into dollars at call time.
Every complete() returns tokens, computed cost_usd, latency_ms, and a 16-hex-char SHA-256 prompt hash. Stage runners persist each response as a row in a pipeline_runs table. Judge calls are recorded under "{stage}_judge", so arbitration cost is never hidden inside candidate cost. The repo convention is one line: "Every LLM call is audited." That habit is what made the end of this story cheap. Hold the thought.
One portability gotcha worth stealing: on the OpenAI reasoning path the client switches to max_completion_tokens, passes reasoning_effort through extra_body, and stops sending temperature, because reasoning models reject non-default temperature. extra_body was a deliberate choice over the Responses API, to avoid pinning an openai-python version.
Sensitivity overrides mode
Seven session types, three sensitivity tiers. Therapy sessions are restricted, and the orchestrator force-downgrades restricted content from best_of_3 to single. It's a hard rule in pipeline.py and a "Don't" in the repo's CLAUDE.md. One provider sees that content. Never three.
This is the design point people miss when they bolt fan-out onto an existing pipeline: mode is a function of sensitivity, not just of stage. A jury ships every input to every vendor on the panel, so your privacy posture becomes the union of all their terms. Route around that at the orchestrator, where it cannot be forgotten. Not in a prompt, where it can.
The eval loop I actually have
Full disclosure: there is no golden set. No benchmark harness, no held-out labeled transcripts, no dashboard scoring "merge quality" against reference answers. I've read enough eval posts to know I'm supposed to be embarrassed by this. I'm not.
What exists instead is operational. Three mechanisms running, one built and never wired:
- The audit trail.
pipeline_runs, one of 12 ORM tables: stage, model, prompt hash, tokens, cost, latency, success, per recording. It feeds the dashboard's costs page and 19 Prometheus metrics on Grafana, sliced by model, stage, and mode. - Corrections retune the ASR. Approved glossary entries sync nightly (03:30 cron) into a GCP Speech PhraseSet, capped at 1,000 phrases (
src/knowledge/phraseset_sync.py). Terms the pipeline keeps fixing become hints that stop Chirp from mangling them next week. Approved by whom? Me, by hand, in the dashboard. A feedback loop with a human hinge. - Engine agreement scoring. Chirp versus WhisperX (wired in, currently disabled as an engine), per-segment
SequenceMatchersimilarity plus word-set agreement, aggregated and stored as its ownpipeline_runsrow. - Correction mining with auto-promotion. Built, tested, unwired.
src/enhancement/correction_logger.pycan log every cleanup and domain edit, GROUP-BY them, and auto-promote any domain correction seen 10 or more times, tagged "Auto-promoted: corrected from '' (x)". Nothing in production calls it. The cleanup stage counts its regex edits and drops them; the domain stage logs nothing. The fully closed loop (transcribe, enhance, log, promote, sync, transcribe better) is real code sitting one wiring PR short of running. I know that because I grepped for callers while writing this post.
Plus one blunt guardrail: a $5.00 per-recording cost alert.
Name for the family: exhaust evals. Evaluation built from the production exhaust you already emit, instead of the benchmark suite you'll never maintain. Exhaust evals won't tell you a merged transcript scored 0.87. They tell you what every call cost, whether your two ASR engines drift apart, and the moment a recording burns $5. For a single-user pipeline, that's not a compromise. That's the correctly sized tool.
The kill
On 2026-05-11 every stage flipped to a single OpenAI gpt-5.5 call at reasoning_effort: xhigh. All best_of_3 modes dropped. No more Claude and Gemini fan-out, no more judge. From docs/decisions.md:
Per-recording LLM cost is up several-fold versus the old single-Claude / best-of-3 mix. [...] No more cross-model arbitration on
cleanupanddomain; we're betting that GPT-5.5's xhigh first take beats the old best-of-3 + judge by enough to be worth the cost and latency. If not, revert by editing one YAML file.
Enhancement latency went from about 1 minute to 5-15 minutes per recording. Cost went up several-fold: gpt-5.5 bills $5/$30 per million tokens, against a jury where two of three members billed under a dollar. That latency only works because the same day's scale-out took the workers from one process at max_jobs=3 to six replicas at max_jobs=1. Same day, the database also moved off a Cloud SQL instance that had been deleted five days before anyone noticed the failing polls, and 3,099 historical recordings were accepted as lost. Playground, not product. The eval loop started fresh; the code that feeds it didn't change.
Be precise about what this decision was: a judgment call informed by audit data, not a formal A/B. The pipeline_runs rows said exactly what the jury cost, per stage, per model, per merge. Reading gpt-5.5's xhigh first takes next to the old merged outputs said what the jury bought. The gap no longer justified four calls and an arbitration step. One very strong take beat three good takes plus a merge. That's the honest verdict, and it's dated, and it's revisable.
Multipliers multiply your mistakes
One more scar the jury left, from a week before the kill. Zoom returns an audio-only file plus two MP4 renderings per meeting. My ingest treated each as a separate recording. Three transcriptions, three best-of-3 passes, three Drive uploads, three Slack pings, per meeting. The changelog line for the v0.3.1 dedup fix: "~62% of recent Zoom LLM/transcription spend was wasted on duplicates."
Best-of-3 didn't cause that bug. It tripled its blast radius. Every per-recording multiplier you add multiplies your bugs at exactly the same rate as your quality. Price that in before you fan out.
Parked machinery
Now the part I'd defend in any design review. The losing architecture was not deleted. The Claude and Gemini paths, the judge, the prompts: all still on main. Runtime config lives in the database (app_config); the YAML is only the seed. The ops dashboard renders "Best of 3" versus "Single" per stage and validates model names against PROVIDER_MODELS. Re-enabling best-of-3 is a dashboard PUT. Not a deploy. Not a revert PR. One HTTP call.
Parked machinery: when an architecture loses a judgment call, park it behind config with its tests green instead of deleting it. Deletion is for code that was wrong. Parking is for code that lost a bet whose odds can change. These odds move constantly. Prices shift, new models ship, and the audit rows will show the drift the week it happens.
Parking has a tax, and the repo pays it in tests: 732 test functions across 67 files, with 14,120 lines of test code against 12,218 lines of src. The judge's eight tests run on every commit whether the judge is employed or not. That's the whole discipline. Parked machinery with rotting tests is a landmine with a tarp on it.
What to take
Four things, portable beyond transcripts:
- Fan out where models disagree, not everywhere. 2 of 8 stages earned a jury: the subjective-rewrite stages, nothing else.
- Make the judge merge, anchored to the source. No-add, no-remove, original in the prompt. Build the failure ladder first: drop failed candidates, skip the judge for a lone survivor, fail loudly at zero.
- Sensitivity overrides mode, at the orchestrator. If restricted content must never fan out, make it routing logic, not a prompt line.
- Audit every call, and park what loses. One row per LLM call is what made the kill a readable decision and the revert one config write.
The judge is not dead
Seven weeks of running a jury taught me less about model quality than about decision design. Strong opinions are cheap. Reversible strong opinions are engineering. Make the bet cheap to lose and you can afford to make it early.
The judge sits on main today. 99 lines of Python, 44 lines of prompt, tests green, unemployed. Every recording that flows through the pipeline still writes its audit rows, and the rows still get read. The day gpt-5.5's first take stops being worth its premium, I won't need a migration, a post-mortem, or a meeting. I'll need one PUT.