Most write-ups show only the clean final number. This one shows the wreckage behind it. We keep this log for two reasons: it is how real measurement actually goes, and a result you can only trust if you can see what was done to earn it. Several of our early, exciting numbers turned out to be artifacts of bugs, not findings — we caught them, fixed them, and re-measured. That process is documented here in full.
Our first orchestration attempt — route each question to the single model judged best at its domain — scored below the best single model. With only four calibration questions per domain, the router mis-picked the specialist in half the domains. Lesson kept: naive selection is fragile to calibration noise; aggregation (voting) doesn't depend on one brittle decision. Not a defect — a real, useful negative result.
Our earliest panels included a 550B model. On the free tier it was effectively unusable: empty/whitespace responses, extremely slow, only a handful of calls per day. Fix: dropped it for smaller, reliably-served models. Update (this session): re-benched cleanly via NVIDIA's direct free endpoint, it completed 94% of the 196 items and scored 87.0% on those — just below our three-model selection ceiling (90.8), well clear of the best single member (82.1). So the giant is genuinely brilliant, not weak — but coverage-limited (~6% time out; an earlier 84%-coverage run read an inflated 90.9). Lesson: availability beats raw size for a panel member — but as a one-shot soloist it reframes the goal (/research §5.2): a single free model now sits near the ceiling a weaker trio is coordinated to reach.
When a model was rate-limited or timed out, the call returned nothing — and the harness counted that blank as a wrong answer. This quietly depressed scores: in one run a strong model (gemma) looked like 73% when its true score was 80.6%, which in turn made a coordination "lift" look far bigger than it was. Fix: a completeness guard (never finalise until every model has answered every question), a larger token budget, retry-on-failure, and never caching an empty response.
When a model didn't emit the instructed "ANSWER: X", our parser fell back to grabbing the last stray letter A–J anywhere in the text — which, on a truncated response, is usually a letter from the middle of a calculation. Garbage. The damage was stark: Nemotron-Super scored 88.8% on the questions where it actually wrote "ANSWER:", but 15.5% on the truncated ones where we mis-read it — a 73-point gap that is pure extraction failure, not difficulty. Its reported 62.8% was a severe undercount. Fix: strict parsing — trust only an explicit final answer; anything else is treated as "no answer" and the model is re-asked with room to finish, rather than having a wrong letter invented for it.
Our first headline lifts were products of bugs 3 and 4 stacked: the strongest model's score was being pushed down, which inflated the apparent gain from voting. Once measured cleanly, those numbers moved — and the +7.6 vanished. Status (resolved): the clean re-measurement is in. On the fixed pipeline, voting does not beat the best single model (−1.9), and the +7.6 was purely an artifact of bugs 3–4. The real, robust finding is the selection ceiling: +8.8 above the best single (and above the frontier target) — the diversity needed to win is genuinely there; selection simply cannot capture it. The paper now carries only these clean figures.
We cache every model answer so nothing is paid for twice. The cache key used Python's built-in hash(), which is randomly salted per process unless pinned — so a run that forgot to pin it wrote answers under keys no later run could find, orphaning them. Fix: pin the hash seed across every runner so the cache is consistent and fully reusable.
The free tier enforces per-minute and per-day caps; an un-paced burst of requests trips a 429 almost immediately. And even at temperature 0, the free providers are not perfectly deterministic — the same prompt can return different text on different calls. Handling: pace requests, make every run resumable and resilient to mid-run death, and freeze each answer in the cache so the analysis is reproducible.
The first real test of synthesis (Tier 2): have one model read all three panel drafts and adjudicate the answer. With a weak judge (gpt-oss-20b, our lowest-scoring member at 69.9%), fusion produced no lift — on the disagreement questions it could resolve it landed exactly as many right as a plain majority vote (23 correct, identical to the vote). Lesson kept: a weak aggregator cannot capture the panel's latent headroom — it defaults to echoing the majority. Capturing the +8.8 selection ceiling needs a capable judge (or a richer, multi-round process); the weak-judge null is the first evidence that judge quality — not just the fusion idea — is the lever.
Critique fusion feeds the judge every member's full draft, reasoning included. Three verbose drafts run 5,000–9,000 tokens — and on that much input the small free judges silently returned empty: gpt-oss-20b completed only 41 of 52 fuse calls, gemma just 4 of 52. (Not a rate limit — short calls to the same models succeeded fine.) Fix: trim each draft to its head + tail (the problem framing and the conclusion) before handing it to the judge, keeping the fuse prompt under ~1k tokens. The judge still sees each model's setup and final answer; it just no longer drowns in the middle of three chains of thought.
The most uncomfortable finding came late: re-running the identical model over the identical 196 questions returned 73.0%, then 66.3%, then 80.6% on separate runs. A ±7-point swing with nothing changed but the wall clock — larger than any coordination lift we were trying to claim. The reflex is to shrug "LLMs are non-deterministic," but most of that swing was contamination, not nature. We decomposed it into four sources:
| Source of variance | What it is | Status |
|---|---|---|
| Question sampling | which items are drawn from MMLU-Pro | Controlled — frozen seed; identical across runs |
| Provider routing / quant | OpenRouter silently serves one "model" from different backends & quantisations per call — you measure a mixture, not a model | The big culprit → now pinned |
| Parsing artifacts | strict-vs-loose extraction (bugs 3–4) scoring the same reply differently | Fixed — strict + reliability split |
| Decoding non-determinism | even at temperature 0, kernels & batching aren't bit-exact | Irreducible — but small (~±1–2), and now measured |
Fix: pin the provider (allow_fallbacks:false — which also blocks any silent fall-through to a paid endpoint), a stable content-hash cache, and run-stamped repeat runs so a re-run makes genuinely fresh calls. Kill the three controllable sources and the residual collapses to the ±1–2 that is honestly just the model. To quantify even that, the protocol now runs the reference model 30–50 times and reports the run-to-run standard deviation — and a lift only counts if it clears that band, not merely zero. Lesson kept: "the model is noisy" is usually an excuse for un-pinned infrastructure. Name the variance before you tolerate it.
Bug 4 taught the specific case; this is the principle we now build on. A model that truncates 28% of the time before stating an answer is not "58% accurate" — it is ~87% accurate on the 72% it finishes. Collapsing those into one figure punishes verbose thinkers and is simply the wrong measurement. Every model now carries two separate numbers: coverage (% of questions answered within budget = reliability) and accuracy-among-answered (= intelligence). Anything below 90% coverage is flagged and may not silently join a panel — and the flag then forces a choice: fix the reliability, or drop the model. Our strongest-on-paper reasoner (Nemotron-Super, ~28% raw truncation) was the motivating case; the strict-parse + re-ask fix (§4) recovered its coverage, which is exactly what lets it serve as the trio's strongest member today. Where the failure is intrinsic instead (Nemotron-Nano's repetition collapse, §12), the same flag drops the model. Capability you cannot reliably access is not capability you can ship.
Coverage (§11) actually hides two failures that need different responses. Type A — reachability: can we reach the model and get any response on a free tier at all? Type B — answer-usability: when it does respond, is the answer extractable? They're independent, and conflating them mis-attributes the cause. Alibaba's qwen3-next-80b is a pure Type-A failure — its only free route is through Venice's tiny shared free tier, which rate-limits us to zero answers in a burst; nothing wrong with the model, we simply can't reach it. Nemotron-Super (§4) is a Type-B failure — always reachable, but truncates before stating an answer. A model has to clear both bars to ship.
And Type B has more than one mechanism. Nemotron-Super runs out of token budget mid-reasoning (truncation — fixed by §4's strict-parse + re-ask). But NVIDIA's smaller Nemotron-Nano-30B fails a different way: degenerate repetition collapse. On the hardest ~13% of questions it falls into a loop — "3 days + 17 hours = 3 days + 17 hours = …" repeated for 38,000+ characters until it hits the token cap, never reaching an answer. It is 100% reachable (Type A) but only 87% usable (Type B). What we tried: repetition_penalty 1.2 recovered 1 of 5 failures; 1.5 recovered 0 of 6 (a stronger penalty just makes a shorter loop, it doesn't break it); a forced "reply with only the letter, no reasoning" was ignored — the model reasons anyway. Lesson: extraction discipline rescues truncation, but repetition collapse is model-intrinsic — no sampling or prompt trick we tried fixes it. For those models the honest move isn't a fix, it's the flag: measure the 87%, surface it, and prefer a better-behaved member for the live panel rather than pretend the failure isn't there.
The whole premise — coordinate free models to reach a higher tier — assumes a stable free-access layer. In practice it is the most fragile part of the system, and it fails three independent ways. (A) Daily quota walls: Google AI Studio serves the frontier Geminis for free, but the free tier exhausts after ~25–30 calls/day (RESOURCE_EXHAUSTED) — not enough to finish a single 196-question bench, let alone run a live panel. (B) Silent free→paid churn: OpenRouter quietly moved deepseek-r1, qwen3-235b and deepseek-v3 off their :free routes — the slugs now 404 with "unavailable for free." A model that's a panel member one week is paid the next, with no notice; the free-claims need continuous re-probing (and the engine's own model cascade was found still 404-ing on two of them every call). (C) Datacenter-IP blocks: Groq — the broadest, fastest free tier — returns 403 "Access denied, check your network settings" to our server outright, because its egress is a hosting-provider IP that Groq blocks for abuse control. A residential connection would work; our infrastructure cannot reach it.
What we do about it: treat free access as an unreliable, churning resource, not a fixed list — re-probe every model's free status on a schedule (the table now carries the access path per row), spread benching across multiple providers and keys so no single daily cap is load-bearing, and prefer providers with real headroom (Cerebras at ~1M tokens/day, GitHub Models' generous request budget) over thin-but-frontier ones (AI Studio) for anything that needs volume. Lesson kept: "it's free" is not "it's available." The access layer is a first-class engineering problem, not a footnote — a coordination system meant to run continuously has to be built to rotate around it.
Testing a strong judge (Nemotron-Super) that selects the most trustworthy draft rather than re-solving, the first run scored 62.2% — below the weakest panel member (75%). A composite scoring beneath its own floor is almost always a measurement fault, so we stopped and read the raw outputs before reporting anything. The fault was ours: we had labelled the three drafts Expert A / B / C, but the answer options are also A–J — so the literal reasoner, told to "report the letter," returned the expert label (A/B/C) instead of the option letter on 17% of items, each scored wrong. Fix: relabel the experts 1/2/3 and ask explicitly for the option letter; corrected, the judge scored ~78, in line with the other selecting judge. Lesson kept: sanity-check every composite against the single-model floor before believing it, and never let a label space collide with the answer space. Had we reported the 62.2 we'd have "discovered" that strong judges are terrible selectors — the truth is that selection ≈ chance among the drafts regardless of judge strength (/research §5.2).
Solo accuracy on our 196-question stratified MMLU-Pro mini-bench (14 questions × 14 disciplines), shown beside each model's published full-benchmark MMLU-Pro score. The two should agree within a few points — that agreement is itself the validation that a 196-question sample faithfully estimates the full ~12,000-question benchmark. Mini-scores marked re-measuring were corrupted by the bugs above and are being re-run clean; the published column is being verified from primary sources (we are not eyeballing it, given what this page documents).
| Model | Lab | Full MMLU-Pro (published) | Our 196 mini-bench | Issues observed |
|---|---|---|---|---|
| Opus 4.8 (frontier target, ~1 gen up) | Anthropic | ~88–90 | 86.2% | Reference "target" bar — answered blind, in snap mode (no max-reasoning), same conditions as the panel. Not a ceiling; the gap to close. |
| gemma-4-31b | ~85.2* | 80.6% | None — concise, always emits a clean final answer. Our reliable anchor. | |
| nemotron-3-super-120b | NVIDIA | ~83.7* | 82.1% (was mis-measured 62.8 — the extraction bug) | Verbose reasoner — truncates before stating an answer; slow; rate-limited. Actually our STRONGEST member once measured cleanly. |
| gpt-oss-20b | OpenAI | 73.6 | 69.9% | Reliable and fast; occasional empty response. |
| gpt-oss-120b | OpenAI | 80.8 | 75.0% | Reliable; moderate speed. |
| nemotron-3-ultra-550b | NVIDIA | n/a | 87.0% (94% coverage) | The "unreliable giant" (§2): re-benched brilliant on NVIDIA-direct, near the selection ceiling — but ~6% time out and it's too slow for a live panel. Barred on coverage, not intelligence. |
| qwen3-next-80b | Alibaba | 80.6 | pending | Rate-limited on burst testing. |
| hermes-3-llama-405b | Nous / Meta | ~54.1 | pending | Rate-limited on burst testing. |
| llama-3.3-70b | Meta | ~68.9* | pending | Rate-limited on burst testing. |
* Published figures from secondary sources (llm-stats, vendor model cards/blogs) and reflect each model's OPTIMAL configuration — often maximum reasoning effort. Our mini-bench uses ONE standardized simple prompt for every model, so our absolute numbers run lower, and the gap is largest for reasoning models (gpt-oss, nemotron) that we don't run in their reasoning mode. The comparison therefore validates relative RANKING and exposes config-sensitivity — it is not an absolute-equivalence check. Several published figures (e.g. 85–90 for small open models) look high for MMLU-Pro and HAVE now been primary-source-checked — that pass corrected gpt-oss-120b (90→80.8), gpt-oss-20b (85.3→73.6), hermes-3-405b (73→54.1) and qwen3-next (81.6→80.6); the sourced, banded master table lives at /research. Figures still marked * (gemma, llama, nemotron-super) remain secondary-source.
The corrected view: only one number was ever fully trustworthy (gemma's 80.6%), because it was the one model that never tripped the bugs. Everything else is being re-measured on clean extraction. The method was always sound — stratified sampling, diverse panels, bootstrap confidence intervals — but measurement discipline, not orchestration cleverness, was the hard part. If there is a single transferable lesson, it is this: benchmarking reasoning models requires strict answer-extraction and enough room to finish, or you will quietly undercount your best thinkers.