Skip to content

Benchmark results

Generated on 2026-11-19 from run v3-headline. The verdicts below are the verdict engine's text, unedited; how to read them is in BENCHMARKING.md.

The full report with its charts is benchmark-report.html.

2 job(s) halted on their first attempt (a call timed out or the panel had no quorum) and were run again, with longer timeouts where it was a timeout: solo-opus (2). Only the last attempt is scored; the first attempt's cost is in the totals. A halted job scores 0, so without the retry these arms would be understated by the harness's limit, not by their answers. Some prices were unknown for solo-luna, solo-sol, fusion-default, fusion-best: its costs are a lower bound. Items replayed from the response cache are left out of the latency figures (solo-luna, fusion-default, fusion-best): their wall time is not a measurement. 1 repeat(s) per task: variance across repeats is poorly estimated. Frontend and performance scores measure what the answer builds (tests, timings, screenshots); they do not judge whether it matches the author's intent.

Headline

Each arm against solo-opus, on all tasks. ✓ yes, ✗ no, ? inconclusive, ⊘ blocked. The reasons are listed below the table.

Arm Paired tasks Cheaper (per solved task) Faster Not worse Better
solo-haiku 47 ✓ yes ✓ yes ? inconclusive ? inconclusive
solo-luna 47 ✓ yes ✓ yes ? inconclusive ? inconclusive
solo-sol 47 ✓ yes ? inconclusive ✗ no ✗ no
fusion-default 47 ✓ yes ✓ yes ? inconclusive ? inconclusive
fusion-best 47 ✓ yes ✓ yes ? inconclusive ? inconclusive
  • solo-haiku vs solo-opus on all (47 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.01 [0.01, 0.02] (below 1 is better): the whole interval is below 1.
  • solo-haiku vs solo-opus on all (47 tasks): faster: yes. seconds per task ratio 0.32 [0.29, 0.35] (below 1 is better): the whole interval is below 1.
  • solo-haiku vs solo-opus on all (47 tasks): not worse: inconclusive. quality difference +0.053 [-0.052, +0.147] over 47 tasks: the interval straddles the -0.03 margin.
  • solo-haiku vs solo-opus on all (47 tasks): better: inconclusive. quality difference +0.053 [-0.052, +0.147] over 47 tasks: the interval includes 0.
  • solo-luna vs solo-opus on all (47 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.01 [0.01, 0.02] (below 1 is better): the whole interval is below 1.
  • solo-luna vs solo-opus on all (47 tasks): faster: yes. seconds per task ratio 0.48 [0.44, 0.52] (below 1 is better): the whole interval is below 1.
  • solo-luna vs solo-opus on all (47 tasks): not worse: inconclusive. quality difference -0.053 [-0.144, +0.047] over 47 tasks: the interval straddles the -0.03 margin.
  • solo-luna vs solo-opus on all (47 tasks): better: inconclusive. quality difference -0.053 [-0.144, +0.047] over 47 tasks: the interval includes 0.
  • solo-sol vs solo-opus on all (47 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.37 [0.25, 0.53] (below 1 is better): the whole interval is below 1.
  • solo-sol vs solo-opus on all (47 tasks): faster: inconclusive. seconds per task ratio 0.96 [0.89, 1.03] (below 1 is better): the interval includes 1.
  • solo-sol vs solo-opus on all (47 tasks): not worse: no. quality difference -0.112 [-0.191, -0.034] over 47 tasks: the upper bound is below -0.03: it loses more than the margin.
  • solo-sol vs solo-opus on all (47 tasks): better: no. quality difference -0.112 [-0.191, -0.034] over 47 tasks: the interval is at or below 0.
  • fusion-default vs solo-opus on all (47 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.11 [0.05, 0.23] (below 1 is better): the whole interval is below 1.
  • fusion-default vs solo-opus on all (47 tasks): faster: yes. seconds per task ratio 0.69 [0.59, 0.78] (below 1 is better): the whole interval is below 1.
  • fusion-default vs solo-opus on all (47 tasks): not worse: inconclusive. quality difference +0.022 [-0.072, +0.117] over 47 tasks: the interval straddles the -0.03 margin.
  • fusion-default vs solo-opus on all (47 tasks): better: inconclusive. quality difference +0.022 [-0.072, +0.117] over 47 tasks: the interval includes 0.
  • fusion-best vs solo-opus on all (47 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.06 [0.04, 0.09] (below 1 is better): the whole interval is below 1.
  • fusion-best vs solo-opus on all (47 tasks): faster: yes. seconds per task ratio 0.50 [0.47, 0.53] (below 1 is better): the whole interval is below 1.
  • fusion-best vs solo-opus on all (47 tasks): not worse: inconclusive. quality difference +0.009 [-0.079, +0.098] over 47 tasks: the interval straddles the -0.03 margin.
  • fusion-best vs solo-opus on all (47 tasks): better: inconclusive. quality difference +0.009 [-0.079, +0.098] over 47 tasks: the interval includes 0.

Arms

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90 Frontier
solo-haiku 47 0.643 [0.548, 0.731] 57% $0.0009 0.0016 6.7s 8.8s $ s
solo-luna 47 0.537 [0.432, 0.648] 49% $0.0007 0.0014 9.7s 14.0s $
solo-opus 47 0.590 [0.500, 0.679] 43% $0.0544 0.1279 20.7s 29.9s -
solo-sol 47 0.478 [0.380, 0.580] 40% $0.0191 0.0472 20.7s 28.9s -
fusion-default 47 0.612 [0.519, 0.702] 53% $0.0073 0.0137 13.2s 23.6s -
fusion-best 47 0.599 [0.492, 0.705] 53% $0.0042 0.0078 11.2s 13.7s -

Frontier: $ = on the cost-quality frontier (no arm is cheaper per task and at least as good), s = on the latency-quality frontier (median seconds).

Efficiency and speed

Arm Out tok/s (effective) Tokens / solved Quality / $ Quality / min Calls Errors
solo-haiku 208 6,805 707.7 5.834 1.0 0
solo-luna 103 5,052 803.5 3.342 1.1 0
solo-opus 110 11,327 10.8 1.690 1.0 0
solo-sol 81 7,420 25.1 1.430 1.6 0
fusion-default 207 17,310 83.8 2.580 3.1 0
fusion-best 237 17,434 144.1 3.202 4.0 0

Streamed speed per model:

Arm Model Decode tok/s TTFT ms
solo-haiku claude-haiku 266 1276
solo-luna gpt-luna 199 4094
solo-opus claude-opus 134 3356
solo-sol claude-sonnet 360 15577
solo-sol gpt-sol 104 3883
fusion-default claude-haiku 248 1046
fusion-default gemini-flash 144 3959
fusion-default gpt-luna 206 4178
fusion-best claude-haiku 259 1302
fusion-best gemini-flash 161 2816
fusion-best gpt-luna 215 3632

Cost of measuring

Scoring and judging cost and time, kept apart: never part of an arm's cost or latency.

Arm Eval $ total Eval $ / item Eval s / item
solo-haiku $0.0653 $0.0014 4.3s
solo-luna $0.0603 $0.0013 4.2s
solo-opus $0.0765 $0.0016 4.7s
solo-sol $0.0707 $0.0015 4.0s
fusion-default $0.0606 $0.0013 3.3s
fusion-best $0.0595 $0.0013 3.2s

Variance across repeats

Arm Mean sd of quality across repeats Tasks whose solved flips
solo-haiku - -
solo-luna - -
solo-opus - -
solo-sol - -
fusion-default - -
fusion-best - -

Comparisons against the baseline

solo-haiku vs solo-opus

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 47 +0.053 [-0.052, +0.147] 23/12/12 0.090 +0.16 0.01 [0.01, 0.02] 0.32 [0.29, 0.35] ✓ ✓ ? ?
architecture 10 +0.130 [-0.050, +0.310] 7/0/3 0.344 +0.41 0.00 0.34 [0.31, 0.36] ? ✓ ? ?
code_review 10 +0.179 [+0.060, +0.313] 8/1/1 0.039 +0.85 0.01 [0.00, 0.02] 0.34 [0.29, 0.40] ✓ ✓ ✓ ✓
coding 3 -0.074 [-0.222, +0.000] 0/2/1 1.000 -0.58 0.03 [0.02, 0.07] 0.38 [0.25, 0.58] ? ? ? ?
debugging 10 +0.000 [-0.084, +0.084] 2/6/2 1.000 +0.00 0.02 [0.01, 0.02] 0.32 [0.28, 0.38] ✓ ✓ ? ?
frontend 2 -0.464 [-0.929, +0.000] 0/1/1 1.000 -0.71 - 0.27 [0.25, 0.29] ? ? ? ?
performance 2 +0.016 [+0.004, +0.029] 1/1/0 1.000 +0.91 0.01 [0.01, 0.03] 0.26 [0.11, 0.58] ? ? ? ?
planning 10 +0.050 [-0.261, +0.353] 5/1/4 1.000 +0.10 0.01 [0.00, 0.03] 0.29 [0.26, 0.33] ✓ ✓ ? ?
  • solo-haiku vs solo-opus on architecture (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.00; too few tasks for an interval.
  • solo-haiku vs solo-opus on architecture (10 tasks): faster: yes. seconds per task ratio 0.34 [0.31, 0.36] (below 1 is better): the whole interval is below 1.
  • solo-haiku vs solo-opus on architecture (10 tasks): not worse: inconclusive. quality difference +0.130 [-0.050, +0.310] over 10 tasks: the interval straddles the -0.03 margin.
  • solo-haiku vs solo-opus on architecture (10 tasks): better: inconclusive. quality difference +0.130 [-0.050, +0.310] over 10 tasks: the interval includes 0.
  • solo-haiku vs solo-opus on code_review (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.01 [0.00, 0.02] (below 1 is better): the whole interval is below 1.
  • solo-haiku vs solo-opus on code_review (10 tasks): faster: yes. seconds per task ratio 0.34 [0.29, 0.40] (below 1 is better): the whole interval is below 1.
  • solo-haiku vs solo-opus on code_review (10 tasks): not worse: yes. quality difference +0.179 [+0.060, +0.313] over 10 tasks: the lower bound is above -0.03, the non-inferiority margin.
  • solo-haiku vs solo-opus on code_review (10 tasks): better: yes. quality difference +0.179 [+0.060, +0.313] over 10 tasks: the interval is above 0.
  • coding (3 tasks): only 3 task(s) answered by both arms; at least 10 are needed to claim anything.
  • solo-haiku vs solo-opus on debugging (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.02 [0.01, 0.02] (below 1 is better): the whole interval is below 1.
  • solo-haiku vs solo-opus on debugging (10 tasks): faster: yes. seconds per task ratio 0.32 [0.28, 0.38] (below 1 is better): the whole interval is below 1.
  • solo-haiku vs solo-opus on debugging (10 tasks): not worse: inconclusive. quality difference +0.000 [-0.084, +0.084] over 10 tasks: the interval straddles the -0.03 margin.
  • solo-haiku vs solo-opus on debugging (10 tasks): better: inconclusive. quality difference +0.000 [-0.084, +0.084] over 10 tasks: the interval includes 0.
  • frontend (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • solo-haiku vs solo-opus on planning (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.01 [0.00, 0.03] (below 1 is better): the whole interval is below 1.
  • solo-haiku vs solo-opus on planning (10 tasks): faster: yes. seconds per task ratio 0.29 [0.26, 0.33] (below 1 is better): the whole interval is below 1.
  • solo-haiku vs solo-opus on planning (10 tasks): not worse: inconclusive. quality difference +0.050 [-0.261, +0.353] over 10 tasks: the interval straddles the -0.03 margin.
  • solo-haiku vs solo-opus on planning (10 tasks): better: inconclusive. quality difference +0.050 [-0.261, +0.353] over 10 tasks: the interval includes 0.

solo-luna vs solo-opus

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 47 -0.053 [-0.144, +0.047] 12/14/21 0.163 -0.16 0.01 [0.01, 0.02] 0.48 [0.44, 0.52] ✓ ✓ ? ?
architecture 10 -0.123 [-0.250, -0.017] 1/3/6 0.125 -0.65 0.01 0.45 [0.40, 0.52] ? ✓ ? ✗
code_review 10 +0.271 [+0.139, +0.404] 7/3/0 0.016 +1.19 0.00 [0.00, 0.01] 0.39 [0.35, 0.44] ✓ ✓ ✓ ✓
coding 3 +0.000 [+0.000, +0.000] 0/3/0 - - 0.01 [0.01, 0.01] 0.40 [0.36, 0.44] ? ? ? ?
debugging 10 -0.060 [-0.286, +0.191] 2/2/6 0.289 -0.15 0.01 [0.01, 0.02] 0.60 [0.52, 0.71] ✓ ✓ ? ?
frontend 2 -0.464 [-0.929, +0.000] 0/1/1 1.000 -0.71 - - ? ? ? ?
performance 2 -0.005 [-0.007, -0.004] 0/2/0 - -2.12 0.01 [0.01, 0.01] 0.32 [0.27, 0.45] ? ? ? ?
planning 10 -0.240 [-0.393, -0.082] 2/0/8 0.109 -0.88 - 0.50 [0.42, 0.57] ? ✓ ✗ ✗
  • solo-luna vs solo-opus on architecture (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.01; too few tasks for an interval.
  • solo-luna vs solo-opus on architecture (10 tasks): faster: yes. seconds per task ratio 0.45 [0.40, 0.52] (below 1 is better): the whole interval is below 1.
  • solo-luna vs solo-opus on architecture (10 tasks): not worse: inconclusive. quality difference -0.123 [-0.250, -0.017] over 10 tasks: the interval straddles the -0.03 margin.
  • solo-luna vs solo-opus on architecture (10 tasks): better: no. quality difference -0.123 [-0.250, -0.017] over 10 tasks: the interval is at or below 0.
  • solo-luna vs solo-opus on code_review (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.00 [0.00, 0.01] (below 1 is better): the whole interval is below 1.
  • solo-luna vs solo-opus on code_review (10 tasks): faster: yes. seconds per task ratio 0.39 [0.35, 0.44] (below 1 is better): the whole interval is below 1.
  • solo-luna vs solo-opus on code_review (10 tasks): not worse: yes. quality difference +0.271 [+0.139, +0.404] over 10 tasks: the lower bound is above -0.03, the non-inferiority margin.
  • solo-luna vs solo-opus on code_review (10 tasks): better: yes. quality difference +0.271 [+0.139, +0.404] over 10 tasks: the interval is above 0.
  • coding (3 tasks): only 3 task(s) answered by both arms; at least 10 are needed to claim anything.
  • solo-luna vs solo-opus on debugging (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.01 [0.01, 0.02] (below 1 is better): the whole interval is below 1.
  • solo-luna vs solo-opus on debugging (10 tasks): faster: yes. seconds per task ratio 0.60 [0.52, 0.71] (below 1 is better): the whole interval is below 1.
  • solo-luna vs solo-opus on debugging (10 tasks): not worse: inconclusive. quality difference -0.060 [-0.286, +0.191] over 10 tasks: the interval straddles the -0.03 margin.
  • solo-luna vs solo-opus on debugging (10 tasks): better: inconclusive. quality difference -0.060 [-0.286, +0.191] over 10 tasks: the interval includes 0.
  • frontend (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • solo-luna vs solo-opus on planning (10 tasks): cheaper (per solved task): inconclusive. an arm solved no task, so its cost per solved task is undefined.
  • solo-luna vs solo-opus on planning (10 tasks): faster: yes. seconds per task ratio 0.50 [0.42, 0.57] (below 1 is better): the whole interval is below 1.
  • solo-luna vs solo-opus on planning (10 tasks): not worse: no. quality difference -0.240 [-0.393, -0.082] over 10 tasks: the upper bound is below -0.03: it loses more than the margin.
  • solo-luna vs solo-opus on planning (10 tasks): better: no. quality difference -0.240 [-0.393, -0.082] over 10 tasks: the interval is at or below 0.

solo-sol vs solo-opus

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 47 -0.112 [-0.191, -0.034] 8/11/28 0.001 -0.41 0.37 [0.25, 0.53] 0.96 [0.89, 1.03] ✓ ? ✗ ✗
architecture 10 -0.160 [-0.307, +0.053] 1/0/9 0.021 -0.52 0.40 1.04 [0.93, 1.14] ? ? ? ?
code_review 10 -0.136 [-0.263, -0.017] 2/1/7 0.180 -0.65 1.08 0.93 [0.79, 1.07] ? ? ? ✗
coding 3 +0.000 [+0.000, +0.000] 0/3/0 - - 0.30 [0.30, 0.32] 1.03 [0.88, 1.21] ? ? ? ?
debugging 10 -0.046 [-0.171, +0.095] 2/3/5 0.453 -0.21 0.19 [0.11, 0.26] 0.66 [0.58, 0.77] ✓ ✓ ? ?
frontend 2 -0.005 [-0.010, +0.000] 0/2/0 - -0.71 0.32 0.91 [0.79, 1.04] ? ? ? ?
performance 2 -0.034 [-0.046, -0.021] 0/0/2 0.500 -1.92 0.45 [0.42, 0.49] 1.17 [1.00, 1.53] ? ? ? ?
planning 10 -0.178 [-0.433, +0.046] 3/2/5 0.727 -0.43 0.59 1.08 [0.96, 1.20] ? ? ? ?
  • solo-sol vs solo-opus on architecture (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.40; too few tasks for an interval.
  • solo-sol vs solo-opus on architecture (10 tasks): faster: inconclusive. seconds per task ratio 1.04 [0.93, 1.14] (below 1 is better): the interval includes 1.
  • solo-sol vs solo-opus on architecture (10 tasks): not worse: inconclusive. quality difference -0.160 [-0.307, +0.053] over 10 tasks: the interval straddles the -0.03 margin.
  • solo-sol vs solo-opus on architecture (10 tasks): better: inconclusive. quality difference -0.160 [-0.307, +0.053] over 10 tasks: the interval includes 0.
  • solo-sol vs solo-opus on code_review (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 1.08; too few tasks for an interval.
  • solo-sol vs solo-opus on code_review (10 tasks): faster: inconclusive. seconds per task ratio 0.93 [0.79, 1.07] (below 1 is better): the interval includes 1.
  • solo-sol vs solo-opus on code_review (10 tasks): not worse: inconclusive. quality difference -0.136 [-0.263, -0.017] over 10 tasks: the interval straddles the -0.03 margin.
  • solo-sol vs solo-opus on code_review (10 tasks): better: no. quality difference -0.136 [-0.263, -0.017] over 10 tasks: the interval is at or below 0.
  • coding (3 tasks): only 3 task(s) answered by both arms; at least 10 are needed to claim anything.
  • solo-sol vs solo-opus on debugging (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.19 [0.11, 0.26] (below 1 is better): the whole interval is below 1.
  • solo-sol vs solo-opus on debugging (10 tasks): faster: yes. seconds per task ratio 0.66 [0.58, 0.77] (below 1 is better): the whole interval is below 1.
  • solo-sol vs solo-opus on debugging (10 tasks): not worse: inconclusive. quality difference -0.046 [-0.171, +0.095] over 10 tasks: the interval straddles the -0.03 margin.
  • solo-sol vs solo-opus on debugging (10 tasks): better: inconclusive. quality difference -0.046 [-0.171, +0.095] over 10 tasks: the interval includes 0.
  • frontend (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • solo-sol vs solo-opus on planning (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.59; too few tasks for an interval.
  • solo-sol vs solo-opus on planning (10 tasks): faster: inconclusive. seconds per task ratio 1.08 [0.96, 1.20] (below 1 is better): the interval includes 1.
  • solo-sol vs solo-opus on planning (10 tasks): not worse: inconclusive. quality difference -0.178 [-0.433, +0.046] over 10 tasks: the interval straddles the -0.03 margin.
  • solo-sol vs solo-opus on planning (10 tasks): better: inconclusive. quality difference -0.178 [-0.433, +0.046] over 10 tasks: the interval includes 0.

fusion-default vs solo-opus

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 47 +0.022 [-0.072, +0.117] 18/18/11 0.265 +0.07 0.11 [0.05, 0.23] 0.69 [0.59, 0.78] ✓ ✓ ? ?
architecture 10 +0.217 [+0.057, +0.373] 8/1/1 0.039 +0.80 0.02 0.80 [0.66, 0.93] ? ✓ ✓ ✓
code_review 10 -0.019 [-0.138, +0.130] 3/3/4 1.000 -0.08 0.11 0.48 [0.35, 0.67] ? ✓ ? ?
coding 3 +0.000 [+0.000, +0.000] 0/3/0 - - 0.10 [0.01, 0.14] 0.64 [0.27, 0.91] ? ? ? ?
debugging 10 +0.056 [-0.042, +0.175] 2/7/1 1.000 +0.29 0.04 [0.02, 0.07] 0.42 [0.32, 0.54] ✓ ✓ ? ?
frontend 2 -0.464 [-0.929, +0.000] 0/1/1 1.000 -0.71 - 0.96 ? ? ? ?
performance 2 +0.007 [+0.004, +0.011] 0/2/0 - +1.41 0.06 [0.01, 0.08] 0.48 [0.23, 0.59] ? ? ? ?
planning 10 -0.057 [-0.326, +0.214] 5/1/4 1.000 -0.12 0.08 [0.02, 0.23] 0.92 [0.75, 1.13] ✓ ? ? ?
  • fusion-default vs solo-opus on architecture (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.02; too few tasks for an interval.
  • fusion-default vs solo-opus on architecture (10 tasks): faster: yes. seconds per task ratio 0.80 [0.66, 0.93] (below 1 is better): the whole interval is below 1.
  • fusion-default vs solo-opus on architecture (10 tasks): not worse: yes. quality difference +0.217 [+0.057, +0.373] over 10 tasks: the lower bound is above -0.03, the non-inferiority margin.
  • fusion-default vs solo-opus on architecture (10 tasks): better: yes. quality difference +0.217 [+0.057, +0.373] over 10 tasks: the interval is above 0.
  • fusion-default vs solo-opus on code_review (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.11; too few tasks for an interval.
  • fusion-default vs solo-opus on code_review (10 tasks): faster: yes. seconds per task ratio 0.48 [0.35, 0.67] (below 1 is better): the whole interval is below 1.
  • fusion-default vs solo-opus on code_review (10 tasks): not worse: inconclusive. quality difference -0.019 [-0.138, +0.130] over 10 tasks: the interval straddles the -0.03 margin.
  • fusion-default vs solo-opus on code_review (10 tasks): better: inconclusive. quality difference -0.019 [-0.138, +0.130] over 10 tasks: the interval includes 0.
  • coding (3 tasks): only 3 task(s) answered by both arms; at least 10 are needed to claim anything.
  • fusion-default vs solo-opus on debugging (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.04 [0.02, 0.07] (below 1 is better): the whole interval is below 1.
  • fusion-default vs solo-opus on debugging (10 tasks): faster: yes. seconds per task ratio 0.42 [0.32, 0.54] (below 1 is better): the whole interval is below 1.
  • fusion-default vs solo-opus on debugging (10 tasks): not worse: inconclusive. quality difference +0.056 [-0.042, +0.175] over 10 tasks: the interval straddles the -0.03 margin.
  • fusion-default vs solo-opus on debugging (10 tasks): better: inconclusive. quality difference +0.056 [-0.042, +0.175] over 10 tasks: the interval includes 0.
  • frontend (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • fusion-default vs solo-opus on planning (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.08 [0.02, 0.23] (below 1 is better): the whole interval is below 1.
  • fusion-default vs solo-opus on planning (10 tasks): faster: inconclusive. seconds per task ratio 0.92 [0.75, 1.13] (below 1 is better): the interval includes 1.
  • fusion-default vs solo-opus on planning (10 tasks): not worse: inconclusive. quality difference -0.057 [-0.326, +0.214] over 10 tasks: the interval straddles the -0.03 margin.
  • fusion-default vs solo-opus on planning (10 tasks): better: inconclusive. quality difference -0.057 [-0.326, +0.214] over 10 tasks: the interval includes 0.

fusion-best vs solo-opus

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 47 +0.009 [-0.079, +0.098] 17/14/16 1.000 +0.03 0.06 [0.04, 0.09] 0.50 [0.47, 0.53] ✓ ✓ ? ?
architecture 10 -0.110 [-0.290, +0.077] 3/2/5 0.727 -0.35 0.07 0.48 [0.44, 0.52] ? ✓ ? ?
code_review 10 +0.235 [+0.129, +0.338] 8/1/1 0.039 +1.24 0.02 [0.00, 0.04] 0.51 [0.45, 0.60] ✓ ✓ ✓ ✓
coding 3 +0.000 [+0.000, +0.000] 0/3/0 - - 0.08 [0.03, 0.12] 0.58 [0.52, 0.66] ? ? ? ?
debugging 10 -0.030 [-0.138, +0.069] 2/5/3 1.000 -0.17 0.11 [0.09, 0.12] 0.55 [0.41, 0.81] ✓ ✓ ? ?
frontend 2 +0.524 [+0.062, +0.987] 2/0/0 0.500 +0.80 0.02 0.54 [0.53, 0.55] ? ? ? ?
performance 2 -0.020 [-0.032, -0.007] 0/1/1 1.000 -1.11 0.03 [0.03, 0.05] 0.41 [0.33, 0.61] ? ? ? ?
planning 10 -0.155 [-0.376, +0.071] 2/2/6 0.289 -0.41 0.28 0.49 [0.45, 0.54] ? ✓ ? ?
  • fusion-best vs solo-opus on architecture (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.07; too few tasks for an interval.
  • fusion-best vs solo-opus on architecture (10 tasks): faster: yes. seconds per task ratio 0.48 [0.44, 0.52] (below 1 is better): the whole interval is below 1.
  • fusion-best vs solo-opus on architecture (10 tasks): not worse: inconclusive. quality difference -0.110 [-0.290, +0.077] over 10 tasks: the interval straddles the -0.03 margin.
  • fusion-best vs solo-opus on architecture (10 tasks): better: inconclusive. quality difference -0.110 [-0.290, +0.077] over 10 tasks: the interval includes 0.
  • fusion-best vs solo-opus on code_review (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.02 [0.00, 0.04] (below 1 is better): the whole interval is below 1.
  • fusion-best vs solo-opus on code_review (10 tasks): faster: yes. seconds per task ratio 0.51 [0.45, 0.60] (below 1 is better): the whole interval is below 1.
  • fusion-best vs solo-opus on code_review (10 tasks): not worse: yes. quality difference +0.235 [+0.129, +0.338] over 10 tasks: the lower bound is above -0.03, the non-inferiority margin.
  • fusion-best vs solo-opus on code_review (10 tasks): better: yes. quality difference +0.235 [+0.129, +0.338] over 10 tasks: the interval is above 0.
  • coding (3 tasks): only 3 task(s) answered by both arms; at least 10 are needed to claim anything.
  • fusion-best vs solo-opus on debugging (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.11 [0.09, 0.12] (below 1 is better): the whole interval is below 1.
  • fusion-best vs solo-opus on debugging (10 tasks): faster: yes. seconds per task ratio 0.55 [0.41, 0.81] (below 1 is better): the whole interval is below 1.
  • fusion-best vs solo-opus on debugging (10 tasks): not worse: inconclusive. quality difference -0.030 [-0.138, +0.069] over 10 tasks: the interval straddles the -0.03 margin.
  • fusion-best vs solo-opus on debugging (10 tasks): better: inconclusive. quality difference -0.030 [-0.138, +0.069] over 10 tasks: the interval includes 0.
  • frontend (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • fusion-best vs solo-opus on planning (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.28; too few tasks for an interval.
  • fusion-best vs solo-opus on planning (10 tasks): faster: yes. seconds per task ratio 0.49 [0.45, 0.54] (below 1 is better): the whole interval is below 1.
  • fusion-best vs solo-opus on planning (10 tasks): not worse: inconclusive. quality difference -0.155 [-0.376, +0.071] over 10 tasks: the interval straddles the -0.03 margin.
  • fusion-best vs solo-opus on planning (10 tasks): better: inconclusive. quality difference -0.155 [-0.376, +0.071] over 10 tasks: the interval includes 0.

Δ quality is the arm minus the baseline, paired by task. Ratios are the arm over the baseline (below 1 is better). W/T/L counts tasks the arm won, tied (within 0.02) or lost. Effect d is the mean paired difference over its spread across tasks.

By category

architecture

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
solo-haiku 10 0.497 [0.297, 0.700] 40% $0.0009 0.0023 7.4s 8.3s
solo-luna 10 0.243 [0.117, 0.370] 10% $0.0006 0.0062 9.6s 11.5s
solo-opus 10 0.367 [0.260, 0.460] 10% $0.0534 0.5339 20.8s 25.8s
solo-sol 10 0.207 [0.090, 0.337] 10% $0.0211 0.2114 22.9s 26.4s
fusion-default 10 0.583 [0.400, 0.767] 50% $0.0052 0.0103 18.2s 21.6s
fusion-best 10 0.257 [0.123, 0.423] 10% $0.0037 0.0366 10.5s 11.7s

code_review

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
solo-haiku 10 0.731 [0.621, 0.848] 70% $0.0006 0.0009 4.3s 5.3s
solo-luna 10 0.822 [0.722, 0.922] 100% $0.0004 0.0004 4.9s 7.2s
solo-opus 10 0.552 [0.502, 0.602] 30% $0.0368 0.1227 10.7s 18.8s
solo-sol 10 0.415 [0.330, 0.500] 10% $0.0133 0.1327 11.6s 17.0s
fusion-default 10 0.532 [0.437, 0.659] 20% $0.0027 0.0135 5.2s 10.6s
fusion-best 10 0.787 [0.660, 0.913] 90% $0.0019 0.0021 7.8s 9.5s

coding

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
solo-haiku 3 0.926 [0.778, 1.000] 67% $0.0016 0.0024 5.3s 10.8s
solo-luna 3 1.000 [1.000, 1.000] 100% $0.0008 0.0008 7.6s 7.6s
solo-opus 3 1.000 [1.000, 1.000] 100% $0.0686 0.0686 18.6s 19.7s
solo-sol 3 1.000 [1.000, 1.000] 100% $0.0208 0.0208 19.5s 22.5s
fusion-default 3 1.000 [1.000, 1.000] 100% $0.0071 0.0071 13.2s 17.8s
fusion-best 3 1.000 [1.000, 1.000] 100% $0.0055 0.0055 10.5s 11.5s

debugging

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
solo-haiku 10 0.811 [0.685, 0.930] 70% $0.0008 0.0012 6.1s 7.1s
solo-luna 10 0.751 [0.541, 0.921] 70% $0.0006 0.0008 12.4s 14.2s
solo-opus 10 0.811 [0.643, 0.958] 70% $0.0516 0.0738 17.5s 28.5s
solo-sol 10 0.765 [0.604, 0.895] 90% $0.0127 0.0142 13.3s 14.8s
fusion-default 10 0.867 [0.769, 0.958] 90% $0.0028 0.0031 6.7s 10.7s
fusion-best 10 0.781 [0.619, 0.930] 70% $0.0055 0.0079 11.0s 13.2s

frontend

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
solo-haiku 2 0.000 [0.000, 0.000] 0% $0.0012 - 8.8s 9.0s
solo-luna 2 0.000 [0.000, 0.000] 0% $0.0012 - - -
solo-opus 2 0.464 [0.000, 0.929] 50% $0.0877 0.1754 29.9s 36.1s
solo-sol 2 0.460 [0.000, 0.919] 50% $0.0279 0.0558 28.7s 31.1s
fusion-default 2 0.000 [0.000, 0.000] 0% $0.0713 - 34.6s 34.6s
fusion-best 2 0.989 [0.987, 0.990] 100% $0.0028 0.0028 16.6s 19.2s

performance

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
solo-haiku 2 0.996 [0.993, 1.000] 100% $0.0009 0.0009 4.3s 10.5s
solo-luna 2 0.975 [0.961, 0.989] 100% $0.0006 0.0006 8.2s 10.6s
solo-opus 2 0.980 [0.964, 0.996] 100% $0.0633 0.0633 18.1s 39.8s
solo-sol 2 0.946 [0.918, 0.975] 100% $0.0282 0.0282 27.8s 39.7s
fusion-default 2 0.987 [0.975, 1.000] 100% $0.0038 0.0038 4.1s 23.6s
fusion-best 2 0.961 [0.957, 0.964] 100% $0.0021 0.0021 11.0s 13.0s

planning

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
solo-haiku 10 0.505 [0.288, 0.712] 50% $0.0010 0.0019 7.2s 8.7s
solo-luna 10 0.214 [0.114, 0.310] 0% $0.0009 - 12.6s 14.5s
solo-opus 10 0.454 [0.250, 0.676] 30% $0.0631 0.2103 24.1s 29.0s
solo-sol 10 0.276 [0.090, 0.490] 20% $0.0250 0.1249 26.8s 34.1s
fusion-default 10 0.397 [0.198, 0.594] 40% $0.0065 0.0163 20.6s 33.1s
fusion-best 10 0.300 [0.129, 0.507] 10% $0.0058 0.0582 11.7s 13.7s

By difficulty

easy

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
solo-haiku 13 0.709 [0.544, 0.849] 54% $0.0008 0.0015 5.3s 7.9s
solo-luna 13 0.558 [0.376, 0.747] 62% $0.0006 0.0010 7.6s 12.6s
solo-opus 13 0.637 [0.474, 0.797] 62% $0.0455 0.0740 17.5s 22.0s
solo-sol 13 0.502 [0.309, 0.700] 38% $0.0157 0.0409 15.4s 26.8s
fusion-default 13 0.755 [0.616, 0.885] 62% $0.0034 0.0055 7.3s 22.5s
fusion-best 13 0.709 [0.546, 0.868] 62% $0.0038 0.0061 11.0s 13.5s

hard

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
solo-haiku 15 0.613 [0.426, 0.782] 53% $0.0008 0.0016 7.0s 7.9s
solo-luna 15 0.561 [0.362, 0.757] 47% $0.0007 0.0015 11.0s 14.2s
solo-opus 15 0.490 [0.321, 0.656] 27% $0.0562 0.2108 21.6s 30.0s
solo-sol 15 0.510 [0.361, 0.645] 47% $0.0196 0.0421 22.5s 28.7s
fusion-default 15 0.523 [0.359, 0.685] 40% $0.0055 0.0137 13.4s 23.8s
fusion-best 15 0.571 [0.384, 0.753] 53% $0.0046 0.0086 11.2s 13.1s

medium

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
solo-haiku 19 0.621 [0.458, 0.773] 63% $0.0010 0.0016 7.2s 10.5s
solo-luna 19 0.505 [0.335, 0.672] 42% $0.0007 0.0016 10.1s 14.5s
solo-opus 19 0.637 [0.516, 0.772] 42% $0.0591 0.1403 23.4s 35.0s
solo-sol 19 0.436 [0.266, 0.610] 37% $0.0209 0.0567 22.9s 35.1s
fusion-default 19 0.585 [0.431, 0.740] 58% $0.0114 0.0197 13.2s 23.6s
fusion-best 19 0.545 [0.369, 0.708] 47% $0.0041 0.0086 11.6s 13.7s

Judge reliability

Accuracy floor 80%: mimo-pro 100%, muse-ember 88%.

Most different answers, with evidence

fe-data-table (frontend): completion differs by +0.99

  • fusion-best (repeat 1): completion 0.99
  • gate tests-pass: pass
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: all 7 hidden tests pass
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +171 -2 lines in 3 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: All 7 hidden tests pass (E1) and static/build/a11y checks are clean (E2, E3, E5). index.html has lang, viewport meta, a title, a single h1, a , column headers with scope=col, row headers with scope=row, and a button in each column header with data-sort (lines 14-30). app.js updates aria-sort on the active column (and removes it elsewhere) and toggles ascending/descending per column, using numeric comparison for tickets and localeCompare for text (lines 12-31). The wrapper .table-scroll has tabindex=0 + role=region + aria-label so it is keyboard-reachable, and overflow-x:auto with table min-width:36rem means the table scrolls inside its own container on narrow screens while body max-width keeps the page from breaking (style.css lines 28-47). Design matches the brief: bordered/tinted header row (#e8eef6), zebra rows (nth-child(even)), borderless text-looking header buttons, caption on top. Focus-visible outlines are defined for both the wrapper and buttons (style.css lines 36-40). E3 shows no external requests, viewport meta present, no lint issues; E4 shows a compact 171-line addition across 3 files. Minor nits: numeric detection is hardcoded to the 'tickets' key rather tha
  • solo-opus (repeat 1): completion 0.00
  • gate tests-pass: FAIL
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: 6/7 hidden tests pass; failing: HiddenTests.test_exactly_one_column_reports_its_sort
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +178 -2 lines in 3 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: Output A builds a real table with a caption (#people-caption), column headers with sorting buttons, row headers (scope="row" for each name), and a keyboard-reachable scroll wrapper (role=region, tabindex=0, overflow-x:auto with min-width:36rem so it scrolls on a 390px phone) — E3 static PASS, E5 a11y PASS. Sorting works for text and numeric columns via data-type, and the sorted column is toggled to aria-sort with a live status line. However, the hidden suite fails: E1 shows 6/7 with HiddenTests.test_exactly_one_column_reports_its_sort failing consistently across 3 runs, and the tests-pass hard gate is FAIL. This directly undercuts the core sort-reporting requirement even though the JS attempts to reset other headers to "none" (app.js lines 31-40), so correctness and ux_polish are reduced. The tinted header (#eef2f8), bordered wrapper, zebra-capable rows (only border-top given; no alternating row tint is actually applied — a design_fidelity gap), text-like header buttons, caption on top, visible :focus-visible outline, and viewport meta/lang/title all satisfy the design and a11y expectations (E2 build PASS, E4 diff_stats PASS), and everything is local/offline (works-offline pass). C

fe-login-dark (frontend): completion differs by -0.93

  • fusion-default (repeat 1): completion 0.00
  • gate tests-pass: FAIL
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +156 -2 lines in 3 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: The output delivers the requested structure: index.html has lang, title, viewport meta, one h1, a labelled Username and Password field, a Show password checkbox with an 8-line app.js toggle (app.js lines 1-8), a full-width Sign in button and a quiet Forgot your password? link (E3, E5). Accessibility is strong: E5 reports zero contrast failures and a 3px yellow :focus-visible outline (style.css line 118-121); autocomplete attributes are present. Design matches the brief: centred dark card (max-width 24rem, .page flex-centred), full-width primary button and underlined secondary link (E3 max_fixed_width_px=0). However, the hidden test suite fails 6/7, specifically HiddenTests.test_fits_small_screens (E1), so the responsiveness criterion cannot be scored highly; the remaining build/static/a11y checks pass (E2, E3, E5) and the file diff is minimal and clean (E4: +156 -2 over 3 files). | E1 shows 6/7 hidden tests pass but test_fits_small_screens fails, so responsiveness is weak and tests hard gate fails despite E2 build PASS, E3 static PASS and E5 a11y PASS indicating correct structure, labels and contrast. E6 screenshot n/a and E7 console n/a limit visual verification, so design and pol
  • solo-opus (repeat 1): completion 0.93
  • gate tests-pass: pass
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: all 7 hidden tests pass
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +170 -2 lines in 3 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: Output A implements the brief cleanly: index.html (E1/E3) has lang, title, viewport meta, a single h1 'Sign in', a semantic
    /
    with properly associated

fe-login-dark (frontend): completion differs by -0.93

  • solo-haiku (repeat 1): completion 0.00
  • gate tests-pass: FAIL
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +44 -2 lines in 1 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: Output A delivers a single self-contained index.html with lang, viewport meta, title, one h1, labelled Username/Password inputs with autocomplete, a working Show password checkbox toggle (inline script lines 42-50), a full-width Sign in button and a quiet 'Forgot your password?' link, all centred in a dark card (E3, E5: 0 contrast failures, 0 lint issues, 0 external requests). However, E1 reports 6/7 hidden tests passing with HiddenTests.test_fits_small_screens failing, so the responsiveness requirement from 390px phone to 1440px desktop is not fully verified despite the CSS using width:100%, max-width:360px and flex centering; this also trips the hard gate tests-pass (FAIL, E1). Everything else builds and passes static/a11y checks (E2, E3, E5). | File shows centred dark card with h1, labelled Username/Password, Show password checkbox toggling type, full-width Sign in button, Forgot link, bright #ffd54a focus-visible ring and high-contrast colours, matching a11y PASS E5 contrast_failures=0. However hard test gate fails: E1 tests FAIL 6/7 with HiddenTests.test_fits_small_screens failing, so responsiveness penalised despite viewport meta and max-width layout; build/static/diff PASS p
  • solo-opus (repeat 1): completion 0.93
  • gate tests-pass: pass
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: all 7 hidden tests pass
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +170 -2 lines in 3 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: Output A implements the brief cleanly: index.html (E1/E3) has lang, title, viewport meta, a single h1 'Sign in', a semantic
    / with properly associated

Methodology

  • Run: v3-headline (completed), created 2026-11-19 10:42:16
  • Dataset: evals/datasets/v2, split test, 47 tasks, hash 84c35e7998b364bd
  • Arm solo-opus: strategy solo-opus: solo, models claude-opus, aggregator auto, rounds 1, judge off
  • Arm solo-sol: strategy solo-sol: solo, models gpt-sol, aggregator auto, rounds 1, judge off
  • Arm solo-haiku: strategy solo-haiku: solo, models claude-haiku, aggregator auto, rounds 1, judge off
  • Arm solo-luna: strategy solo-luna: solo, models gpt-luna, aggregator auto, rounds 1, judge off
  • Arm fusion-default: strategy auto: auto, models claude-haiku, gpt-luna, gemini-flash, gpt-terra, gemini-hydrogen, claude-sonnet, aggregator auto, rounds 1, judge off
  • Arm fusion-best: strategy panel-nano-select, overrides {'aggregator': 'select'}: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator select, rounds 1, judge off
  • Model claude-opus: anthropic claude-opus-5-5: $4 in / $20 out per 1M tokens, verified 2026-11-17
  • Model gpt-sol: openai gpt-6.2-sol: $2 in / $10 out per 1M tokens, verified 2026-11-17
  • Model claude-haiku: anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Model gpt-luna: openai gpt-6-luna: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Model gemini-flash: google gemini-3.8-flash: $0.75 in / $3.75 out per 1M tokens, verified 2026-11-17
  • Model gpt-terra: openai gpt-6.1-terra: $1 in / $5 out per 1M tokens, verified 2026-11-17
  • Model gemini-hydrogen: google gemini-4-hydrogen: $1.5 in / $7.5 out per 1M tokens, verified 2026-11-17
  • Model claude-sonnet: anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
  • Model mimo-pro: openrouter xiaomi/mimo-v2.6-pro: $0.435 in / $0.87 out per 1M tokens, verified 2026-11-17
  • Model muse-ember: openrouter meta/muse-ember: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Judge models: mimo-pro, muse-ember
  • Repeats / seed: 1 per task / seed 100
  • Intervals: 95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
  • Verdict rules: non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline solo-opus
  • Spend: $4.03 on the arms + $0.3940 on scoring and judging (cap $6.00)
  • Fusion version: 0.2.0 (git 5abab47, with uncommitted changes)

Supporting study: v3-ablation

Arms: vote, lenses-off, digest, select, mixed, verified-synth, open, nano, mid, cascade-0.5, auto-route, cascade-0.9, auto-route-frontier. Baseline for the comparisons: vote.

Some prices were unknown for vote, lenses-off, digest, select, mixed, verified-synth, open, nano, mid, cascade-0.5, auto-route, cascade-0.9, auto-route-frontier: its costs are a lower bound. Items replayed from the response cache are left out of the latency figures (vote, lenses-off, digest, mixed, verified-synth, open, mid, cascade-0.5, auto-route, cascade-0.9, auto-route-frontier): their wall time is not a measurement. 1 repeat(s) per task: variance across repeats is poorly estimated. Fewer than 10 tasks are answered by both an arm and the baseline: no verdict is possible for it. Frontend and performance scores measure what the answer builds (tests, timings, screenshots); they do not judge whether it matches the author's intent.

Headline

Each arm against vote, on all tasks. ✓ yes, ✗ no, ? inconclusive, ⊘ blocked. The reasons are listed below the table.

Arm Paired tasks Cheaper (per solved task) Faster Not worse Better
lenses-off 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
digest 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
select 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
mixed 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
verified-synth 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
open 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
nano 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
mid 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
cascade-0.5 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
auto-route 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
cascade-0.9 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
auto-route-frontier 9 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
  • lenses-off vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
  • digest vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
  • select vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
  • mixed vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
  • verified-synth vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
  • open vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
  • nano vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
  • mid vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
  • cascade-0.5 vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
  • auto-route vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
  • cascade-0.9 vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
  • auto-route-frontier vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.

Arms

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90 Frontier
vote 9 0.661 [0.414, 0.891] 56% $0.0055 0.0098 8.8s 13.3s s
lenses-off 9 0.619 [0.353, 0.881] 56% $0.0053 0.0096 21.0s 29.1s $
digest 9 0.199 [0.037, 0.422] 11% $0.0068 0.0615 11.5s 11.5s -
select 9 0.714 [0.454, 0.941] 56% $0.0054 0.0097 13.2s 23.7s $ s
mixed 9 0.461 [0.161, 0.765] 44% $0.0040 0.0090 20.2s 29.9s $
verified-synth 9 0.721 [0.454, 0.917] 67% $0.0065 0.0098 15.0s 19.8s $ s
open 9 0.438 [0.140, 0.730] 44% $0.0070 0.0157 29.8s 38.4s -
nano 9 0.499 [0.213, 0.796] 44% $0.0088 0.0197 21.7s 33.1s -
mid 9 0.495 [0.199, 0.792] 44% $0.0730 0.1643 29.3s 41.5s -
cascade-0.5 9 0.676 [0.417, 0.908] 67% $0.0418 0.0627 - - -
auto-route 9 0.696 [0.422, 0.907] 67% $0.0236 0.0355 13.0s 29.5s s
cascade-0.9 9 0.728 [0.467, 0.933] 67% $0.0391 0.0587 - - $
auto-route-frontier 9 0.642 [0.385, 0.884] 67% $0.0391 0.0586 20.6s 30.9s -

Frontier: $ = on the cost-quality frontier (no arm is cheaper per task and at least as good), s = on the latency-quality frontier (median seconds).

Efficiency and speed

Arm Out tok/s (effective) Tokens / solved Quality / $ Quality / min Calls Errors
vote 214 25,296 121.0 3.981 3.2 0
lenses-off 230 32,653 115.8 1.806 4.4 0
digest 222 121,367 29.2 1.038 3.2 0
select 188 24,857 133.0 2.991 4.2 0
mixed 197 43,452 115.5 1.105 4.8 0
verified-synth 193 20,161 110.1 2.656 4.0 0
open 200 43,548 62.9 0.972 7.0 0
nano 229 44,460 57.1 1.286 4.2 0
mid 191 44,941 6.8 1.013 4.7 0
cascade-0.5 - 35,659 16.2 - 4.2 0
auto-route 180 20,460 29.4 2.680 3.1 0
cascade-0.9 - 35,078 18.6 - 4.2 0
auto-route-frontier 180 26,849 16.4 1.974 3.6 0

Streamed speed per model:

Arm Model Decode tok/s TTFT ms
vote claude-haiku 237 1066
vote gemini-flash 142 4138
vote gpt-luna 186 4413
lenses-off claude-haiku 323 2489
lenses-off gemini-flash 141 3350
lenses-off gpt-luna 177 4506
digest claude-haiku 217 1282
digest gemini-flash 147 2964
digest gpt-luna 149 5000
select claude-haiku 270 1500
select gemini-flash 143 2784
select gpt-luna 219 5894
mixed claude-haiku 303 2506
mixed gpt-luna 193 5554
mixed minimax-flash 416 716
verified-synth claude-haiku 286 1196
verified-synth gemini-flash 144 3481
verified-synth gpt-luna 223 5874
open claude-haiku 330 2933
open deepseek-flash 447 6642
open glm 296 3107
open gpt-luna 255 3951
open mimo-flash 184 6159
open minimax-flash 290 1504
nano claude-haiku 351 3664
nano gemini-flash 144 3246
nano gpt-luna 288 3916
mid claude-sonnet 207 4647
mid gemini-hydrogen 1188 5491
mid gpt-terra 120 3877
auto-route claude-haiku 253 1488
auto-route gemini-flash 136 2861
auto-route gpt-luna 215 5554
auto-route-frontier claude-haiku 368 3351
auto-route-frontier gemini-flash 138 6825
auto-route-frontier gpt-luna 219 4967

Cost of measuring

Scoring and judging cost and time, kept apart: never part of an arm's cost or latency.

Arm Eval $ total Eval $ / item Eval s / item
vote $0.0067 $0.0007 40.7s
lenses-off $0.0094 $0.0010 4.6s
digest $0.0068 $0.0008 0.0s
select $0.0109 $0.0012 40.1s
mixed $0.0104 $0.0012 4.5s
verified-synth $0.0054 $0.0006 38.0s
open $0.0121 $0.0013 4.9s
nano $0.0130 $0.0014 7.6s
mid $0.0118 $0.0013 39.6s
cascade-0.5 $0.0089 $0.0010 38.5s
auto-route $0.0087 $0.0010 29.7s
cascade-0.9 $0.0104 $0.0012 58.0s
auto-route-frontier $0.0094 $0.0010 37.4s

Variance across repeats

Arm Mean sd of quality across repeats Tasks whose solved flips
vote - -
lenses-off - -
digest - -
select - -
mixed - -
verified-synth - -
open - -
nano - -
mid - -
cascade-0.5 - -
auto-route - -
cascade-0.9 - -
auto-route-frontier - -

Comparisons against the baseline

lenses-off vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 -0.042 [-0.185, +0.080] 4/3/2 0.688 -0.19 0.98 [0.47, 3.02] 1.96 [1.66, 2.63] ? ? ? ?
architecture 1 +0.167 1/0/0 1.000 - - 1.79 ? ? ? ?
code_review 2 -0.400 [-0.467, -0.333] 0/0/2 0.500 -4.24 - 2.93 ? ? ? ?
coding 2 +0.040 [+0.000, +0.080] 1/1/0 1.000 +0.71 0.35 - ? ? ? ?
debugging 1 +0.000 0/1/0 - - 1.91 1.86 ? ? ? ?
frontend 1 -0.017 0/1/0 - - 0.36 - ? ? ? ?
performance 1 +0.025 1/0/0 1.000 - 1.32 - ? ? ? ?
planning 1 +0.167 1/0/0 1.000 - - 1.62 ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

digest vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 -0.462 [-0.789, -0.117] 2/1/6 0.289 -0.84 6.25 1.11 ? ? ? ?
architecture 1 +0.333 1/0/0 1.000 - - 1.11 ? ? ? ?
code_review 2 -0.520 [-0.606, -0.433] 0/0/2 0.500 -4.26 - - ? ? ? ?
coding 2 -0.960 [-1.000, -0.920] 0/0/2 0.500 -16.97 - - ? ? ? ?
debugging 1 +0.000 0/1/0 - - 3.88 - ? ? ? ?
frontend 1 -0.902 0/0/1 1.000 - - - ? ? ? ?
performance 1 -0.964 0/0/1 1.000 - - - ? ? ? ?
planning 1 +0.333 1/0/0 1.000 - - - ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

select vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 +0.052 [-0.046, +0.156] 5/2/2 0.453 +0.32 0.98 [0.48, 1.52] 1.28 [1.04, 1.54] ? ? ? ?
architecture 1 +0.167 1/0/0 1.000 - - 1.59 ? ? ? ?
code_review 2 +0.042 [-0.250, +0.333] 1/0/1 1.000 +0.10 0.53 1.21 ? ? ? ?
coding 2 +0.020 [+0.000, +0.040] 1/1/0 1.000 +0.71 1.57 - ? ? ? ?
debugging 1 +0.000 0/1/0 - - 1.27 1.38 ? ? ? ?
frontend 1 -0.024 0/0/1 1.000 - 0.27 - ? ? ? ?
performance 1 +0.036 1/0/0 1.000 - 0.94 - ? ? ? ?
planning 1 +0.167 1/0/0 1.000 - - 1.00 ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

mixed vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 -0.201 [-0.433, +0.006] 2/3/4 0.688 -0.55 0.91 [0.39, 3.16] 2.88 ? ? ? ?
architecture 1 +0.000 0/1/0 - - - 2.88 ? ? ? ?
code_review 2 -0.451 [-0.485, -0.417] 0/0/2 0.500 -9.35 - - ? ? ? ?
coding 2 -0.460 [-1.000, +0.080] 1/0/1 1.000 -0.60 0.41 - ? ? ? ?
debugging 1 +0.000 0/1/0 - - 3.17 - ? ? ? ?
frontend 1 -0.021 0/0/1 1.000 - 0.50 - ? ? ? ?
performance 1 +0.036 1/0/0 1.000 - 0.43 - ? ? ? ?
planning 1 +0.000 0/1/0 - - - - ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

verified-synth vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 +0.059 [-0.038, +0.156] 5/3/1 0.219 +0.37 1.00 [0.47, 1.77] - ? ? ? ?
architecture 1 +0.167 1/0/0 1.000 - - - ? ? ? ?
code_review 2 +0.042 [-0.250, +0.333] 1/0/1 1.000 +0.10 1.85 - ? ? ? ?
coding 2 +0.040 [+0.000, +0.080] 1/1/0 1.000 +0.71 0.86 - ? ? ? ?
debugging 1 +0.000 0/1/0 - - 1.28 - ? ? ? ?
frontend 1 -0.000 0/1/0 - - 0.23 - ? ? ? ?
performance 1 +0.036 1/0/0 1.000 - 1.25 - ? ? ? ?
planning 1 +0.167 1/0/0 1.000 - - - ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

open vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 -0.223 [-0.435, -0.037] 1/4/4 0.375 -0.67 1.59 [0.73, 4.49] 3.82 ? ? ? ?
architecture 1 +0.000 0/1/0 - - - 3.71 ? ? ? ?
code_review 2 -0.477 [-0.513, -0.441] 0/0/2 0.500 -9.42 - 3.98 ? ? ? ?
coding 2 -0.460 [-0.920, +0.000] 0/1/1 1.000 -0.71 1.28 - ? ? ? ?
debugging 1 -0.150 0/0/1 1.000 - 4.22 - ? ? ? ?
frontend 1 -0.010 0/1/0 - - 0.49 - ? ? ? ?
performance 1 +0.025 1/0/0 1.000 - 0.61 - ? ? ? ?
planning 1 +0.000 0/1/0 - - - - ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

nano vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 -0.162 [-0.424, +0.060] 2/4/3 1.000 -0.41 2.00 [0.67, 7.41] 2.13 [1.58, 2.76] ? ? ? ?
architecture 1 +0.333 1/0/0 1.000 - - 2.47 ? ? ? ?
code_review 2 -0.442 [-0.467, -0.417] 0/0/2 0.500 -12.49 - 2.89 ? ? ? ?
coding 2 -0.460 [-1.000, +0.080] 1/0/1 1.000 -0.60 1.76 - ? ? ? ?
debugging 1 +0.000 0/1/0 - - 5.10 2.21 ? ? ? ?
frontend 1 -0.008 0/1/0 - - 0.43 - ? ? ? ?
performance 1 +0.018 0/1/0 - - 1.30 - ? ? ? ?
planning 1 +0.000 0/1/0 - - - 1.37 ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

mid vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 -0.166 [-0.371, +0.013] 2/4/3 1.000 -0.52 16.70 [6.72, 54.93] 2.80 [2.38, 3.25] ? ? ? ?
architecture 1 +0.000 0/1/0 - - - 2.83 ? ? ? ?
code_review 2 -0.354 [-0.375, -0.333] 0/0/2 0.500 -12.02 - 3.25 ? ? ? ?
coding 2 +0.040 [+0.000, +0.080] 1/1/0 1.000 +0.71 4.59 - ? ? ? ?
debugging 1 +0.000 0/1/0 - - 49.68 2.38 ? ? ? ?
frontend 1 -0.902 0/0/1 1.000 - - - ? ? ? ?
performance 1 +0.036 1/0/0 1.000 - 10.62 - ? ? ? ?
planning 1 +0.000 0/1/0 - - - - ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

cascade-0.5 vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 +0.015 [-0.043, +0.066] 3/5/1 0.625 +0.17 6.37 [3.52, 11.42] - ? ? ? ?
architecture 1 +0.000 0/1/0 - - - - ? ? ? ?
code_review 2 -0.083 [-0.167, +0.000] 0/1/1 1.000 -0.71 10.77 - ? ? ? ?
coding 2 +0.040 [+0.000, +0.080] 1/1/0 1.000 +0.71 2.50 - ? ? ? ?
debugging 1 +0.000 0/1/0 - - 24.74 - ? ? ? ?
frontend 1 +0.018 0/1/0 - - 6.01 - ? ? ? ?
performance 1 +0.036 1/0/0 1.000 - 7.41 - ? ? ? ?
planning 1 +0.167 1/0/0 1.000 - - - ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

auto-route vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 +0.035 [-0.096, +0.149] 5/2/2 0.453 +0.17 3.61 [0.70, 9.11] 1.28 [0.43, 2.54] ? ? ? ?
architecture 1 +0.167 1/0/0 1.000 - - 2.54 ? ? ? ?
code_review 2 -0.033 [-0.400, +0.333] 1/0/1 1.000 -0.06 0.99 0.43 ? ? ? ?
coding 2 +0.040 [+0.000, +0.080] 1/1/0 1.000 +0.71 2.72 - ? ? ? ?
debugging 1 +0.000 0/1/0 - - 0.51 0.52 ? ? ? ?
frontend 1 -0.070 0/0/1 1.000 - 1.16 - ? ? ? ?
performance 1 +0.036 1/0/0 1.000 - 0.15 - ? ? ? ?
planning 1 +0.167 1/0/0 1.000 - - - ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

cascade-0.9 vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 +0.066 [-0.047, +0.185] 4/4/1 0.375 +0.34 5.97 [3.20, 10.54] - ? ? ? ?
architecture 1 +0.000 0/1/0 - - - - ? ? ? ?
code_review 2 +0.042 [-0.250, +0.333] 1/0/1 1.000 +0.10 10.53 - ? ? ? ?
coding 2 +0.040 [+0.000, +0.080] 1/1/0 1.000 +0.71 2.24 - ? ? ? ?
debugging 1 +0.000 0/1/0 - - 22.34 - ? ? ? ?
frontend 1 -0.001 0/1/0 - - 5.97 - ? ? ? ?
performance 1 +0.036 1/0/0 1.000 - 7.61 - ? ? ? ?
planning 1 +0.400 1/0/0 1.000 - - - ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

auto-route-frontier vs vote

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 9 -0.019 [-0.320, +0.246] 5/2/2 0.453 -0.04 5.96 [0.40, 23.54] 1.49 [0.64, 2.42] ? ? ? ?
architecture 1 +0.667 1/0/0 1.000 - - 2.42 ? ? ? ?
code_review 2 -0.200 [-0.400, +0.000] 0/1/1 1.000 -0.71 1.57 1.21 ? ? ? ?
coding 2 -0.460 [-1.000, +0.080] 1/0/1 1.000 -0.60 11.89 - ? ? ? ?
debugging 1 +0.000 0/1/0 - - 0.63 0.64 ? ? ? ?
frontend 1 +0.043 1/0/0 1.000 - 1.07 - ? ? ? ?
performance 1 +0.036 1/0/0 1.000 - 0.15 - ? ? ? ?
planning 1 +0.400 1/0/0 1.000 - - - ? ? ? ?
  • architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.

Δ quality is the arm minus the baseline, paired by task. Ratios are the arm over the baseline (below 1 is better). W/T/L counts tasks the arm won, tied (within 0.02) or lost. Effect d is the mean paired difference over its spread across tasks.

By category

architecture

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
vote 1 0.000 0% $0.0046 - 10.4s 10.4s
lenses-off 1 0.167 0% $0.0063 - 18.6s 18.6s
digest 1 0.333 0% $0.0050 - 11.5s 11.5s
select 1 0.167 0% $0.0021 - 16.5s 16.5s
mixed 1 0.000 0% $0.0035 - 29.9s 29.9s
verified-synth 1 0.167 0% $0.0022 - - -
open 1 0.000 0% $0.0048 - 38.4s 38.4s
nano 1 0.333 0% $0.0066 - 25.6s 25.6s
mid 1 0.000 0% $0.0714 - 29.3s 29.3s
cascade-0.5 1 0.000 0% $0.0494 - - -
auto-route 1 0.167 0% $0.0063 - 26.3s 26.3s
cascade-0.9 1 0.000 0% $0.0413 - - -
auto-route-frontier 1 0.667 100% $0.0065 0.0065 25.1s 25.1s

code_review

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
vote 2 0.583 [0.500, 0.667] 50% $0.0025 0.0050 7.5s 7.5s
lenses-off 2 0.183 [0.167, 0.200] 0% $0.0057 - 15.6s 21.9s
digest 2 0.064 [0.061, 0.067] 0% $0.0047 - - -
select 2 0.625 [0.250, 1.000] 50% $0.0013 0.0026 9.1s 10.7s
mixed 2 0.133 [0.083, 0.182] 0% $0.0050 - - -
verified-synth 2 0.625 [0.250, 1.000] 50% $0.0046 0.0093 - -
open 2 0.106 [0.059, 0.154] 0% $0.0078 - 29.8s 32.8s
nano 2 0.142 [0.083, 0.200] 0% $0.0062 - 20.3s 21.7s
mid 2 0.229 [0.125, 0.333] 0% $0.0610 - 24.4s 30.0s
cascade-0.5 2 0.500 [0.333, 0.667] 50% $0.0270 0.0541 - -
auto-route 2 0.550 [0.100, 1.000] 50% $0.0025 0.0050 3.2s 13.0s
cascade-0.9 2 0.625 [0.250, 1.000] 50% $0.0264 0.0528 - -
auto-route-frontier 2 0.383 [0.100, 0.667] 50% $0.0039 0.0079 9.1s 20.6s

coding

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
vote 2 0.960 [0.920, 1.000] 50% $0.0102 0.0204 - -
lenses-off 2 1.000 [1.000, 1.000] 100% $0.0072 0.0072 21.0s 21.0s
digest 2 0.000 [0.000, 0.000] 0% $0.0150 - - -
select 2 0.980 [0.960, 1.000] 50% $0.0160 0.0320 13.2s 17.6s
mixed 2 0.500 [0.000, 1.000] 50% $0.0041 0.0083 20.2s 20.2s
verified-synth 2 1.000 [1.000, 1.000] 100% $0.0175 0.0175 12.7s 19.8s
open 2 0.500 [0.000, 1.000] 50% $0.0131 0.0261 16.6s 16.6s
nano 2 0.500 [0.000, 1.000] 50% $0.0180 0.0359 20.3s 28.5s
mid 2 1.000 [1.000, 1.000] 100% $0.0936 0.0936 28.5s 28.5s
cascade-0.5 2 1.000 [1.000, 1.000] 100% $0.0510 0.0510 - -
auto-route 2 1.000 [1.000, 1.000] 100% $0.0556 0.0556 16.9s 16.9s
cascade-0.9 2 1.000 [1.000, 1.000] 100% $0.0458 0.0458 - -
auto-route-frontier 2 0.500 [0.000, 1.000] 50% $0.1213 0.2427 25.9s 25.9s

debugging

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
vote 1 1.000 100% $0.0012 0.0012 8.8s 8.8s
lenses-off 1 1.000 100% $0.0022 0.0022 16.3s 16.3s
digest 1 1.000 100% $0.0046 0.0046 - -
select 1 1.000 100% $0.0015 0.0015 12.1s 12.1s
mixed 1 1.000 100% $0.0037 0.0037 - -
verified-synth 1 1.000 100% $0.0015 0.0015 - -
open 1 0.850 100% $0.0050 0.0050 - -
nano 1 1.000 100% $0.0060 0.0060 19.4s 19.4s
mid 1 1.000 100% $0.0583 0.0583 20.9s 20.9s
cascade-0.5 1 1.000 100% $0.0290 0.0290 - -
auto-route 1 1.000 100% $0.0006 0.0006 4.6s 4.6s
cascade-0.9 1 1.000 100% $0.0262 0.0262 - -
auto-route-frontier 1 1.000 100% $0.0007 0.0007 5.6s 5.6s

frontend

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
vote 1 0.902 100% $0.0113 0.0113 - -
lenses-off 1 0.885 100% $0.0041 0.0041 29.1s 29.1s
digest 1 0.000 0% $0.0025 - - -
select 1 0.878 100% $0.0031 0.0031 23.7s 23.7s
mixed 1 0.881 100% $0.0057 0.0057 - -
verified-synth 1 0.902 100% $0.0026 0.0026 17.6s 17.6s
open 1 0.892 100% $0.0055 0.0055 - -
nano 1 0.894 100% $0.0049 0.0049 33.1s 33.1s
mid 1 0.000 0% $0.0835 - 41.5s 41.5s
cascade-0.5 1 0.920 100% $0.0676 0.0676 - -
auto-route 1 0.832 100% $0.0130 0.0130 29.5s 29.5s
cascade-0.9 1 0.900 100% $0.0673 0.0673 - -
auto-route-frontier 1 0.945 100% $0.0121 0.0121 30.9s 30.9s

performance

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
vote 1 0.964 100% $0.0050 0.0050 - -
lenses-off 1 0.989 100% $0.0066 0.0066 - -
digest 1 0.000 0% $0.0050 - - -
select 1 1.000 100% $0.0047 0.0047 12.6s 12.6s
mixed 1 1.000 100% $0.0022 0.0022 - -
verified-synth 1 1.000 100% $0.0063 0.0063 15.0s 15.0s
open 1 0.989 100% $0.0030 0.0030 17.6s 17.6s
nano 1 0.982 100% $0.0065 0.0065 22.6s 22.6s
mid 1 1.000 100% $0.0534 0.0534 30.9s 30.9s
cascade-0.5 1 1.000 100% $0.0372 0.0372 - -
auto-route 1 1.000 100% $0.0007 0.0007 - -
cascade-0.9 1 1.000 100% $0.0383 0.0383 - -
auto-route-frontier 1 1.000 100% $0.0007 0.0007 - -

planning

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
vote 1 0.000 0% $0.0017 - 13.3s 13.3s
lenses-off 1 0.167 0% $0.0032 - 21.5s 21.5s
digest 1 0.333 0% $0.0049 - - -
select 1 0.167 0% $0.0022 - 13.3s 13.3s
mixed 1 0.000 0% $0.0025 - - -
verified-synth 1 0.167 0% $0.0022 - - -
open 1 0.000 0% $0.0025 - - -
nano 1 0.000 0% $0.0064 - 18.2s 18.2s
mid 1 0.000 0% $0.0812 - - -
cascade-0.5 1 0.167 0% $0.0365 - - -
auto-route 1 0.167 0% $0.0759 - - -
cascade-0.9 1 0.400 0% $0.0347 - - -
auto-route-frontier 1 0.400 0% $0.0810 - - -

By difficulty

easy

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
vote 1 1.000 100% $0.0107 0.0107 - -
lenses-off 1 1.000 100% $0.0032 0.0032 21.0s 21.0s
digest 1 0.000 0% $0.0216 - - -
select 1 1.000 100% $0.0208 0.0208 13.2s 13.2s
mixed 1 0.000 0% $0.0031 - 20.2s 20.2s
verified-synth 1 1.000 100% $0.0224 0.0224 12.7s 12.7s
open 1 1.000 100% $0.0216 0.0216 16.6s 16.6s
nano 1 0.000 0% $0.0233 - 20.3s 20.3s
mid 1 1.000 100% $0.1009 0.1009 28.5s 28.5s
cascade-0.5 1 1.000 100% $0.0436 0.0436 - -
auto-route 1 1.000 100% $0.0993 0.0993 - -
cascade-0.9 1 1.000 100% $0.0381 0.0381 - -
auto-route-frontier 1 0.000 0% $0.2387 - - -

hard

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
vote 3 0.488 [0.000, 0.964] 33% $0.0046 0.0139 10.4s 10.4s
lenses-off 3 0.441 [0.167, 0.989] 33% $0.0061 0.0183 15.6s 18.6s
digest 3 0.133 [0.000, 0.333] 0% $0.0049 - 11.5s 11.5s
select 3 0.472 [0.167, 1.000] 33% $0.0028 0.0085 12.6s 16.5s
mixed 3 0.361 [0.000, 1.000] 33% $0.0039 0.0117 29.9s 29.9s
verified-synth 3 0.472 [0.167, 1.000] 33% $0.0045 0.0135 15.0s 15.0s
open 3 0.349 [0.000, 0.989] 33% $0.0049 0.0146 32.8s 38.4s
nano 3 0.466 [0.083, 0.982] 33% $0.0064 0.0191 22.6s 25.6s
mid 3 0.375 [0.000, 1.000] 33% $0.0639 0.1916 30.0s 30.9s
cascade-0.5 3 0.444 [0.000, 1.000] 33% $0.0387 0.1160 - -
auto-route 3 0.422 [0.100, 1.000] 33% $0.0039 0.0116 13.0s 26.3s
cascade-0.9 3 0.417 [0.000, 1.000] 33% $0.0363 0.1088 - -
auto-route-frontier 3 0.589 [0.100, 1.000] 67% $0.0039 0.0058 20.6s 25.1s

medium

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
vote 5 0.698 [0.333, 0.948] 60% $0.0049 0.0082 8.8s 13.3s
lenses-off 5 0.650 [0.330, 0.977] 60% $0.0053 0.0089 21.5s 29.1s
digest 5 0.279 [0.012, 0.667] 20% $0.0051 0.0253 - -
select 5 0.801 [0.476, 0.992] 60% $0.0038 0.0063 13.3s 23.7s
mixed 5 0.613 [0.236, 0.976] 60% $0.0042 0.0070 - -
verified-synth 5 0.814 [0.480, 1.000] 80% $0.0046 0.0058 17.6s 19.8s
open 5 0.379 [0.031, 0.727] 40% $0.0053 0.0132 29.8s 29.8s
nano 5 0.619 [0.219, 0.979] 60% $0.0073 0.0121 21.7s 33.1s
mid 5 0.467 [0.067, 0.867] 40% $0.0729 0.1822 24.4s 41.5s
cascade-0.5 5 0.751 [0.433, 0.984] 80% $0.0433 0.0541 - -
auto-route 5 0.800 [0.466, 1.000] 80% $0.0204 0.0255 4.6s 29.5s
cascade-0.9 5 0.860 [0.620, 1.000] 80% $0.0410 0.0513 - -
auto-route-frontier 5 0.802 [0.573, 0.989] 80% $0.0202 0.0253 9.1s 30.9s

Judge reliability

Accuracy floor 80%: mimo-pro 100%, muse-ember 88%.

Most different answers, with evidence

fe-contact-form (frontend): completion differs by -0.90

  • mid (repeat 1): completion 0.00
  • gate tests-pass: FAIL
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: 5/6 hidden tests pass; failing: HiddenTests.test_form_fits_small_screens
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +184 -2 lines in 3 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: Output A implements the requested structure: name/email inputs with autocomplete and required, a fieldset+legend grouping the three radios under 'What is this about?', a textarea with maxlength=1000 and hint, and a 'Send message' submit button (index.html lines 12-44). Semantic HTML with lang, title, viewport meta, a single h1, and every control labelled; E5 reports no contrast/focus/a11y violations and E3 reports clean static analysis with no external requests, satisfying offline and contrast basics. However, E1 shows 5/6 hidden tests pass with HiddenTests.test_form_fits_small_screens failing (pass_fraction 0.833), which directly undermines the responsiveness criterion. Also, design_fidelity is weakened because the brief asks for radios grouped in a bordered set but the CSS resets fieldset borders to 0 (style.css lines 51-56), and the radio choices never stack/row-switch at wider viewports since .radio-group is fixed flex-direction: column (lines 83-87) with no media query; the form is otherwise narrow single-column with a roomy 9rem textarea. Focus style (:focus-visible outline) is present. Overall it is a clean, largely correct implementation but fails the small-screen test and
  • vote (repeat 1): completion 0.90
  • gate tests-pass: pass
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: all 6 hidden tests pass
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +88 -2 lines in 1 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: A delivers a single self-contained index.html with all requested structure: name and email inputs with autocomplete and required, a fieldset/legend radio group 'What is this about?' (Sales/Support/Press) sharing name="topic" so the question is announced, a textarea with maxlength=1000 plus hint and required, and a 'Send message' submit button (E1 tests PASS, E2/E3 build and static PASS, E5 a11y PASS with 0 contrast failures). lang, title, viewport meta, one h1 in order, labelled controls, and a :focus-visible outline are all present; layout uses width:min(100% - 2rem, 42rem) so it fits 390px through 1440px. Deductions: the radio group is in a fieldset with border:0, so it is not the bordered set the design_fidelity brief specifies; there are no explicit responsive breakpoints, only flexible widths; and the textarea min-height 9rem is roomy but modest. Code is clean and minimal but all CSS/JS is inline rather than split into style.css/app.js (acceptable per brief, still slightly less tidy). | Evidence E1 shows 6/6 hidden tests pass, E3 confirms viewport and no external requests, and E5 shows no a11y violations. Index.html implements name/email with autocomplete and required, fieldse

fe-contact-form (frontend): completion differs by -0.07

  • auto-route (repeat 1): completion 0.83
  • gate tests-pass: pass
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: all 6 hidden tests pass
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +43 -2 lines in 1 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: All 6 hidden tests pass (E1), static and build checks are clean (E2, E3), and the a11y scanner reports zero violations (E5). The page (index.html) has lang, title, viewport meta, one h1, labels for every control, a fieldset/legend grouping the radios, maxlength=1000, autocomplete attributes, a visible :focus-visible outline, and a responsive max-width single-column layout with a bordered fieldset and an 8rem-tall textarea (design fidelity and responsiveness are solid). However, the form carries novalidate="" with no accompanying script, which disables the browser's native validation UI despite the brief explicitly asking to use native validation; required attributes are present so required marking is native, but the submit-time validation experience is off (E1's passing tests apparently only check attributes). Minor gaps: radio labels rely on implicit wrapping with no id/for, and there is no app.js despite it being permitted. These are deducted in correctness and ux_polish, though the tested structure and accessibility are otherwise complete. | E1 shows all 6 hidden tests pass and E5/E3 show no a11y/static violations; index.html has name/email with autocomplete, fieldset/legend rad
  • vote (repeat 1): completion 0.90
  • gate tests-pass: pass
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: all 6 hidden tests pass
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +88 -2 lines in 1 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: A delivers a single self-contained index.html with all requested structure: name and email inputs with autocomplete and required, a fieldset/legend radio group 'What is this about?' (Sales/Support/Press) sharing name="topic" so the question is announced, a textarea with maxlength=1000 plus hint and required, and a 'Send message' submit button (E1 tests PASS, E2/E3 build and static PASS, E5 a11y PASS with 0 contrast failures). lang, title, viewport meta, one h1 in order, labelled controls, and a :focus-visible outline are all present; layout uses width:min(100% - 2rem, 42rem) so it fits 390px through 1440px. Deductions: the radio group is in a fieldset with border:0, so it is not the bordered set the design_fidelity brief specifies; there are no explicit responsive breakpoints, only flexible widths; and the textarea min-height 9rem is roomy but modest. Code is clean and minimal but all CSS/JS is inline rather than split into style.css/app.js (acceptable per brief, still slightly less tidy). | Evidence E1 shows 6/6 hidden tests pass, E3 confirms viewport and no external requests, and E5 shows no a11y violations. Index.html implements name/email with autocomplete and required, fieldse

fe-contact-form (frontend): completion differs by +0.04

  • auto-route-frontier (repeat 1): completion 0.94
  • gate tests-pass: pass
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: all 6 hidden tests pass
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +50 -2 lines in 1 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: E1 (all 6 hidden tests pass), E2 (build clean), E3 (static: viewport meta, no external requests, no lint issues) and E5 (a11y: zero violations, no contrast failures) confirm the brief is met. index.html has lang/title/viewport (lines 2-6), one h1 (line 31), labelled name/email inputs with autocomplete and required (lines 35-40), radio group in a fieldset with legend 'What is this about?' (lines 42-49), a textarea with maxlength=1000 (line 52) and a submit button (line 54). Focus-visible outline (line 25) and 48px min-height button aid keyboard/mobile use. Layout is a single-column min(100%, 640px) card with clamp padding and flexible choices (lines 11, 21) which fits 390px-1440px. Minor design gaps: the form lacks a live submit handler, has no explicit 'required' asterisk/native-required marking visible to sighted users, and the radio choices wrap rather than stack at phone widths; the message box is roomy (min-height 150px) but the card is 640px which slightly exceeds the strict 'narrow' form width ideal. | All 6 hidden tests pass (E1) with no a11y or static violations (E3,E5); index.html shows native required/type=email validation, autocomplete name/email, fieldset/legend aroun
  • vote (repeat 1): completion 0.90
  • gate tests-pass: pass
  • gate works-offline: pass
  • gate no-console-errors: unverified
  • evidence E1: tests measured: all 6 hidden tests pass
  • evidence E2: build measured: everything compiles
  • evidence E3: static measured: clean
  • evidence E4: diff_stats measured: +88 -2 lines in 1 files
  • evidence E5: a11y measured: no violations (static source rules only)
  • evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • judge: A delivers a single self-contained index.html with all requested structure: name and email inputs with autocomplete and required, a fieldset/legend radio group 'What is this about?' (Sales/Support/Press) sharing name="topic" so the question is announced, a textarea with maxlength=1000 plus hint and required, and a 'Send message' submit button (E1 tests PASS, E2/E3 build and static PASS, E5 a11y PASS with 0 contrast failures). lang, title, viewport meta, one h1 in order, labelled controls, and a :focus-visible outline are all present; layout uses width:min(100% - 2rem, 42rem) so it fits 390px through 1440px. Deductions: the radio group is in a fieldset with border:0, so it is not the bordered set the design_fidelity brief specifies; there are no explicit responsive breakpoints, only flexible widths; and the textarea min-height 9rem is roomy but modest. Code is clean and minimal but all CSS/JS is inline rather than split into style.css/app.js (acceptable per brief, still slightly less tidy). | Evidence E1 shows 6/6 hidden tests pass, E3 confirms viewport and no external requests, and E5 shows no a11y violations. Index.html implements name/email with autocomplete and required, fieldse

Methodology

  • Run: v3-ablation (completed), created 2026-11-19 10:06:52
  • Dataset: evals/datasets/v2, split dev, 9 tasks, hash abf13a9e14e00523
  • Arm nano: strategy panel-nano: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
  • Arm open: strategy panel-open: panel, models deepseek-flash, minimax-flash, qwen-flash, glm, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
  • Arm mixed: strategy panel-mixed: panel, models claude-haiku, gpt-luna, deepseek-flash, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
  • Arm mid: strategy panel-mid: panel, models gpt-terra, gemini-hydrogen, claude-sonnet, synthesizer claude-sonnet, aggregator llm, rounds 1, judge off
  • Arm lenses-off: strategy panel-nano, overrides {'lenses': False}: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
  • Arm select: strategy panel-nano-select, overrides {'aggregator': 'select'}: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator select, rounds 1, judge off
  • Arm vote: strategy panel-vote: panel, models claude-haiku, gpt-luna, gemini-flash, aggregator vote, rounds 1, judge off
  • Arm digest: strategy panel-digest: panel, models claude-haiku, gpt-luna, gemini-flash, aggregator digest, rounds 1, judge off
  • Arm cascade-0.5: strategy panel-cascade, overrides {'cascade': {'agreement_threshold': 0.5}}: cascade, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-sonnet, aggregator llm, rounds 1, judge off
  • Arm cascade-0.9: strategy panel-cascade, overrides {'cascade': {'agreement_threshold': 0.9}}: cascade, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-sonnet, aggregator llm, rounds 1, judge off
  • Arm verified-synth: strategy verified-synth: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator verified-synth, rounds 1, judge off
  • Arm auto-route: strategy auto: auto, models claude-haiku, gpt-luna, gemini-flash, gpt-terra, gemini-hydrogen, claude-sonnet, aggregator auto, rounds 1, judge off
  • Arm auto-route-frontier: strategy auto-frontier: auto, models claude-haiku, gpt-luna, gemini-flash, gpt-terra, gemini-hydrogen, claude-sonnet, claude-opus, aggregator auto, rounds 1, judge off
  • Model claude-haiku: anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Model gpt-luna: openai gpt-6-luna: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Model gemini-flash: google gemini-3.8-flash: $0.75 in / $3.75 out per 1M tokens, verified 2026-11-17
  • Model deepseek-flash: openrouter deepseek/deepseek-v4.1-flash-1017: $0.036 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Model minimax-flash: openrouter minimax/minimax-m3.1-flash: $0.15 in / $0.6 out per 1M tokens, verified 2026-11-17
  • Model qwen-flash: openrouter qwen/qwen3.8-flash: $0.15 in / $0.47 out per 1M tokens, verified 2026-11-17
  • Model glm: openrouter z-ai/glm-5.3: $0.0699 in / $1.4 out per 1M tokens, verified 2026-11-17
  • Model gpt-terra: openai gpt-6.1-terra: $1 in / $5 out per 1M tokens, verified 2026-11-17
  • Model gemini-hydrogen: google gemini-4-hydrogen: $1.5 in / $7.5 out per 1M tokens, verified 2026-11-17
  • Model claude-sonnet: anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
  • Model claude-opus: anthropic claude-opus-5-5: $4 in / $20 out per 1M tokens, verified 2026-11-17
  • Model mimo-pro: openrouter xiaomi/mimo-v2.6-pro: $0.435 in / $0.87 out per 1M tokens, verified 2026-11-17
  • Model muse-ember: openrouter meta/muse-ember: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Judge models: mimo-pro, muse-ember
  • Repeats / seed: 1 per task / seed 0
  • Intervals: 95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
  • Verdict rules: non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline vote
  • Spend: $1.96 on the arms + $0.1240 on scoring and judging (cap $3.40)
  • Fusion version: 0.2.0 (git 5abab47, with uncommitted changes)

Supporting study: v3-route

Arms: rung0, rung1, rung2. Baseline for the comparisons: rung0.

Some prices were unknown for rung1, rung2: its costs are a lower bound. 1 repeat(s) per task: variance across repeats is poorly estimated.

Headline

Each arm against rung0, on all tasks. ✓ yes, ✗ no, ? inconclusive, ⊘ blocked. The reasons are listed below the table.

Arm Paired tasks Cheaper (per solved task) Faster Not worse Better
rung1 20 ✗ no ✗ no ✓ yes ✓ yes
rung2 20 ✗ no ✗ no ✓ yes ✓ yes
  • rung1 vs rung0 on all (20 tasks): cheaper (per solved task): no. cost per solved task ratio 5.16 [2.57, 9.97] (below 1 is better): the whole interval is at or above 1.
  • rung1 vs rung0 on all (20 tasks): faster: no. seconds per task ratio 1.86 [1.57, 2.15] (below 1 is better): the whole interval is at or above 1.
  • rung1 vs rung0 on all (20 tasks): not worse: yes. quality difference +0.097 [+0.000, +0.189] over 20 tasks: the lower bound is above -0.03, the non-inferiority margin.
  • rung1 vs rung0 on all (20 tasks): better: yes. quality difference +0.097 [+0.000, +0.189] over 20 tasks: the interval is above 0.
  • rung2 vs rung0 on all (20 tasks): cheaper (per solved task): no. cost per solved task ratio 81.23 [49.46, 129.12] (below 1 is better): the whole interval is at or above 1.
  • rung2 vs rung0 on all (20 tasks): faster: no. seconds per task ratio 4.96 [4.30, 5.70] (below 1 is better): the whole interval is at or above 1.
  • rung2 vs rung0 on all (20 tasks): not worse: yes. quality difference +0.144 [+0.064, +0.243] over 20 tasks: the lower bound is above -0.03, the non-inferiority margin.
  • rung2 vs rung0 on all (20 tasks): better: yes. quality difference +0.144 [+0.064, +0.243] over 20 tasks: the interval is above 0.

Arms

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90 Frontier
rung0 20 0.464 [0.346, 0.592] 40% $0.0008 0.0019 5.6s 7.2s $ s
rung1 20 0.560 [0.434, 0.692] 45% $0.0044 0.0099 10.7s 14.7s $ s
rung2 20 0.608 [0.469, 0.750] 45% $0.0700 0.1556 27.2s 36.3s $ s

Frontier: $ = on the cost-quality frontier (no arm is cheaper per task and at least as good), s = on the latency-quality frontier (median seconds).

Efficiency and speed

Arm Out tok/s (effective) Tokens / solved Quality / $ Quality / min Calls Errors
rung0 198 8,740 605.0 4.877 1.0 0
rung1 234 19,147 125.9 3.175 3.5 0
rung2 208 36,888 8.7 1.288 4.2 0

Streamed speed per model:

Arm Model Decode tok/s TTFT ms
rung0 claude-haiku 284 1493
rung1 claude-haiku 276 1560
rung1 gemini-flash 165 3827
rung1 gpt-luna 306 4135
rung2 claude-sonnet 175 1934
rung2 gemini-hydrogen 4648 5954
rung2 gpt-terra 115 2729

Cost of measuring

Scoring and judging cost and time, kept apart: never part of an arm's cost or latency.

Arm Eval $ total Eval $ / item Eval s / item
rung0 $0.0217 $0.0011 0.0s
rung1 $0.0238 $0.0012 0.0s
rung2 $0.0306 $0.0015 0.0s

Variance across repeats

Arm Mean sd of quality across repeats Tasks whose solved flips
rung0 - -
rung1 - -
rung2 - -

Comparisons against the baseline

rung1 vs rung0

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 20 +0.097 [+0.000, +0.189] 11/6/3 0.057 +0.45 5.16 [2.57, 9.97] 1.86 [1.57, 2.15] ✗ ✗ ✓ ✓
architecture 5 +0.073 [-0.200, +0.360] 2/1/2 1.000 +0.20 4.05 2.07 [1.74, 2.46] ? ? ? ?
code_review 5 +0.055 [-0.122, +0.200] 3/1/1 0.625 +0.27 5.47 1.50 [0.90, 2.72] ? ? ? ?
debugging 5 +0.072 [+0.000, +0.156] 2/3/0 0.500 +0.71 6.07 [5.05, 7.47] 1.75 [1.36, 2.29] ? ? ? ?
planning 5 +0.187 [+0.086, +0.288] 4/1/0 0.125 +1.35 - 2.06 [1.89, 2.34] ? ? ? ?
  • architecture (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.

rung2 vs rung0

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 20 +0.144 [+0.064, +0.243] 12/7/1 0.003 +0.70 81.23 [49.46, 129.12] 4.96 [4.30, 5.70] ✗ ✗ ✓ ✓
architecture 5 +0.127 [+0.013, +0.273] 3/2/0 0.250 +0.76 100.40 [89.73, 111.07] 5.33 [4.77, 5.93] ? ? ? ?
code_review 5 +0.126 [-0.072, +0.383] 4/0/1 0.375 +0.44 73.94 4.71 [2.92, 7.56] ? ? ? ?
debugging 5 +0.114 [+0.000, +0.258] 2/3/0 0.500 +0.69 87.84 [77.63, 101.33] 4.25 [3.85, 4.69] ? ? ? ?
planning 5 +0.210 [+0.029, +0.410] 3/2/0 0.250 +0.86 - 5.48 [4.32, 6.79] ? ? ? ?
  • architecture (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.

Δ quality is the arm minus the baseline, paired by task. Ratios are the arm over the baseline (below 1 is better). W/T/L counts tasks the arm won, tied (within 0.02) or lost. Effect d is the mean paired difference over its spread across tasks.

By category

architecture

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
rung0 5 0.467 [0.347, 0.587] 40% $0.0008 0.0019 6.1s 6.6s
rung1 5 0.540 [0.347, 0.733] 40% $0.0031 0.0078 12.3s 14.9s
rung2 5 0.593 [0.420, 0.813] 40% $0.0772 0.1929 32.1s 37.5s

code_review

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
rung0 5 0.304 [0.148, 0.509] 20% $0.0008 0.0038 4.9s 7.7s
rung1 5 0.359 [0.224, 0.517] 20% $0.0041 0.0206 6.9s 14.9s
rung2 5 0.430 [0.210, 0.740] 20% $0.0558 0.2790 23.8s 37.8s

debugging

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
rung0 5 0.856 [0.742, 0.970] 100% $0.0008 0.0008 5.4s 7.3s
rung1 5 0.928 [0.844, 1.000] 100% $0.0047 0.0047 9.4s 11.5s
rung2 5 0.970 [0.910, 1.000] 100% $0.0683 0.0683 24.5s 26.7s

planning

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
rung0 5 0.228 [0.113, 0.331] 0% $0.0008 - 5.8s 7.2s
rung1 5 0.414 [0.186, 0.600] 20% $0.0058 0.0292 11.5s 14.7s
rung2 5 0.437 [0.186, 0.686] 20% $0.0788 0.3941 34.6s 36.3s

By difficulty

easy

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
rung0 9 0.491 [0.373, 0.619] 44% $0.0007 0.0017 6.0s 6.6s
rung1 9 0.628 [0.487, 0.772] 56% $0.0041 0.0074 11.1s 14.9s
rung2 9 0.652 [0.493, 0.830] 44% $0.0726 0.1633 29.9s 37.8s

hard

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
rung0 7 0.383 [0.218, 0.576] 29% $0.0007 0.0026 5.6s 7.2s
rung1 7 0.472 [0.320, 0.656] 29% $0.0049 0.0170 11.5s 14.7s
rung2 7 0.584 [0.366, 0.795] 43% $0.0660 0.1540 27.2s 36.3s

medium

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
rung0 4 0.542 [0.083, 1.000] 50% $0.0009 0.0017 5.4s 7.7s
rung1 4 0.562 [0.125, 1.000] 50% $0.0045 0.0090 8.5s 10.4s
rung2 4 0.550 [0.100, 1.000] 50% $0.0713 0.1425 26.4s 35.3s

Methodology

  • Run: v3-route (completed), created 2026-11-19 10:22:39
  • Dataset: evals/datasets/v2, split dev, 20 tasks, hash cc49b32c66c27cbd
  • Arm rung0: strategy solo-haiku-check: solo, models claude-haiku, aggregator auto, rounds 1, judge off
  • Arm rung1: strategy panel-nano-auto: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator auto, rounds 1, judge off
  • Arm rung2: strategy panel-mid: panel, models gpt-terra, gemini-hydrogen, claude-sonnet, synthesizer claude-sonnet, aggregator llm, rounds 1, judge off
  • Model claude-haiku: anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Model gpt-luna: openai gpt-6-luna: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Model gemini-flash: google gemini-3.8-flash: $0.75 in / $3.75 out per 1M tokens, verified 2026-11-17
  • Model gpt-terra: openai gpt-6.1-terra: $1 in / $5 out per 1M tokens, verified 2026-11-17
  • Model gemini-hydrogen: google gemini-4-hydrogen: $1.5 in / $7.5 out per 1M tokens, verified 2026-11-17
  • Model claude-sonnet: anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
  • Model mimo-pro: openrouter xiaomi/mimo-v2.6-pro: $0.435 in / $0.87 out per 1M tokens, verified 2026-11-17
  • Model muse-ember: openrouter meta/muse-ember: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Judge models: mimo-pro, muse-ember
  • Repeats / seed: 1 per task / seed 7
  • Intervals: 95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
  • Verdict rules: non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline rung0
  • Spend: $1.50 on the arms + $0.0761 on scoring and judging (cap $2.40)
  • Fusion version: 0.2.0 (git 5abab47, with uncommitted changes)

Supporting study: v3-latency

Arms: luna-none, solo-opus, panel-duo, auto-route. Baseline for the comparisons: solo-opus.

1 job(s) halted on their first attempt (a call timed out or the panel had no quorum) and were run again, with longer timeouts where it was a timeout: solo-opus (1). Only the last attempt is scored; the first attempt's cost is in the totals. A halted job scores 0, so without the retry these arms would be understated by the harness's limit, not by their answers. Some prices were unknown for auto-route: its costs are a lower bound. 1 repeat(s) per task: variance across repeats is poorly estimated.

Headline

Each arm against solo-opus, on all tasks. ✓ yes, ✗ no, ? inconclusive, ⊘ blocked. The reasons are listed below the table.

Arm Paired tasks Cheaper (per solved task) Faster Not worse Better
luna-none 30 ✓ yes ✓ yes ? inconclusive ? inconclusive
panel-duo 30 ✓ yes ✓ yes ? inconclusive ? inconclusive
auto-route 30 ✓ yes ✓ yes ? inconclusive ✗ no
  • luna-none vs solo-opus on all (30 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.04 [0.03, 0.06] (below 1 is better): the whole interval is below 1.
  • luna-none vs solo-opus on all (30 tasks): faster: yes. seconds per task ratio 0.70 [0.63, 0.78] (below 1 is better): the whole interval is below 1.
  • luna-none vs solo-opus on all (30 tasks): not worse: inconclusive. quality difference -0.052 [-0.182, +0.075] over 30 tasks: the interval straddles the -0.03 margin.
  • luna-none vs solo-opus on all (30 tasks): better: inconclusive. quality difference -0.052 [-0.182, +0.075] over 30 tasks: the interval includes 0.
  • panel-duo vs solo-opus on all (30 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.05 [0.04, 0.06] (below 1 is better): the whole interval is below 1.
  • panel-duo vs solo-opus on all (30 tasks): faster: yes. seconds per task ratio 0.82 [0.74, 0.91] (below 1 is better): the whole interval is below 1.
  • panel-duo vs solo-opus on all (30 tasks): not worse: inconclusive. quality difference -0.065 [-0.178, +0.043] over 30 tasks: the interval straddles the -0.03 margin.
  • panel-duo vs solo-opus on all (30 tasks): better: inconclusive. quality difference -0.065 [-0.178, +0.043] over 30 tasks: the interval includes 0.
  • auto-route vs solo-opus on all (30 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.11 [0.08, 0.15] (below 1 is better): the whole interval is below 1.
  • auto-route vs solo-opus on all (30 tasks): faster: yes. seconds per task ratio 0.74 [0.64, 0.83] (below 1 is better): the whole interval is below 1.
  • auto-route vs solo-opus on all (30 tasks): not worse: inconclusive. quality difference -0.134 [-0.259, -0.009] over 30 tasks: the interval straddles the -0.03 margin.
  • auto-route vs solo-opus on all (30 tasks): better: no. quality difference -0.134 [-0.259, -0.009] over 30 tasks: the interval is at or below 0.

Arms

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90 Frontier
luna-none 30 0.743 [0.635, 0.845] 73% $0.0021 0.0028 14.2s 17.4s $ s
solo-opus 30 0.794 [0.684, 0.890] 77% $0.0486 0.0634 18.5s 25.0s $ s
panel-duo 30 0.730 [0.628, 0.829] 73% $0.0022 0.0030 16.8s 20.5s -
auto-route 30 0.660 [0.544, 0.772] 63% $0.0044 0.0069 12.6s 22.9s s

Frontier: $ = on the cost-quality frontier (no arm is cheaper per task and at least as good), s = on the latency-quality frontier (median seconds).

Efficiency and speed

Arm Out tok/s (effective) Tokens / solved Quality / $ Quality / min Calls Errors
luna-none 227 12,957 355.6 3.313 2.8 0
solo-opus 106 5,660 16.3 2.478 1.0 0
panel-duo 210 13,345 328.8 2.767 2.8 0
auto-route 192 13,222 151.6 2.792 3.1 0

Streamed speed per model:

Arm Model Decode tok/s TTFT ms
luna-none claude-haiku 275 1850
luna-none gpt-luna 126 776
solo-opus claude-opus 142 3212
panel-duo claude-haiku 284 1733
panel-duo gpt-luna 240 4544
auto-route claude-haiku 262 1360
auto-route gemini-flash 143 4091
auto-route gpt-luna 223 4133

Cost of measuring

Scoring and judging cost and time, kept apart: never part of an arm's cost or latency.

Arm Eval $ total Eval $ / item Eval s / item
luna-none $0 $0 0.0s
solo-opus $0 $0 0.0s
panel-duo $0 $0 0.0s
auto-route $0 $0 0.0s

Variance across repeats

Arm Mean sd of quality across repeats Tasks whose solved flips
luna-none - -
solo-opus - -
panel-duo - -
auto-route - -

Comparisons against the baseline

luna-none vs solo-opus

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 30 -0.052 [-0.182, +0.075] 7/10/13 0.263 -0.14 0.04 [0.03, 0.06] 0.70 [0.63, 0.78] ✓ ✓ ? ?
architecture 7 -0.119 [-0.238, +0.024] 1/2/4 0.375 -0.64 0.04 [0.04, 0.05] 0.64 [0.55, 0.77] ? ? ? ?
code_review 8 -0.231 [-0.491, -0.006] 2/2/4 0.688 -0.63 0.15 [0.06, 0.37] 0.85 [0.67, 1.02] ? ? ? ?
debugging 8 +0.162 [-0.175, +0.500] 3/3/2 1.000 +0.32 0.03 [0.01, 0.05] 0.72 [0.56, 0.88] ? ? ? ?
planning 7 -0.024 [-0.122, +0.099] 1/3/3 0.625 -0.15 0.03 [0.02, 0.04] 0.65 [0.57, 0.73] ? ? ? ?
  • architecture (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.

panel-duo vs solo-opus

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 30 -0.065 [-0.178, +0.043] 5/15/10 0.302 -0.20 0.05 [0.04, 0.06] 0.82 [0.74, 0.91] ✓ ✓ ? ?
architecture 7 -0.119 [-0.238, -0.024] 0/4/3 0.250 -0.75 0.04 [0.04, 0.05] 0.78 [0.64, 0.92] ? ? ? ?
code_review 8 -0.201 [-0.485, +0.040] 2/3/3 1.000 -0.52 0.10 [0.05, 0.35] 0.85 [0.65, 1.05] ? ? ? ?
debugging 8 +0.099 [-0.206, +0.366] 3/3/2 1.000 +0.23 0.03 [0.01, 0.07] 0.94 [0.71, 1.17] ? ? ? ?
planning 7 -0.041 [-0.082, +0.000] 0/5/2 0.500 -0.59 0.04 [0.04, 0.05] 0.76 [0.70, 0.81] ? ? ? ?
  • architecture (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.

auto-route vs solo-opus

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 30 -0.134 [-0.259, -0.009] 5/11/14 0.064 -0.38 0.11 [0.08, 0.15] 0.74 [0.64, 0.83] ✓ ✓ ? ✗
architecture 7 -0.152 [-0.286, -0.048] 0/3/4 0.125 -0.89 0.12 [0.09, 0.20] 0.87 [0.74, 1.04] ? ? ? ?
code_review 8 -0.275 [-0.554, -0.004] 2/1/5 0.453 -0.62 0.23 0.69 [0.48, 0.97] ? ? ? ?
debugging 8 +0.024 [-0.314, +0.297] 3/4/1 0.625 +0.05 0.05 [0.02, 0.12] 0.45 [0.35, 0.58] ? ? ? ?
planning 7 -0.137 [-0.235, -0.048] 0/3/4 0.125 -0.97 0.11 [0.11, 0.12] 0.87 [0.76, 0.96] ? ? ? ?
  • architecture (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.
  • code_review (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
  • debugging (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
  • planning (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.

Δ quality is the arm minus the baseline, paired by task. Ratios are the arm over the baseline (below 1 is better). W/T/L counts tasks the arm won, tied (within 0.02) or lost. Effect d is the mean paired difference over its spread across tasks.

By category

architecture

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
luna-none 7 0.748 [0.600, 0.881] 86% $0.0025 0.0029 15.6s 18.6s
solo-opus 7 0.867 [0.695, 0.976] 86% $0.0594 0.0693 23.1s 39.9s
panel-duo 7 0.748 [0.614, 0.881] 86% $0.0026 0.0030 19.1s 22.9s
auto-route 7 0.714 [0.548, 0.881] 71% $0.0060 0.0084 21.0s 27.9s

code_review

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
luna-none 8 0.430 [0.264, 0.635] 25% $0.0017 0.0067 12.1s 14.4s
solo-opus 8 0.661 [0.427, 0.875] 75% $0.0338 0.0451 11.9s 16.6s
panel-duo 8 0.460 [0.281, 0.654] 38% $0.0018 0.0047 11.5s 15.2s
auto-route 8 0.387 [0.211, 0.607] 25% $0.0026 0.0105 9.0s 15.8s

debugging

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
luna-none 8 0.900 [0.725, 1.000] 88% $0.0019 0.0022 13.2s 17.4s
solo-opus 8 0.738 [0.475, 0.912] 62% $0.0441 0.0705 16.0s 22.0s
panel-duo 8 0.836 [0.650, 0.975] 88% $0.0020 0.0023 15.5s 23.5s
auto-route 8 0.761 [0.537, 0.948] 75% $0.0026 0.0034 6.5s 10.9s

planning

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
luna-none 7 0.915 [0.867, 0.976] 100% $0.0024 0.0024 14.9s 18.9s
solo-opus 7 0.939 [0.816, 1.000] 86% $0.0599 0.0698 23.0s 26.3s
panel-duo 7 0.898 [0.776, 0.980] 86% $0.0026 0.0030 18.2s 20.2s
auto-route 7 0.802 [0.645, 0.952] 86% $0.0067 0.0078 22.6s 24.3s

By difficulty

easy

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
luna-none 7 0.881 [0.762, 0.976] 100% $0.0021 0.0021 14.7s 16.7s
solo-opus 7 0.729 [0.505, 0.929] 71% $0.0482 0.0675 18.7s 26.1s
panel-duo 7 0.811 [0.730, 0.891] 100% $0.0023 0.0023 19.0s 22.9s
auto-route 7 0.805 [0.647, 0.940] 71% $0.0043 0.0061 18.1s 27.9s

hard

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
luna-none 7 0.711 [0.457, 0.932] 71% $0.0023 0.0032 14.4s 18.9s
solo-opus 7 0.571 [0.314, 0.820] 43% $0.0462 0.1079 18.5s 23.3s
panel-duo 7 0.658 [0.448, 0.867] 57% $0.0022 0.0038 16.8s 23.5s
auto-route 7 0.545 [0.361, 0.714] 57% $0.0039 0.0069 12.6s 20.5s

medium

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
luna-none 16 0.696 [0.545, 0.832] 62% $0.0020 0.0032 12.4s 16.6s
solo-opus 16 0.921 [0.825, 1.000] 94% $0.0498 0.0531 17.5s 26.3s
panel-duo 16 0.725 [0.575, 0.880] 69% $0.0022 0.0032 15.5s 20.5s
auto-route 16 0.647 [0.456, 0.817] 62% $0.0045 0.0073 11.0s 24.3s

Methodology

  • Run: v3-latency (completed), created 2026-11-19 10:33:44
  • Dataset: evals/datasets/v2, split dev, 30 tasks, hash de8a9a49c422425a
  • Arm solo-opus: strategy solo-opus: solo, models claude-opus, aggregator auto, rounds 1, judge off
  • Arm auto-route: strategy auto: auto, models claude-haiku, gpt-luna, gemini-flash, gpt-terra, gemini-hydrogen, claude-sonnet, aggregator auto, rounds 1, judge off
  • Arm panel-duo: strategy panel-duo: panel, models claude-haiku, gpt-luna, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
  • Arm luna-none: strategy panel-duo, overrides {'members': [{'model': 'claude-haiku'}, {'model': 'gpt-luna', 'reasoning_effort': 'none'}]}: panel, models claude-haiku, gpt-luna, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
  • Model claude-opus: anthropic claude-opus-5-5: $4 in / $20 out per 1M tokens, verified 2026-11-17
  • Model claude-haiku: anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Model gpt-luna: openai gpt-6-luna: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Model gemini-flash: google gemini-3.8-flash: $0.75 in / $3.75 out per 1M tokens, verified 2026-11-17
  • Model gpt-terra: openai gpt-6.1-terra: $1 in / $5 out per 1M tokens, verified 2026-11-17
  • Model gemini-hydrogen: google gemini-4-hydrogen: $1.5 in / $7.5 out per 1M tokens, verified 2026-11-17
  • Model claude-sonnet: anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
  • Judge models: none (deterministic scorers only)
  • Repeats / seed: 1 per task / seed 0
  • Intervals: 95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
  • Verdict rules: non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline solo-opus
  • Spend: $1.72 on the arms + $0 on scoring and judging (cap $2.35)
  • Fusion version: 0.2.0 (git 5abab47, with uncommitted changes)

Supporting study: v3-e2e

Arms: sonnet, opus, sonnet+fusion, opus+fusion, haiku, haiku+fusion. Baseline for the comparisons: sonnet.

1 repeat(s) per task: variance across repeats is poorly estimated. Fewer than 10 tasks are answered by both an arm and the baseline: no verdict is possible for it.

Headline

Each arm against sonnet, on all tasks. ✓ yes, ✗ no, ? inconclusive, ⊘ blocked. The reasons are listed below the table.

Arm Paired tasks Cheaper (per solved task) Faster Not worse Better
opus 4 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
sonnet+fusion 4 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
opus+fusion 4 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
haiku 4 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
haiku+fusion 4 ? inconclusive ? inconclusive ? inconclusive ? inconclusive
  • opus vs sonnet (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
  • sonnet+fusion vs sonnet (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
  • opus+fusion vs sonnet (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
  • haiku vs sonnet (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
  • haiku+fusion vs sonnet (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.

Arms

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90 Frontier
sonnet 4 1.000 [1.000, 1.000] 100% $0.0256 0.0256 12.9s 18.9s s
opus 4 1.000 [1.000, 1.000] 100% $0.0678 0.0678 25.6s 27.4s -
sonnet+fusion 4 1.000 [1.000, 1.000] 100% $0.0504 0.0504 13.3s 24.6s -
opus+fusion 4 1.000 [1.000, 1.000] 100% $0.1221 0.1221 23.1s 43.0s -
haiku 4 0.992 [0.976, 1.000] 75% $0.0027 0.0035 17.2s 22.7s $
haiku+fusion 4 1.000 [1.000, 1.000] 100% $0.0058 0.0058 21.5s 34.8s $

Frontier: $ = on the cost-quality frontier (no arm is cheaper per task and at least as good), s = on the latency-quality frontier (median seconds).

Efficiency and speed

Arm Out tok/s (effective) Tokens / solved Quality / $ Quality / min Calls Errors
sonnet - 21,748 39.0 3.834 5.2 0
opus - 21,376 14.7 2.361 5.8 0
sonnet+fusion - 96,689 19.9 3.545 4.8 0
opus+fusion - 105,501 8.2 2.028 5.8 0
haiku - 44,076 374.1 3.187 6.0 0
haiku+fusion - 169,798 173.4 2.216 7.8 0

Cost of measuring

Scoring and judging cost and time, kept apart: never part of an arm's cost or latency.

Arm Eval $ total Eval $ / item Eval s / item
sonnet $0 $0 0.7s
opus $0 $0 0.7s
sonnet+fusion $0 $0 0.6s
opus+fusion $0 $0 1.4s
haiku $0 $0 1.2s
haiku+fusion $0 $0 0.6s

Variance across repeats

Arm Mean sd of quality across repeats Tasks whose solved flips
sonnet - -
opus - -
sonnet+fusion - -
opus+fusion - -
haiku - -
haiku+fusion - -

Claude Code end to end

Each run is headless Claude Code on a temp copy of the task; the hidden tests decide solved. Costs are means per run: the host's from Claude Code's own JSON, Fusion's from its ledger.

Arm Executor Fusion Runs Solved Host $ Fusion $ Total $ $/solved p50 Turns Tool calls Fusion calls
sonnet sonnet off 4 100% $0.0256 $0 $0.0256 $0.0256 12.9s 5.2 4.2 0.0
opus opus off 4 100% $0.0678 $0 $0.0678 $0.0678 25.6s 5.8 4.8 0.0
sonnet+fusion sonnet on 4 100% $0.0504 $0 $0.0504 $0.0504 13.3s 4.8 3.8 0.0
opus+fusion opus on 4 100% $0.1218 $0.0003 $0.1221 $0.1221 23.1s 5.5 4.5 0.2
haiku haiku off 4 75% $0.0027 $0 $0.0027 $0.0035 17.2s 6.0 5.0 0.0
haiku+fusion haiku on 4 100% $0.0055 $0.0003 $0.0058 $0.0058 21.5s 7.5 6.5 0.2

Comparisons against the baseline

opus vs sonnet

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 4 +0.000 [+0.000, +0.000] 0/4/0 - - 2.65 [2.26, 3.31] 1.62 [1.38, 2.00] ? ? ? ?
coding 4 +0.000 [+0.000, +0.000] 0/4/0 - - 2.65 [2.26, 3.31] 1.62 [1.41, 2.00] ? ? ? ?
  • coding (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.

sonnet+fusion vs sonnet

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 4 +0.000 [+0.000, +0.000] 0/4/0 - - 1.97 [1.06, 2.45] 1.08 [0.94, 1.23] ? ? ? ?
coding 4 +0.000 [+0.000, +0.000] 0/4/0 - - 1.97 [1.14, 2.45] 1.08 [0.94, 1.23] ? ? ? ?
  • coding (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.

opus+fusion vs sonnet

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 4 +0.000 [+0.000, +0.000] 0/4/0 - - 4.76 [3.17, 6.17] 1.89 [1.67, 2.19] ? ? ? ?
coding 4 +0.000 [+0.000, +0.000] 0/4/0 - - 4.76 [3.17, 6.17] 1.89 [1.67, 2.19] ? ? ? ?
  • coding (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.

haiku vs sonnet

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 4 -0.008 [-0.024, +0.000] 0/3/1 1.000 -0.50 0.14 [0.10, 0.40] 1.19 [1.10, 1.35] ? ? ? ?
coding 4 -0.008 [-0.024, +0.000] 0/3/1 1.000 -0.50 0.14 [0.10, 0.40] 1.19 [1.10, 1.35] ? ? ? ?
  • coding (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.

haiku+fusion vs sonnet

Scope Tasks Δ quality (95% CI) W/T/L Sign test p Effect d $/solved ratio Seconds ratio Cheaper Faster Not worse Better
all 4 +0.000 [+0.000, +0.000] 0/4/0 - - 0.23 [0.17, 0.27] 1.73 [1.56, 1.83] ? ? ? ?
coding 4 +0.000 [+0.000, +0.000] 0/4/0 - - 0.23 [0.17, 0.27] 1.73 [1.56, 1.83] ? ? ? ?
  • coding (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.

Δ quality is the arm minus the baseline, paired by task. Ratios are the arm over the baseline (below 1 is better). W/T/L counts tasks the arm won, tied (within 0.02) or lost. Effect d is the mean paired difference over its spread across tasks.

By category

coding

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
sonnet 4 1.000 [1.000, 1.000] 100% $0.0256 0.0256 12.9s 18.9s
opus 4 1.000 [1.000, 1.000] 100% $0.0678 0.0678 25.6s 27.4s
sonnet+fusion 4 1.000 [1.000, 1.000] 100% $0.0504 0.0504 13.3s 24.6s
opus+fusion 4 1.000 [1.000, 1.000] 100% $0.1221 0.1221 23.1s 43.0s
haiku 4 0.992 [0.976, 1.000] 75% $0.0027 0.0035 17.2s 22.7s
haiku+fusion 4 1.000 [1.000, 1.000] 100% $0.0058 0.0058 21.5s 34.8s

By difficulty

easy

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
sonnet 2 1.000 [1.000, 1.000] 100% $0.0240 0.0240 12.9s 18.9s
opus 2 1.000 [1.000, 1.000] 100% $0.0635 0.0635 21.3s 25.6s
sonnet+fusion 2 1.000 [1.000, 1.000] 100% $0.0533 0.0533 11.7s 24.6s
opus+fusion 2 1.000 [1.000, 1.000] 100% $0.0763 0.0763 23.1s 31.1s
haiku 2 0.984 [0.968, 1.000] 50% $0.0026 0.0052 14.6s 22.7s
haiku+fusion 2 1.000 [1.000, 1.000] 100% $0.0064 0.0064 18.7s 34.8s

medium

Arm Tasks Quality (95% CI) Solved $/task $/solved p50 p90
sonnet 2 1.000 [1.000, 1.000] 100% $0.0272 0.0272 12.0s 18.8s
opus 2 1.000 [1.000, 1.000] 100% $0.0722 0.0722 27.4s 27.4s
sonnet+fusion 2 1.000 [1.000, 1.000] 100% $0.0474 0.0474 13.3s 18.1s
opus+fusion 2 1.000 [1.000, 1.000] 100% $0.1678 0.1678 21.2s 43.0s
haiku 2 1.000 [1.000, 1.000] 100% $0.0027 0.0027 17.2s 20.2s
haiku+fusion 2 1.000 [1.000, 1.000] 100% $0.0052 0.0052 21.5s 33.3s

Methodology

  • Run: v3-e2e (completed), created 2026-11-19 10:59:06
  • Dataset: evals/datasets/v2, split test, 4 tasks, hash a20ac5229b42b8bb
  • Arm opus: claude-code opus, model claude-opus
  • Arm opus+fusion: claude-code opus + fusion, model claude-opus
  • Arm sonnet: claude-code sonnet, model claude-sonnet
  • Arm sonnet+fusion: claude-code sonnet + fusion, model claude-sonnet
  • Arm haiku: claude-code haiku, model claude-haiku
  • Arm haiku+fusion: claude-code haiku + fusion, model claude-haiku
  • Model claude-opus: anthropic claude-opus-5-5: $4 in / $20 out per 1M tokens, verified 2026-11-17
  • Model claude-sonnet: anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
  • Model claude-haiku: anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
  • Judge models: none (deterministic scorers only)
  • Repeats / seed: 1 per task / seed 0
  • Intervals: 95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
  • Verdict rules: non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline sonnet
  • Spend: $1.10 on the arms + $0 on scoring and judging (cap $4.00)
  • Fusion version: 0.2.0 (git 5abab47, with uncommitted changes)

Spend

Run Live spend
bench-e2e $1.20
calibration $0.1968
task-4 $0.0000
tune $0.0005
tune-20261118-183817-01-tune-cc96c980 $0.2920
tune-20261118-183817-02-tune-6ffc6f08 $0.0727
tune-20261118-183817-03-tune-237ca193 $0.0045
v3-ablation $2.08
v3-headline $4.42
v3-latency $1.72
v3-route $1.58
v3-route-plus $0.1140
Total $11.68 of the $15 cap

Limitations

  • Most tasks are synthetic, and no person has reviewed them yet. An LLM drafted the text, review and debugging tasks; 34 coding tasks are real upstream bugs of permissively licensed projects, with prompts written for the study. See evals/datasets/README.md. They test whether an answer finds the seeded defect, names the root cause or passes the hidden tests, not how an answer feels in a real project.
  • It is a small study. The $15 budget caps it at a few dozen tasks with one repeat, and the executable categories (coding, frontend, performance) at two or three tasks each in the headline, so the intervals are wide and many verdicts are inconclusive. That is a result: the study could not tell, which is not the same as the arms being equal.
  • The scorers decide the numbers. Review, debugging and rubric scores come from keyword and structure matching, with neutral LLM judges (from vendors on no arm) for what those cannot settle; a judge may still favour a style. Frontend and performance scores are measured (tests, timings, accessibility), not judged by a model, and they check structure and speed, not intent or taste; browser checks that need Playwright were skipped where it was not installed, the same for every arm.
  • Solo arms are not always one model. With hedging on (the shipped default), a solo arm whose model is slow also asks a same-tier model and keeps the first answer; the arms' warnings and call ledgers say when that happened.
  • Prices and speeds are those of the day. Costs use the catalog's prices (dates in the methodology); latencies depend on provider load at the time of the run, and a job that timed out was run again with a longer timeout (only the last attempt is scored).
  • One dataset, one set of models. A different task mix, a newer model or another judge can change a verdict. The tuning used the dev split and this report is from the held-out test split, but both come from the same author and generator.
  • The end-to-end runs are few. The Claude Code A/B covers four coding tasks per arm: enough to see costs, turns and tool calls inside a real session, not to rank the arms.

Reproduce

uv run fusion bench report v3-headline --format html
uv run fusion bench results v3-headline --also v3-ablation --also v3-route --also v3-latency --also v3-e2e --html docs/benchmark-report.html --svg docs/assets/benchmark-cost-quality.svg