Benchmark results
Generated on 2026-11-19 from run v3-headline. The verdicts below are the verdict engine's text, unedited; how to read them is in BENCHMARKING.md.
The full report with its charts is benchmark-report.html.
2 job(s) halted on their first attempt (a call timed out or the panel had no quorum) and were run again, with longer timeouts where it was a timeout: solo-opus (2). Only the last attempt is scored; the first attempt's cost is in the totals. A halted job scores 0, so without the retry these arms would be understated by the harness's limit, not by their answers.
Some prices were unknown for solo-luna, solo-sol, fusion-default, fusion-best: its costs are a lower bound.
Items replayed from the response cache are left out of the latency figures (solo-luna, fusion-default, fusion-best): their wall time is not a measurement.
1 repeat(s) per task: variance across repeats is poorly estimated.
Frontend and performance scores measure what the answer builds (tests, timings, screenshots); they do not judge whether it matches the author's intent.
Headline
Each arm against solo-opus, on all tasks. ✓ yes, ✗ no, ? inconclusive, ⊘ blocked. The reasons are listed below the table.
| Arm |
Paired tasks |
Cheaper (per solved task) |
Faster |
Not worse |
Better |
| solo-haiku |
47 |
✓ yes |
✓ yes |
? inconclusive |
? inconclusive |
| solo-luna |
47 |
✓ yes |
✓ yes |
? inconclusive |
? inconclusive |
| solo-sol |
47 |
✓ yes |
? inconclusive |
✗ no |
✗ no |
| fusion-default |
47 |
✓ yes |
✓ yes |
? inconclusive |
? inconclusive |
| fusion-best |
47 |
✓ yes |
✓ yes |
? inconclusive |
? inconclusive |
- solo-haiku vs solo-opus on all (47 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.01 [0.01, 0.02] (below 1 is better): the whole interval is below 1.
- solo-haiku vs solo-opus on all (47 tasks): faster: yes. seconds per task ratio 0.32 [0.29, 0.35] (below 1 is better): the whole interval is below 1.
- solo-haiku vs solo-opus on all (47 tasks): not worse: inconclusive. quality difference +0.053 [-0.052, +0.147] over 47 tasks: the interval straddles the -0.03 margin.
- solo-haiku vs solo-opus on all (47 tasks): better: inconclusive. quality difference +0.053 [-0.052, +0.147] over 47 tasks: the interval includes 0.
- solo-luna vs solo-opus on all (47 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.01 [0.01, 0.02] (below 1 is better): the whole interval is below 1.
- solo-luna vs solo-opus on all (47 tasks): faster: yes. seconds per task ratio 0.48 [0.44, 0.52] (below 1 is better): the whole interval is below 1.
- solo-luna vs solo-opus on all (47 tasks): not worse: inconclusive. quality difference -0.053 [-0.144, +0.047] over 47 tasks: the interval straddles the -0.03 margin.
- solo-luna vs solo-opus on all (47 tasks): better: inconclusive. quality difference -0.053 [-0.144, +0.047] over 47 tasks: the interval includes 0.
- solo-sol vs solo-opus on all (47 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.37 [0.25, 0.53] (below 1 is better): the whole interval is below 1.
- solo-sol vs solo-opus on all (47 tasks): faster: inconclusive. seconds per task ratio 0.96 [0.89, 1.03] (below 1 is better): the interval includes 1.
- solo-sol vs solo-opus on all (47 tasks): not worse: no. quality difference -0.112 [-0.191, -0.034] over 47 tasks: the upper bound is below -0.03: it loses more than the margin.
- solo-sol vs solo-opus on all (47 tasks): better: no. quality difference -0.112 [-0.191, -0.034] over 47 tasks: the interval is at or below 0.
- fusion-default vs solo-opus on all (47 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.11 [0.05, 0.23] (below 1 is better): the whole interval is below 1.
- fusion-default vs solo-opus on all (47 tasks): faster: yes. seconds per task ratio 0.69 [0.59, 0.78] (below 1 is better): the whole interval is below 1.
- fusion-default vs solo-opus on all (47 tasks): not worse: inconclusive. quality difference +0.022 [-0.072, +0.117] over 47 tasks: the interval straddles the -0.03 margin.
- fusion-default vs solo-opus on all (47 tasks): better: inconclusive. quality difference +0.022 [-0.072, +0.117] over 47 tasks: the interval includes 0.
- fusion-best vs solo-opus on all (47 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.06 [0.04, 0.09] (below 1 is better): the whole interval is below 1.
- fusion-best vs solo-opus on all (47 tasks): faster: yes. seconds per task ratio 0.50 [0.47, 0.53] (below 1 is better): the whole interval is below 1.
- fusion-best vs solo-opus on all (47 tasks): not worse: inconclusive. quality difference +0.009 [-0.079, +0.098] over 47 tasks: the interval straddles the -0.03 margin.
- fusion-best vs solo-opus on all (47 tasks): better: inconclusive. quality difference +0.009 [-0.079, +0.098] over 47 tasks: the interval includes 0.
Arms
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
Frontier |
| solo-haiku |
47 |
0.643 [0.548, 0.731] |
57% |
$0.0009 |
0.0016 |
6.7s |
8.8s |
$ s |
| solo-luna |
47 |
0.537 [0.432, 0.648] |
49% |
$0.0007 |
0.0014 |
9.7s |
14.0s |
$ |
| solo-opus |
47 |
0.590 [0.500, 0.679] |
43% |
$0.0544 |
0.1279 |
20.7s |
29.9s |
- |
| solo-sol |
47 |
0.478 [0.380, 0.580] |
40% |
$0.0191 |
0.0472 |
20.7s |
28.9s |
- |
| fusion-default |
47 |
0.612 [0.519, 0.702] |
53% |
$0.0073 |
0.0137 |
13.2s |
23.6s |
- |
| fusion-best |
47 |
0.599 [0.492, 0.705] |
53% |
$0.0042 |
0.0078 |
11.2s |
13.7s |
- |
Frontier: $ = on the cost-quality frontier (no arm is cheaper per task and at least as good), s = on the latency-quality frontier (median seconds).
Efficiency and speed
| Arm |
Out tok/s (effective) |
Tokens / solved |
Quality / $ |
Quality / min |
Calls |
Errors |
| solo-haiku |
208 |
6,805 |
707.7 |
5.834 |
1.0 |
0 |
| solo-luna |
103 |
5,052 |
803.5 |
3.342 |
1.1 |
0 |
| solo-opus |
110 |
11,327 |
10.8 |
1.690 |
1.0 |
0 |
| solo-sol |
81 |
7,420 |
25.1 |
1.430 |
1.6 |
0 |
| fusion-default |
207 |
17,310 |
83.8 |
2.580 |
3.1 |
0 |
| fusion-best |
237 |
17,434 |
144.1 |
3.202 |
4.0 |
0 |
Streamed speed per model:
| Arm |
Model |
Decode tok/s |
TTFT ms |
| solo-haiku |
claude-haiku |
266 |
1276 |
| solo-luna |
gpt-luna |
199 |
4094 |
| solo-opus |
claude-opus |
134 |
3356 |
| solo-sol |
claude-sonnet |
360 |
15577 |
| solo-sol |
gpt-sol |
104 |
3883 |
| fusion-default |
claude-haiku |
248 |
1046 |
| fusion-default |
gemini-flash |
144 |
3959 |
| fusion-default |
gpt-luna |
206 |
4178 |
| fusion-best |
claude-haiku |
259 |
1302 |
| fusion-best |
gemini-flash |
161 |
2816 |
| fusion-best |
gpt-luna |
215 |
3632 |
Cost of measuring
Scoring and judging cost and time, kept apart: never part of an arm's cost or latency.
| Arm |
Eval $ total |
Eval $ / item |
Eval s / item |
| solo-haiku |
$0.0653 |
$0.0014 |
4.3s |
| solo-luna |
$0.0603 |
$0.0013 |
4.2s |
| solo-opus |
$0.0765 |
$0.0016 |
4.7s |
| solo-sol |
$0.0707 |
$0.0015 |
4.0s |
| fusion-default |
$0.0606 |
$0.0013 |
3.3s |
| fusion-best |
$0.0595 |
$0.0013 |
3.2s |
Variance across repeats
| Arm |
Mean sd of quality across repeats |
Tasks whose solved flips |
| solo-haiku |
- |
- |
| solo-luna |
- |
- |
| solo-opus |
- |
- |
| solo-sol |
- |
- |
| fusion-default |
- |
- |
| fusion-best |
- |
- |
Comparisons against the baseline
solo-haiku vs solo-opus
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
47 |
+0.053 [-0.052, +0.147] |
23/12/12 |
0.090 |
+0.16 |
0.01 [0.01, 0.02] |
0.32 [0.29, 0.35] |
✓ |
✓ |
? |
? |
| architecture |
10 |
+0.130 [-0.050, +0.310] |
7/0/3 |
0.344 |
+0.41 |
0.00 |
0.34 [0.31, 0.36] |
? |
✓ |
? |
? |
| code_review |
10 |
+0.179 [+0.060, +0.313] |
8/1/1 |
0.039 |
+0.85 |
0.01 [0.00, 0.02] |
0.34 [0.29, 0.40] |
✓ |
✓ |
✓ |
✓ |
| coding |
3 |
-0.074 [-0.222, +0.000] |
0/2/1 |
1.000 |
-0.58 |
0.03 [0.02, 0.07] |
0.38 [0.25, 0.58] |
? |
? |
? |
? |
| debugging |
10 |
+0.000 [-0.084, +0.084] |
2/6/2 |
1.000 |
+0.00 |
0.02 [0.01, 0.02] |
0.32 [0.28, 0.38] |
✓ |
✓ |
? |
? |
| frontend |
2 |
-0.464 [-0.929, +0.000] |
0/1/1 |
1.000 |
-0.71 |
- |
0.27 [0.25, 0.29] |
? |
? |
? |
? |
| performance |
2 |
+0.016 [+0.004, +0.029] |
1/1/0 |
1.000 |
+0.91 |
0.01 [0.01, 0.03] |
0.26 [0.11, 0.58] |
? |
? |
? |
? |
| planning |
10 |
+0.050 [-0.261, +0.353] |
5/1/4 |
1.000 |
+0.10 |
0.01 [0.00, 0.03] |
0.29 [0.26, 0.33] |
✓ |
✓ |
? |
? |
- solo-haiku vs solo-opus on architecture (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.00; too few tasks for an interval.
- solo-haiku vs solo-opus on architecture (10 tasks): faster: yes. seconds per task ratio 0.34 [0.31, 0.36] (below 1 is better): the whole interval is below 1.
- solo-haiku vs solo-opus on architecture (10 tasks): not worse: inconclusive. quality difference +0.130 [-0.050, +0.310] over 10 tasks: the interval straddles the -0.03 margin.
- solo-haiku vs solo-opus on architecture (10 tasks): better: inconclusive. quality difference +0.130 [-0.050, +0.310] over 10 tasks: the interval includes 0.
- solo-haiku vs solo-opus on code_review (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.01 [0.00, 0.02] (below 1 is better): the whole interval is below 1.
- solo-haiku vs solo-opus on code_review (10 tasks): faster: yes. seconds per task ratio 0.34 [0.29, 0.40] (below 1 is better): the whole interval is below 1.
- solo-haiku vs solo-opus on code_review (10 tasks): not worse: yes. quality difference +0.179 [+0.060, +0.313] over 10 tasks: the lower bound is above -0.03, the non-inferiority margin.
- solo-haiku vs solo-opus on code_review (10 tasks): better: yes. quality difference +0.179 [+0.060, +0.313] over 10 tasks: the interval is above 0.
- coding (3 tasks): only 3 task(s) answered by both arms; at least 10 are needed to claim anything.
- solo-haiku vs solo-opus on debugging (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.02 [0.01, 0.02] (below 1 is better): the whole interval is below 1.
- solo-haiku vs solo-opus on debugging (10 tasks): faster: yes. seconds per task ratio 0.32 [0.28, 0.38] (below 1 is better): the whole interval is below 1.
- solo-haiku vs solo-opus on debugging (10 tasks): not worse: inconclusive. quality difference +0.000 [-0.084, +0.084] over 10 tasks: the interval straddles the -0.03 margin.
- solo-haiku vs solo-opus on debugging (10 tasks): better: inconclusive. quality difference +0.000 [-0.084, +0.084] over 10 tasks: the interval includes 0.
- frontend (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- solo-haiku vs solo-opus on planning (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.01 [0.00, 0.03] (below 1 is better): the whole interval is below 1.
- solo-haiku vs solo-opus on planning (10 tasks): faster: yes. seconds per task ratio 0.29 [0.26, 0.33] (below 1 is better): the whole interval is below 1.
- solo-haiku vs solo-opus on planning (10 tasks): not worse: inconclusive. quality difference +0.050 [-0.261, +0.353] over 10 tasks: the interval straddles the -0.03 margin.
- solo-haiku vs solo-opus on planning (10 tasks): better: inconclusive. quality difference +0.050 [-0.261, +0.353] over 10 tasks: the interval includes 0.
solo-luna vs solo-opus
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
47 |
-0.053 [-0.144, +0.047] |
12/14/21 |
0.163 |
-0.16 |
0.01 [0.01, 0.02] |
0.48 [0.44, 0.52] |
✓ |
✓ |
? |
? |
| architecture |
10 |
-0.123 [-0.250, -0.017] |
1/3/6 |
0.125 |
-0.65 |
0.01 |
0.45 [0.40, 0.52] |
? |
✓ |
? |
✗ |
| code_review |
10 |
+0.271 [+0.139, +0.404] |
7/3/0 |
0.016 |
+1.19 |
0.00 [0.00, 0.01] |
0.39 [0.35, 0.44] |
✓ |
✓ |
✓ |
✓ |
| coding |
3 |
+0.000 [+0.000, +0.000] |
0/3/0 |
- |
- |
0.01 [0.01, 0.01] |
0.40 [0.36, 0.44] |
? |
? |
? |
? |
| debugging |
10 |
-0.060 [-0.286, +0.191] |
2/2/6 |
0.289 |
-0.15 |
0.01 [0.01, 0.02] |
0.60 [0.52, 0.71] |
✓ |
✓ |
? |
? |
| frontend |
2 |
-0.464 [-0.929, +0.000] |
0/1/1 |
1.000 |
-0.71 |
- |
- |
? |
? |
? |
? |
| performance |
2 |
-0.005 [-0.007, -0.004] |
0/2/0 |
- |
-2.12 |
0.01 [0.01, 0.01] |
0.32 [0.27, 0.45] |
? |
? |
? |
? |
| planning |
10 |
-0.240 [-0.393, -0.082] |
2/0/8 |
0.109 |
-0.88 |
- |
0.50 [0.42, 0.57] |
? |
✓ |
✗ |
✗ |
- solo-luna vs solo-opus on architecture (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.01; too few tasks for an interval.
- solo-luna vs solo-opus on architecture (10 tasks): faster: yes. seconds per task ratio 0.45 [0.40, 0.52] (below 1 is better): the whole interval is below 1.
- solo-luna vs solo-opus on architecture (10 tasks): not worse: inconclusive. quality difference -0.123 [-0.250, -0.017] over 10 tasks: the interval straddles the -0.03 margin.
- solo-luna vs solo-opus on architecture (10 tasks): better: no. quality difference -0.123 [-0.250, -0.017] over 10 tasks: the interval is at or below 0.
- solo-luna vs solo-opus on code_review (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.00 [0.00, 0.01] (below 1 is better): the whole interval is below 1.
- solo-luna vs solo-opus on code_review (10 tasks): faster: yes. seconds per task ratio 0.39 [0.35, 0.44] (below 1 is better): the whole interval is below 1.
- solo-luna vs solo-opus on code_review (10 tasks): not worse: yes. quality difference +0.271 [+0.139, +0.404] over 10 tasks: the lower bound is above -0.03, the non-inferiority margin.
- solo-luna vs solo-opus on code_review (10 tasks): better: yes. quality difference +0.271 [+0.139, +0.404] over 10 tasks: the interval is above 0.
- coding (3 tasks): only 3 task(s) answered by both arms; at least 10 are needed to claim anything.
- solo-luna vs solo-opus on debugging (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.01 [0.01, 0.02] (below 1 is better): the whole interval is below 1.
- solo-luna vs solo-opus on debugging (10 tasks): faster: yes. seconds per task ratio 0.60 [0.52, 0.71] (below 1 is better): the whole interval is below 1.
- solo-luna vs solo-opus on debugging (10 tasks): not worse: inconclusive. quality difference -0.060 [-0.286, +0.191] over 10 tasks: the interval straddles the -0.03 margin.
- solo-luna vs solo-opus on debugging (10 tasks): better: inconclusive. quality difference -0.060 [-0.286, +0.191] over 10 tasks: the interval includes 0.
- frontend (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- solo-luna vs solo-opus on planning (10 tasks): cheaper (per solved task): inconclusive. an arm solved no task, so its cost per solved task is undefined.
- solo-luna vs solo-opus on planning (10 tasks): faster: yes. seconds per task ratio 0.50 [0.42, 0.57] (below 1 is better): the whole interval is below 1.
- solo-luna vs solo-opus on planning (10 tasks): not worse: no. quality difference -0.240 [-0.393, -0.082] over 10 tasks: the upper bound is below -0.03: it loses more than the margin.
- solo-luna vs solo-opus on planning (10 tasks): better: no. quality difference -0.240 [-0.393, -0.082] over 10 tasks: the interval is at or below 0.
solo-sol vs solo-opus
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
47 |
-0.112 [-0.191, -0.034] |
8/11/28 |
0.001 |
-0.41 |
0.37 [0.25, 0.53] |
0.96 [0.89, 1.03] |
✓ |
? |
✗ |
✗ |
| architecture |
10 |
-0.160 [-0.307, +0.053] |
1/0/9 |
0.021 |
-0.52 |
0.40 |
1.04 [0.93, 1.14] |
? |
? |
? |
? |
| code_review |
10 |
-0.136 [-0.263, -0.017] |
2/1/7 |
0.180 |
-0.65 |
1.08 |
0.93 [0.79, 1.07] |
? |
? |
? |
✗ |
| coding |
3 |
+0.000 [+0.000, +0.000] |
0/3/0 |
- |
- |
0.30 [0.30, 0.32] |
1.03 [0.88, 1.21] |
? |
? |
? |
? |
| debugging |
10 |
-0.046 [-0.171, +0.095] |
2/3/5 |
0.453 |
-0.21 |
0.19 [0.11, 0.26] |
0.66 [0.58, 0.77] |
✓ |
✓ |
? |
? |
| frontend |
2 |
-0.005 [-0.010, +0.000] |
0/2/0 |
- |
-0.71 |
0.32 |
0.91 [0.79, 1.04] |
? |
? |
? |
? |
| performance |
2 |
-0.034 [-0.046, -0.021] |
0/0/2 |
0.500 |
-1.92 |
0.45 [0.42, 0.49] |
1.17 [1.00, 1.53] |
? |
? |
? |
? |
| planning |
10 |
-0.178 [-0.433, +0.046] |
3/2/5 |
0.727 |
-0.43 |
0.59 |
1.08 [0.96, 1.20] |
? |
? |
? |
? |
- solo-sol vs solo-opus on architecture (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.40; too few tasks for an interval.
- solo-sol vs solo-opus on architecture (10 tasks): faster: inconclusive. seconds per task ratio 1.04 [0.93, 1.14] (below 1 is better): the interval includes 1.
- solo-sol vs solo-opus on architecture (10 tasks): not worse: inconclusive. quality difference -0.160 [-0.307, +0.053] over 10 tasks: the interval straddles the -0.03 margin.
- solo-sol vs solo-opus on architecture (10 tasks): better: inconclusive. quality difference -0.160 [-0.307, +0.053] over 10 tasks: the interval includes 0.
- solo-sol vs solo-opus on code_review (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 1.08; too few tasks for an interval.
- solo-sol vs solo-opus on code_review (10 tasks): faster: inconclusive. seconds per task ratio 0.93 [0.79, 1.07] (below 1 is better): the interval includes 1.
- solo-sol vs solo-opus on code_review (10 tasks): not worse: inconclusive. quality difference -0.136 [-0.263, -0.017] over 10 tasks: the interval straddles the -0.03 margin.
- solo-sol vs solo-opus on code_review (10 tasks): better: no. quality difference -0.136 [-0.263, -0.017] over 10 tasks: the interval is at or below 0.
- coding (3 tasks): only 3 task(s) answered by both arms; at least 10 are needed to claim anything.
- solo-sol vs solo-opus on debugging (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.19 [0.11, 0.26] (below 1 is better): the whole interval is below 1.
- solo-sol vs solo-opus on debugging (10 tasks): faster: yes. seconds per task ratio 0.66 [0.58, 0.77] (below 1 is better): the whole interval is below 1.
- solo-sol vs solo-opus on debugging (10 tasks): not worse: inconclusive. quality difference -0.046 [-0.171, +0.095] over 10 tasks: the interval straddles the -0.03 margin.
- solo-sol vs solo-opus on debugging (10 tasks): better: inconclusive. quality difference -0.046 [-0.171, +0.095] over 10 tasks: the interval includes 0.
- frontend (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- solo-sol vs solo-opus on planning (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.59; too few tasks for an interval.
- solo-sol vs solo-opus on planning (10 tasks): faster: inconclusive. seconds per task ratio 1.08 [0.96, 1.20] (below 1 is better): the interval includes 1.
- solo-sol vs solo-opus on planning (10 tasks): not worse: inconclusive. quality difference -0.178 [-0.433, +0.046] over 10 tasks: the interval straddles the -0.03 margin.
- solo-sol vs solo-opus on planning (10 tasks): better: inconclusive. quality difference -0.178 [-0.433, +0.046] over 10 tasks: the interval includes 0.
fusion-default vs solo-opus
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
47 |
+0.022 [-0.072, +0.117] |
18/18/11 |
0.265 |
+0.07 |
0.11 [0.05, 0.23] |
0.69 [0.59, 0.78] |
✓ |
✓ |
? |
? |
| architecture |
10 |
+0.217 [+0.057, +0.373] |
8/1/1 |
0.039 |
+0.80 |
0.02 |
0.80 [0.66, 0.93] |
? |
✓ |
✓ |
✓ |
| code_review |
10 |
-0.019 [-0.138, +0.130] |
3/3/4 |
1.000 |
-0.08 |
0.11 |
0.48 [0.35, 0.67] |
? |
✓ |
? |
? |
| coding |
3 |
+0.000 [+0.000, +0.000] |
0/3/0 |
- |
- |
0.10 [0.01, 0.14] |
0.64 [0.27, 0.91] |
? |
? |
? |
? |
| debugging |
10 |
+0.056 [-0.042, +0.175] |
2/7/1 |
1.000 |
+0.29 |
0.04 [0.02, 0.07] |
0.42 [0.32, 0.54] |
✓ |
✓ |
? |
? |
| frontend |
2 |
-0.464 [-0.929, +0.000] |
0/1/1 |
1.000 |
-0.71 |
- |
0.96 |
? |
? |
? |
? |
| performance |
2 |
+0.007 [+0.004, +0.011] |
0/2/0 |
- |
+1.41 |
0.06 [0.01, 0.08] |
0.48 [0.23, 0.59] |
? |
? |
? |
? |
| planning |
10 |
-0.057 [-0.326, +0.214] |
5/1/4 |
1.000 |
-0.12 |
0.08 [0.02, 0.23] |
0.92 [0.75, 1.13] |
✓ |
? |
? |
? |
- fusion-default vs solo-opus on architecture (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.02; too few tasks for an interval.
- fusion-default vs solo-opus on architecture (10 tasks): faster: yes. seconds per task ratio 0.80 [0.66, 0.93] (below 1 is better): the whole interval is below 1.
- fusion-default vs solo-opus on architecture (10 tasks): not worse: yes. quality difference +0.217 [+0.057, +0.373] over 10 tasks: the lower bound is above -0.03, the non-inferiority margin.
- fusion-default vs solo-opus on architecture (10 tasks): better: yes. quality difference +0.217 [+0.057, +0.373] over 10 tasks: the interval is above 0.
- fusion-default vs solo-opus on code_review (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.11; too few tasks for an interval.
- fusion-default vs solo-opus on code_review (10 tasks): faster: yes. seconds per task ratio 0.48 [0.35, 0.67] (below 1 is better): the whole interval is below 1.
- fusion-default vs solo-opus on code_review (10 tasks): not worse: inconclusive. quality difference -0.019 [-0.138, +0.130] over 10 tasks: the interval straddles the -0.03 margin.
- fusion-default vs solo-opus on code_review (10 tasks): better: inconclusive. quality difference -0.019 [-0.138, +0.130] over 10 tasks: the interval includes 0.
- coding (3 tasks): only 3 task(s) answered by both arms; at least 10 are needed to claim anything.
- fusion-default vs solo-opus on debugging (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.04 [0.02, 0.07] (below 1 is better): the whole interval is below 1.
- fusion-default vs solo-opus on debugging (10 tasks): faster: yes. seconds per task ratio 0.42 [0.32, 0.54] (below 1 is better): the whole interval is below 1.
- fusion-default vs solo-opus on debugging (10 tasks): not worse: inconclusive. quality difference +0.056 [-0.042, +0.175] over 10 tasks: the interval straddles the -0.03 margin.
- fusion-default vs solo-opus on debugging (10 tasks): better: inconclusive. quality difference +0.056 [-0.042, +0.175] over 10 tasks: the interval includes 0.
- frontend (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- fusion-default vs solo-opus on planning (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.08 [0.02, 0.23] (below 1 is better): the whole interval is below 1.
- fusion-default vs solo-opus on planning (10 tasks): faster: inconclusive. seconds per task ratio 0.92 [0.75, 1.13] (below 1 is better): the interval includes 1.
- fusion-default vs solo-opus on planning (10 tasks): not worse: inconclusive. quality difference -0.057 [-0.326, +0.214] over 10 tasks: the interval straddles the -0.03 margin.
- fusion-default vs solo-opus on planning (10 tasks): better: inconclusive. quality difference -0.057 [-0.326, +0.214] over 10 tasks: the interval includes 0.
fusion-best vs solo-opus
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
47 |
+0.009 [-0.079, +0.098] |
17/14/16 |
1.000 |
+0.03 |
0.06 [0.04, 0.09] |
0.50 [0.47, 0.53] |
✓ |
✓ |
? |
? |
| architecture |
10 |
-0.110 [-0.290, +0.077] |
3/2/5 |
0.727 |
-0.35 |
0.07 |
0.48 [0.44, 0.52] |
? |
✓ |
? |
? |
| code_review |
10 |
+0.235 [+0.129, +0.338] |
8/1/1 |
0.039 |
+1.24 |
0.02 [0.00, 0.04] |
0.51 [0.45, 0.60] |
✓ |
✓ |
✓ |
✓ |
| coding |
3 |
+0.000 [+0.000, +0.000] |
0/3/0 |
- |
- |
0.08 [0.03, 0.12] |
0.58 [0.52, 0.66] |
? |
? |
? |
? |
| debugging |
10 |
-0.030 [-0.138, +0.069] |
2/5/3 |
1.000 |
-0.17 |
0.11 [0.09, 0.12] |
0.55 [0.41, 0.81] |
✓ |
✓ |
? |
? |
| frontend |
2 |
+0.524 [+0.062, +0.987] |
2/0/0 |
0.500 |
+0.80 |
0.02 |
0.54 [0.53, 0.55] |
? |
? |
? |
? |
| performance |
2 |
-0.020 [-0.032, -0.007] |
0/1/1 |
1.000 |
-1.11 |
0.03 [0.03, 0.05] |
0.41 [0.33, 0.61] |
? |
? |
? |
? |
| planning |
10 |
-0.155 [-0.376, +0.071] |
2/2/6 |
0.289 |
-0.41 |
0.28 |
0.49 [0.45, 0.54] |
? |
✓ |
? |
? |
- fusion-best vs solo-opus on architecture (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.07; too few tasks for an interval.
- fusion-best vs solo-opus on architecture (10 tasks): faster: yes. seconds per task ratio 0.48 [0.44, 0.52] (below 1 is better): the whole interval is below 1.
- fusion-best vs solo-opus on architecture (10 tasks): not worse: inconclusive. quality difference -0.110 [-0.290, +0.077] over 10 tasks: the interval straddles the -0.03 margin.
- fusion-best vs solo-opus on architecture (10 tasks): better: inconclusive. quality difference -0.110 [-0.290, +0.077] over 10 tasks: the interval includes 0.
- fusion-best vs solo-opus on code_review (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.02 [0.00, 0.04] (below 1 is better): the whole interval is below 1.
- fusion-best vs solo-opus on code_review (10 tasks): faster: yes. seconds per task ratio 0.51 [0.45, 0.60] (below 1 is better): the whole interval is below 1.
- fusion-best vs solo-opus on code_review (10 tasks): not worse: yes. quality difference +0.235 [+0.129, +0.338] over 10 tasks: the lower bound is above -0.03, the non-inferiority margin.
- fusion-best vs solo-opus on code_review (10 tasks): better: yes. quality difference +0.235 [+0.129, +0.338] over 10 tasks: the interval is above 0.
- coding (3 tasks): only 3 task(s) answered by both arms; at least 10 are needed to claim anything.
- fusion-best vs solo-opus on debugging (10 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.11 [0.09, 0.12] (below 1 is better): the whole interval is below 1.
- fusion-best vs solo-opus on debugging (10 tasks): faster: yes. seconds per task ratio 0.55 [0.41, 0.81] (below 1 is better): the whole interval is below 1.
- fusion-best vs solo-opus on debugging (10 tasks): not worse: inconclusive. quality difference -0.030 [-0.138, +0.069] over 10 tasks: the interval straddles the -0.03 margin.
- fusion-best vs solo-opus on debugging (10 tasks): better: inconclusive. quality difference -0.030 [-0.138, +0.069] over 10 tasks: the interval includes 0.
- frontend (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- fusion-best vs solo-opus on planning (10 tasks): cheaper (per solved task): inconclusive. cost per solved task ratio is 0.28; too few tasks for an interval.
- fusion-best vs solo-opus on planning (10 tasks): faster: yes. seconds per task ratio 0.49 [0.45, 0.54] (below 1 is better): the whole interval is below 1.
- fusion-best vs solo-opus on planning (10 tasks): not worse: inconclusive. quality difference -0.155 [-0.376, +0.071] over 10 tasks: the interval straddles the -0.03 margin.
- fusion-best vs solo-opus on planning (10 tasks): better: inconclusive. quality difference -0.155 [-0.376, +0.071] over 10 tasks: the interval includes 0.
Δ quality is the arm minus the baseline, paired by task. Ratios are the arm over the baseline (below 1 is better). W/T/L counts tasks the arm won, tied (within 0.02) or lost. Effect d is the mean paired difference over its spread across tasks.
By category
architecture
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| solo-haiku |
10 |
0.497 [0.297, 0.700] |
40% |
$0.0009 |
0.0023 |
7.4s |
8.3s |
| solo-luna |
10 |
0.243 [0.117, 0.370] |
10% |
$0.0006 |
0.0062 |
9.6s |
11.5s |
| solo-opus |
10 |
0.367 [0.260, 0.460] |
10% |
$0.0534 |
0.5339 |
20.8s |
25.8s |
| solo-sol |
10 |
0.207 [0.090, 0.337] |
10% |
$0.0211 |
0.2114 |
22.9s |
26.4s |
| fusion-default |
10 |
0.583 [0.400, 0.767] |
50% |
$0.0052 |
0.0103 |
18.2s |
21.6s |
| fusion-best |
10 |
0.257 [0.123, 0.423] |
10% |
$0.0037 |
0.0366 |
10.5s |
11.7s |
code_review
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| solo-haiku |
10 |
0.731 [0.621, 0.848] |
70% |
$0.0006 |
0.0009 |
4.3s |
5.3s |
| solo-luna |
10 |
0.822 [0.722, 0.922] |
100% |
$0.0004 |
0.0004 |
4.9s |
7.2s |
| solo-opus |
10 |
0.552 [0.502, 0.602] |
30% |
$0.0368 |
0.1227 |
10.7s |
18.8s |
| solo-sol |
10 |
0.415 [0.330, 0.500] |
10% |
$0.0133 |
0.1327 |
11.6s |
17.0s |
| fusion-default |
10 |
0.532 [0.437, 0.659] |
20% |
$0.0027 |
0.0135 |
5.2s |
10.6s |
| fusion-best |
10 |
0.787 [0.660, 0.913] |
90% |
$0.0019 |
0.0021 |
7.8s |
9.5s |
coding
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| solo-haiku |
3 |
0.926 [0.778, 1.000] |
67% |
$0.0016 |
0.0024 |
5.3s |
10.8s |
| solo-luna |
3 |
1.000 [1.000, 1.000] |
100% |
$0.0008 |
0.0008 |
7.6s |
7.6s |
| solo-opus |
3 |
1.000 [1.000, 1.000] |
100% |
$0.0686 |
0.0686 |
18.6s |
19.7s |
| solo-sol |
3 |
1.000 [1.000, 1.000] |
100% |
$0.0208 |
0.0208 |
19.5s |
22.5s |
| fusion-default |
3 |
1.000 [1.000, 1.000] |
100% |
$0.0071 |
0.0071 |
13.2s |
17.8s |
| fusion-best |
3 |
1.000 [1.000, 1.000] |
100% |
$0.0055 |
0.0055 |
10.5s |
11.5s |
debugging
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| solo-haiku |
10 |
0.811 [0.685, 0.930] |
70% |
$0.0008 |
0.0012 |
6.1s |
7.1s |
| solo-luna |
10 |
0.751 [0.541, 0.921] |
70% |
$0.0006 |
0.0008 |
12.4s |
14.2s |
| solo-opus |
10 |
0.811 [0.643, 0.958] |
70% |
$0.0516 |
0.0738 |
17.5s |
28.5s |
| solo-sol |
10 |
0.765 [0.604, 0.895] |
90% |
$0.0127 |
0.0142 |
13.3s |
14.8s |
| fusion-default |
10 |
0.867 [0.769, 0.958] |
90% |
$0.0028 |
0.0031 |
6.7s |
10.7s |
| fusion-best |
10 |
0.781 [0.619, 0.930] |
70% |
$0.0055 |
0.0079 |
11.0s |
13.2s |
frontend
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| solo-haiku |
2 |
0.000 [0.000, 0.000] |
0% |
$0.0012 |
- |
8.8s |
9.0s |
| solo-luna |
2 |
0.000 [0.000, 0.000] |
0% |
$0.0012 |
- |
- |
- |
| solo-opus |
2 |
0.464 [0.000, 0.929] |
50% |
$0.0877 |
0.1754 |
29.9s |
36.1s |
| solo-sol |
2 |
0.460 [0.000, 0.919] |
50% |
$0.0279 |
0.0558 |
28.7s |
31.1s |
| fusion-default |
2 |
0.000 [0.000, 0.000] |
0% |
$0.0713 |
- |
34.6s |
34.6s |
| fusion-best |
2 |
0.989 [0.987, 0.990] |
100% |
$0.0028 |
0.0028 |
16.6s |
19.2s |
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| solo-haiku |
2 |
0.996 [0.993, 1.000] |
100% |
$0.0009 |
0.0009 |
4.3s |
10.5s |
| solo-luna |
2 |
0.975 [0.961, 0.989] |
100% |
$0.0006 |
0.0006 |
8.2s |
10.6s |
| solo-opus |
2 |
0.980 [0.964, 0.996] |
100% |
$0.0633 |
0.0633 |
18.1s |
39.8s |
| solo-sol |
2 |
0.946 [0.918, 0.975] |
100% |
$0.0282 |
0.0282 |
27.8s |
39.7s |
| fusion-default |
2 |
0.987 [0.975, 1.000] |
100% |
$0.0038 |
0.0038 |
4.1s |
23.6s |
| fusion-best |
2 |
0.961 [0.957, 0.964] |
100% |
$0.0021 |
0.0021 |
11.0s |
13.0s |
planning
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| solo-haiku |
10 |
0.505 [0.288, 0.712] |
50% |
$0.0010 |
0.0019 |
7.2s |
8.7s |
| solo-luna |
10 |
0.214 [0.114, 0.310] |
0% |
$0.0009 |
- |
12.6s |
14.5s |
| solo-opus |
10 |
0.454 [0.250, 0.676] |
30% |
$0.0631 |
0.2103 |
24.1s |
29.0s |
| solo-sol |
10 |
0.276 [0.090, 0.490] |
20% |
$0.0250 |
0.1249 |
26.8s |
34.1s |
| fusion-default |
10 |
0.397 [0.198, 0.594] |
40% |
$0.0065 |
0.0163 |
20.6s |
33.1s |
| fusion-best |
10 |
0.300 [0.129, 0.507] |
10% |
$0.0058 |
0.0582 |
11.7s |
13.7s |
By difficulty
easy
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| solo-haiku |
13 |
0.709 [0.544, 0.849] |
54% |
$0.0008 |
0.0015 |
5.3s |
7.9s |
| solo-luna |
13 |
0.558 [0.376, 0.747] |
62% |
$0.0006 |
0.0010 |
7.6s |
12.6s |
| solo-opus |
13 |
0.637 [0.474, 0.797] |
62% |
$0.0455 |
0.0740 |
17.5s |
22.0s |
| solo-sol |
13 |
0.502 [0.309, 0.700] |
38% |
$0.0157 |
0.0409 |
15.4s |
26.8s |
| fusion-default |
13 |
0.755 [0.616, 0.885] |
62% |
$0.0034 |
0.0055 |
7.3s |
22.5s |
| fusion-best |
13 |
0.709 [0.546, 0.868] |
62% |
$0.0038 |
0.0061 |
11.0s |
13.5s |
hard
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| solo-haiku |
15 |
0.613 [0.426, 0.782] |
53% |
$0.0008 |
0.0016 |
7.0s |
7.9s |
| solo-luna |
15 |
0.561 [0.362, 0.757] |
47% |
$0.0007 |
0.0015 |
11.0s |
14.2s |
| solo-opus |
15 |
0.490 [0.321, 0.656] |
27% |
$0.0562 |
0.2108 |
21.6s |
30.0s |
| solo-sol |
15 |
0.510 [0.361, 0.645] |
47% |
$0.0196 |
0.0421 |
22.5s |
28.7s |
| fusion-default |
15 |
0.523 [0.359, 0.685] |
40% |
$0.0055 |
0.0137 |
13.4s |
23.8s |
| fusion-best |
15 |
0.571 [0.384, 0.753] |
53% |
$0.0046 |
0.0086 |
11.2s |
13.1s |
medium
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| solo-haiku |
19 |
0.621 [0.458, 0.773] |
63% |
$0.0010 |
0.0016 |
7.2s |
10.5s |
| solo-luna |
19 |
0.505 [0.335, 0.672] |
42% |
$0.0007 |
0.0016 |
10.1s |
14.5s |
| solo-opus |
19 |
0.637 [0.516, 0.772] |
42% |
$0.0591 |
0.1403 |
23.4s |
35.0s |
| solo-sol |
19 |
0.436 [0.266, 0.610] |
37% |
$0.0209 |
0.0567 |
22.9s |
35.1s |
| fusion-default |
19 |
0.585 [0.431, 0.740] |
58% |
$0.0114 |
0.0197 |
13.2s |
23.6s |
| fusion-best |
19 |
0.545 [0.369, 0.708] |
47% |
$0.0041 |
0.0086 |
11.6s |
13.7s |
Judge reliability
Accuracy floor 80%: mimo-pro 100%, muse-ember 88%.
Most different answers, with evidence
fe-data-table (frontend): completion differs by +0.99
- fusion-best (repeat 1): completion 0.99
- gate tests-pass: pass
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: all 7 hidden tests pass
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +171 -2 lines in 3 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: All 7 hidden tests pass (E1) and static/build/a11y checks are clean (E2, E3, E5). index.html has lang, viewport meta, a title, a single h1, a , column headers with scope=col, row headers with scope=row, and a button in each column header with data-sort (lines 14-30). app.js updates aria-sort on the active column (and removes it elsewhere) and toggles ascending/descending per column, using numeric comparison for tickets and localeCompare for text (lines 12-31). The wrapper .table-scroll has tabindex=0 + role=region + aria-label so it is keyboard-reachable, and overflow-x:auto with table min-width:36rem means the table scrolls inside its own container on narrow screens while body max-width keeps the page from breaking (style.css lines 28-47). Design matches the brief: bordered/tinted header row (#e8eef6), zebra rows (nth-child(even)), borderless text-looking header buttons, caption on top. Focus-visible outlines are defined for both the wrapper and buttons (style.css lines 36-40). E3 shows no external requests, viewport meta present, no lint issues; E4 shows a compact 171-line addition across 3 files. Minor nits: numeric detection is hardcoded to the 'tickets' key rather tha
- solo-opus (repeat 1): completion 0.00
- gate tests-pass: FAIL
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: 6/7 hidden tests pass; failing: HiddenTests.test_exactly_one_column_reports_its_sort
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +178 -2 lines in 3 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: Output A builds a real table with a caption (#people-caption), column headers with sorting buttons, row headers (scope="row" for each name), and a keyboard-reachable scroll wrapper (role=region, tabindex=0, overflow-x:auto with min-width:36rem so it scrolls on a 390px phone) — E3 static PASS, E5 a11y PASS. Sorting works for text and numeric columns via data-type, and the sorted column is toggled to aria-sort with a live status line. However, the hidden suite fails: E1 shows 6/7 with HiddenTests.test_exactly_one_column_reports_its_sort failing consistently across 3 runs, and the tests-pass hard gate is FAIL. This directly undercuts the core sort-reporting requirement even though the JS attempts to reset other headers to "none" (app.js lines 31-40), so correctness and ux_polish are reduced. The tinted header (#eef2f8), bordered wrapper, zebra-capable rows (only border-top given; no alternating row tint is actually applied — a design_fidelity gap), text-like header buttons, caption on top, visible :focus-visible outline, and viewport meta/lang/title all satisfy the design and a11y expectations (E2 build PASS, E4 diff_stats PASS), and everything is local/offline (works-offline pass). C
fe-login-dark (frontend): completion differs by -0.93
- fusion-default (repeat 1): completion 0.00
- gate tests-pass: FAIL
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +156 -2 lines in 3 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: The output delivers the requested structure: index.html has lang, title, viewport meta, one h1, a labelled Username and Password field, a Show password checkbox with an 8-line app.js toggle (app.js lines 1-8), a full-width Sign in button and a quiet Forgot your password? link (E3, E5). Accessibility is strong: E5 reports zero contrast failures and a 3px yellow :focus-visible outline (style.css line 118-121); autocomplete attributes are present. Design matches the brief: centred dark card (max-width 24rem, .page flex-centred), full-width primary button and underlined secondary link (E3 max_fixed_width_px=0). However, the hidden test suite fails 6/7, specifically HiddenTests.test_fits_small_screens (E1), so the responsiveness criterion cannot be scored highly; the remaining build/static/a11y checks pass (E2, E3, E5) and the file diff is minimal and clean (E4: +156 -2 over 3 files). | E1 shows 6/7 hidden tests pass but test_fits_small_screens fails, so responsiveness is weak and tests hard gate fails despite E2 build PASS, E3 static PASS and E5 a11y PASS indicating correct structure, labels and contrast. E6 screenshot n/a and E7 console n/a limit visual verification, so design and pol
- solo-opus (repeat 1): completion 0.93
- gate tests-pass: pass
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: all 7 hidden tests pass
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +170 -2 lines in 3 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: Output A implements the brief cleanly: index.html (E1/E3) has lang, title, viewport meta, a single h1 'Sign in', a semantic /
fe-login-dark (frontend): completion differs by -0.93
- solo-haiku (repeat 1): completion 0.00
- gate tests-pass: FAIL
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +44 -2 lines in 1 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: Output A delivers a single self-contained index.html with lang, viewport meta, title, one h1, labelled Username/Password inputs with autocomplete, a working Show password checkbox toggle (inline script lines 42-50), a full-width Sign in button and a quiet 'Forgot your password?' link, all centred in a dark card (E3, E5: 0 contrast failures, 0 lint issues, 0 external requests). However, E1 reports 6/7 hidden tests passing with HiddenTests.test_fits_small_screens failing, so the responsiveness requirement from 390px phone to 1440px desktop is not fully verified despite the CSS using width:100%, max-width:360px and flex centering; this also trips the hard gate tests-pass (FAIL, E1). Everything else builds and passes static/a11y checks (E2, E3, E5). | File shows centred dark card with h1, labelled Username/Password, Show password checkbox toggling type, full-width Sign in button, Forgot link, bright #ffd54a focus-visible ring and high-contrast colours, matching a11y PASS E5 contrast_failures=0. However hard test gate fails: E1 tests FAIL 6/7 with HiddenTests.test_fits_small_screens failing, so responsiveness penalised despite viewport meta and max-width layout; build/static/diff PASS p
- solo-opus (repeat 1): completion 0.93
- gate tests-pass: pass
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: all 7 hidden tests pass
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +170 -2 lines in 3 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: Output A implements the brief cleanly: index.html (E1/E3) has lang, title, viewport meta, a single h1 'Sign in', a semantic /
Methodology
- Run: v3-headline (completed), created 2026-11-19 10:42:16
- Dataset: evals/datasets/v2, split test, 47 tasks, hash 84c35e7998b364bd
- Arm solo-opus: strategy solo-opus: solo, models claude-opus, aggregator auto, rounds 1, judge off
- Arm solo-sol: strategy solo-sol: solo, models gpt-sol, aggregator auto, rounds 1, judge off
- Arm solo-haiku: strategy solo-haiku: solo, models claude-haiku, aggregator auto, rounds 1, judge off
- Arm solo-luna: strategy solo-luna: solo, models gpt-luna, aggregator auto, rounds 1, judge off
- Arm fusion-default: strategy auto: auto, models claude-haiku, gpt-luna, gemini-flash, gpt-terra, gemini-hydrogen, claude-sonnet, aggregator auto, rounds 1, judge off
- Arm fusion-best: strategy panel-nano-select, overrides {'aggregator': 'select'}: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator select, rounds 1, judge off
- Model claude-opus: anthropic claude-opus-5-5: $4 in / $20 out per 1M tokens, verified 2026-11-17
- Model gpt-sol: openai gpt-6.2-sol: $2 in / $10 out per 1M tokens, verified 2026-11-17
- Model claude-haiku: anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Model gpt-luna: openai gpt-6-luna: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Model gemini-flash: google gemini-3.8-flash: $0.75 in / $3.75 out per 1M tokens, verified 2026-11-17
- Model gpt-terra: openai gpt-6.1-terra: $1 in / $5 out per 1M tokens, verified 2026-11-17
- Model gemini-hydrogen: google gemini-4-hydrogen: $1.5 in / $7.5 out per 1M tokens, verified 2026-11-17
- Model claude-sonnet: anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
- Model mimo-pro: openrouter xiaomi/mimo-v2.6-pro: $0.435 in / $0.87 out per 1M tokens, verified 2026-11-17
- Model muse-ember: openrouter meta/muse-ember: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Judge models: mimo-pro, muse-ember
- Repeats / seed: 1 per task / seed 100
- Intervals: 95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
- Verdict rules: non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline solo-opus
- Spend: $4.03 on the arms + $0.3940 on scoring and judging (cap $6.00)
- Fusion version: 0.2.0 (git 5abab47, with uncommitted changes)
Supporting study: v3-ablation
Arms: vote, lenses-off, digest, select, mixed, verified-synth, open, nano, mid, cascade-0.5, auto-route, cascade-0.9, auto-route-frontier. Baseline for the comparisons: vote.
Some prices were unknown for vote, lenses-off, digest, select, mixed, verified-synth, open, nano, mid, cascade-0.5, auto-route, cascade-0.9, auto-route-frontier: its costs are a lower bound.
Items replayed from the response cache are left out of the latency figures (vote, lenses-off, digest, mixed, verified-synth, open, mid, cascade-0.5, auto-route, cascade-0.9, auto-route-frontier): their wall time is not a measurement.
1 repeat(s) per task: variance across repeats is poorly estimated.
Fewer than 10 tasks are answered by both an arm and the baseline: no verdict is possible for it.
Frontend and performance scores measure what the answer builds (tests, timings, screenshots); they do not judge whether it matches the author's intent.
Headline
Each arm against vote, on all tasks. ✓ yes, ✗ no, ? inconclusive, ⊘ blocked. The reasons are listed below the table.
| Arm |
Paired tasks |
Cheaper (per solved task) |
Faster |
Not worse |
Better |
| lenses-off |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| digest |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| select |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| mixed |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| verified-synth |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| open |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| nano |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| mid |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| cascade-0.5 |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| auto-route |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| cascade-0.9 |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| auto-route-frontier |
9 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
- lenses-off vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
- digest vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
- select vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
- mixed vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
- verified-synth vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
- open vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
- nano vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
- mid vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
- cascade-0.5 vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
- auto-route vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
- cascade-0.9 vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
- auto-route-frontier vs vote (9 tasks): only 9 task(s) answered by both arms; at least 10 are needed to claim anything.
Arms
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
Frontier |
| vote |
9 |
0.661 [0.414, 0.891] |
56% |
$0.0055 |
0.0098 |
8.8s |
13.3s |
s |
| lenses-off |
9 |
0.619 [0.353, 0.881] |
56% |
$0.0053 |
0.0096 |
21.0s |
29.1s |
$ |
| digest |
9 |
0.199 [0.037, 0.422] |
11% |
$0.0068 |
0.0615 |
11.5s |
11.5s |
- |
| select |
9 |
0.714 [0.454, 0.941] |
56% |
$0.0054 |
0.0097 |
13.2s |
23.7s |
$ s |
| mixed |
9 |
0.461 [0.161, 0.765] |
44% |
$0.0040 |
0.0090 |
20.2s |
29.9s |
$ |
| verified-synth |
9 |
0.721 [0.454, 0.917] |
67% |
$0.0065 |
0.0098 |
15.0s |
19.8s |
$ s |
| open |
9 |
0.438 [0.140, 0.730] |
44% |
$0.0070 |
0.0157 |
29.8s |
38.4s |
- |
| nano |
9 |
0.499 [0.213, 0.796] |
44% |
$0.0088 |
0.0197 |
21.7s |
33.1s |
- |
| mid |
9 |
0.495 [0.199, 0.792] |
44% |
$0.0730 |
0.1643 |
29.3s |
41.5s |
- |
| cascade-0.5 |
9 |
0.676 [0.417, 0.908] |
67% |
$0.0418 |
0.0627 |
- |
- |
- |
| auto-route |
9 |
0.696 [0.422, 0.907] |
67% |
$0.0236 |
0.0355 |
13.0s |
29.5s |
s |
| cascade-0.9 |
9 |
0.728 [0.467, 0.933] |
67% |
$0.0391 |
0.0587 |
- |
- |
$ |
| auto-route-frontier |
9 |
0.642 [0.385, 0.884] |
67% |
$0.0391 |
0.0586 |
20.6s |
30.9s |
- |
Frontier: $ = on the cost-quality frontier (no arm is cheaper per task and at least as good), s = on the latency-quality frontier (median seconds).
Efficiency and speed
| Arm |
Out tok/s (effective) |
Tokens / solved |
Quality / $ |
Quality / min |
Calls |
Errors |
| vote |
214 |
25,296 |
121.0 |
3.981 |
3.2 |
0 |
| lenses-off |
230 |
32,653 |
115.8 |
1.806 |
4.4 |
0 |
| digest |
222 |
121,367 |
29.2 |
1.038 |
3.2 |
0 |
| select |
188 |
24,857 |
133.0 |
2.991 |
4.2 |
0 |
| mixed |
197 |
43,452 |
115.5 |
1.105 |
4.8 |
0 |
| verified-synth |
193 |
20,161 |
110.1 |
2.656 |
4.0 |
0 |
| open |
200 |
43,548 |
62.9 |
0.972 |
7.0 |
0 |
| nano |
229 |
44,460 |
57.1 |
1.286 |
4.2 |
0 |
| mid |
191 |
44,941 |
6.8 |
1.013 |
4.7 |
0 |
| cascade-0.5 |
- |
35,659 |
16.2 |
- |
4.2 |
0 |
| auto-route |
180 |
20,460 |
29.4 |
2.680 |
3.1 |
0 |
| cascade-0.9 |
- |
35,078 |
18.6 |
- |
4.2 |
0 |
| auto-route-frontier |
180 |
26,849 |
16.4 |
1.974 |
3.6 |
0 |
Streamed speed per model:
| Arm |
Model |
Decode tok/s |
TTFT ms |
| vote |
claude-haiku |
237 |
1066 |
| vote |
gemini-flash |
142 |
4138 |
| vote |
gpt-luna |
186 |
4413 |
| lenses-off |
claude-haiku |
323 |
2489 |
| lenses-off |
gemini-flash |
141 |
3350 |
| lenses-off |
gpt-luna |
177 |
4506 |
| digest |
claude-haiku |
217 |
1282 |
| digest |
gemini-flash |
147 |
2964 |
| digest |
gpt-luna |
149 |
5000 |
| select |
claude-haiku |
270 |
1500 |
| select |
gemini-flash |
143 |
2784 |
| select |
gpt-luna |
219 |
5894 |
| mixed |
claude-haiku |
303 |
2506 |
| mixed |
gpt-luna |
193 |
5554 |
| mixed |
minimax-flash |
416 |
716 |
| verified-synth |
claude-haiku |
286 |
1196 |
| verified-synth |
gemini-flash |
144 |
3481 |
| verified-synth |
gpt-luna |
223 |
5874 |
| open |
claude-haiku |
330 |
2933 |
| open |
deepseek-flash |
447 |
6642 |
| open |
glm |
296 |
3107 |
| open |
gpt-luna |
255 |
3951 |
| open |
mimo-flash |
184 |
6159 |
| open |
minimax-flash |
290 |
1504 |
| nano |
claude-haiku |
351 |
3664 |
| nano |
gemini-flash |
144 |
3246 |
| nano |
gpt-luna |
288 |
3916 |
| mid |
claude-sonnet |
207 |
4647 |
| mid |
gemini-hydrogen |
1188 |
5491 |
| mid |
gpt-terra |
120 |
3877 |
| auto-route |
claude-haiku |
253 |
1488 |
| auto-route |
gemini-flash |
136 |
2861 |
| auto-route |
gpt-luna |
215 |
5554 |
| auto-route-frontier |
claude-haiku |
368 |
3351 |
| auto-route-frontier |
gemini-flash |
138 |
6825 |
| auto-route-frontier |
gpt-luna |
219 |
4967 |
Cost of measuring
Scoring and judging cost and time, kept apart: never part of an arm's cost or latency.
| Arm |
Eval $ total |
Eval $ / item |
Eval s / item |
| vote |
$0.0067 |
$0.0007 |
40.7s |
| lenses-off |
$0.0094 |
$0.0010 |
4.6s |
| digest |
$0.0068 |
$0.0008 |
0.0s |
| select |
$0.0109 |
$0.0012 |
40.1s |
| mixed |
$0.0104 |
$0.0012 |
4.5s |
| verified-synth |
$0.0054 |
$0.0006 |
38.0s |
| open |
$0.0121 |
$0.0013 |
4.9s |
| nano |
$0.0130 |
$0.0014 |
7.6s |
| mid |
$0.0118 |
$0.0013 |
39.6s |
| cascade-0.5 |
$0.0089 |
$0.0010 |
38.5s |
| auto-route |
$0.0087 |
$0.0010 |
29.7s |
| cascade-0.9 |
$0.0104 |
$0.0012 |
58.0s |
| auto-route-frontier |
$0.0094 |
$0.0010 |
37.4s |
Variance across repeats
| Arm |
Mean sd of quality across repeats |
Tasks whose solved flips |
| vote |
- |
- |
| lenses-off |
- |
- |
| digest |
- |
- |
| select |
- |
- |
| mixed |
- |
- |
| verified-synth |
- |
- |
| open |
- |
- |
| nano |
- |
- |
| mid |
- |
- |
| cascade-0.5 |
- |
- |
| auto-route |
- |
- |
| cascade-0.9 |
- |
- |
| auto-route-frontier |
- |
- |
Comparisons against the baseline
lenses-off vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
-0.042 [-0.185, +0.080] |
4/3/2 |
0.688 |
-0.19 |
0.98 [0.47, 3.02] |
1.96 [1.66, 2.63] |
? |
? |
? |
? |
| architecture |
1 |
+0.167 |
1/0/0 |
1.000 |
- |
- |
1.79 |
? |
? |
? |
? |
| code_review |
2 |
-0.400 [-0.467, -0.333] |
0/0/2 |
0.500 |
-4.24 |
- |
2.93 |
? |
? |
? |
? |
| coding |
2 |
+0.040 [+0.000, +0.080] |
1/1/0 |
1.000 |
+0.71 |
0.35 |
- |
? |
? |
? |
? |
| debugging |
1 |
+0.000 |
0/1/0 |
- |
- |
1.91 |
1.86 |
? |
? |
? |
? |
| frontend |
1 |
-0.017 |
0/1/0 |
- |
- |
0.36 |
- |
? |
? |
? |
? |
| performance |
1 |
+0.025 |
1/0/0 |
1.000 |
- |
1.32 |
- |
? |
? |
? |
? |
| planning |
1 |
+0.167 |
1/0/0 |
1.000 |
- |
- |
1.62 |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
digest vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
-0.462 [-0.789, -0.117] |
2/1/6 |
0.289 |
-0.84 |
6.25 |
1.11 |
? |
? |
? |
? |
| architecture |
1 |
+0.333 |
1/0/0 |
1.000 |
- |
- |
1.11 |
? |
? |
? |
? |
| code_review |
2 |
-0.520 [-0.606, -0.433] |
0/0/2 |
0.500 |
-4.26 |
- |
- |
? |
? |
? |
? |
| coding |
2 |
-0.960 [-1.000, -0.920] |
0/0/2 |
0.500 |
-16.97 |
- |
- |
? |
? |
? |
? |
| debugging |
1 |
+0.000 |
0/1/0 |
- |
- |
3.88 |
- |
? |
? |
? |
? |
| frontend |
1 |
-0.902 |
0/0/1 |
1.000 |
- |
- |
- |
? |
? |
? |
? |
| performance |
1 |
-0.964 |
0/0/1 |
1.000 |
- |
- |
- |
? |
? |
? |
? |
| planning |
1 |
+0.333 |
1/0/0 |
1.000 |
- |
- |
- |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
select vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
+0.052 [-0.046, +0.156] |
5/2/2 |
0.453 |
+0.32 |
0.98 [0.48, 1.52] |
1.28 [1.04, 1.54] |
? |
? |
? |
? |
| architecture |
1 |
+0.167 |
1/0/0 |
1.000 |
- |
- |
1.59 |
? |
? |
? |
? |
| code_review |
2 |
+0.042 [-0.250, +0.333] |
1/0/1 |
1.000 |
+0.10 |
0.53 |
1.21 |
? |
? |
? |
? |
| coding |
2 |
+0.020 [+0.000, +0.040] |
1/1/0 |
1.000 |
+0.71 |
1.57 |
- |
? |
? |
? |
? |
| debugging |
1 |
+0.000 |
0/1/0 |
- |
- |
1.27 |
1.38 |
? |
? |
? |
? |
| frontend |
1 |
-0.024 |
0/0/1 |
1.000 |
- |
0.27 |
- |
? |
? |
? |
? |
| performance |
1 |
+0.036 |
1/0/0 |
1.000 |
- |
0.94 |
- |
? |
? |
? |
? |
| planning |
1 |
+0.167 |
1/0/0 |
1.000 |
- |
- |
1.00 |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
mixed vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
-0.201 [-0.433, +0.006] |
2/3/4 |
0.688 |
-0.55 |
0.91 [0.39, 3.16] |
2.88 |
? |
? |
? |
? |
| architecture |
1 |
+0.000 |
0/1/0 |
- |
- |
- |
2.88 |
? |
? |
? |
? |
| code_review |
2 |
-0.451 [-0.485, -0.417] |
0/0/2 |
0.500 |
-9.35 |
- |
- |
? |
? |
? |
? |
| coding |
2 |
-0.460 [-1.000, +0.080] |
1/0/1 |
1.000 |
-0.60 |
0.41 |
- |
? |
? |
? |
? |
| debugging |
1 |
+0.000 |
0/1/0 |
- |
- |
3.17 |
- |
? |
? |
? |
? |
| frontend |
1 |
-0.021 |
0/0/1 |
1.000 |
- |
0.50 |
- |
? |
? |
? |
? |
| performance |
1 |
+0.036 |
1/0/0 |
1.000 |
- |
0.43 |
- |
? |
? |
? |
? |
| planning |
1 |
+0.000 |
0/1/0 |
- |
- |
- |
- |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
verified-synth vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
+0.059 [-0.038, +0.156] |
5/3/1 |
0.219 |
+0.37 |
1.00 [0.47, 1.77] |
- |
? |
? |
? |
? |
| architecture |
1 |
+0.167 |
1/0/0 |
1.000 |
- |
- |
- |
? |
? |
? |
? |
| code_review |
2 |
+0.042 [-0.250, +0.333] |
1/0/1 |
1.000 |
+0.10 |
1.85 |
- |
? |
? |
? |
? |
| coding |
2 |
+0.040 [+0.000, +0.080] |
1/1/0 |
1.000 |
+0.71 |
0.86 |
- |
? |
? |
? |
? |
| debugging |
1 |
+0.000 |
0/1/0 |
- |
- |
1.28 |
- |
? |
? |
? |
? |
| frontend |
1 |
-0.000 |
0/1/0 |
- |
- |
0.23 |
- |
? |
? |
? |
? |
| performance |
1 |
+0.036 |
1/0/0 |
1.000 |
- |
1.25 |
- |
? |
? |
? |
? |
| planning |
1 |
+0.167 |
1/0/0 |
1.000 |
- |
- |
- |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
open vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
-0.223 [-0.435, -0.037] |
1/4/4 |
0.375 |
-0.67 |
1.59 [0.73, 4.49] |
3.82 |
? |
? |
? |
? |
| architecture |
1 |
+0.000 |
0/1/0 |
- |
- |
- |
3.71 |
? |
? |
? |
? |
| code_review |
2 |
-0.477 [-0.513, -0.441] |
0/0/2 |
0.500 |
-9.42 |
- |
3.98 |
? |
? |
? |
? |
| coding |
2 |
-0.460 [-0.920, +0.000] |
0/1/1 |
1.000 |
-0.71 |
1.28 |
- |
? |
? |
? |
? |
| debugging |
1 |
-0.150 |
0/0/1 |
1.000 |
- |
4.22 |
- |
? |
? |
? |
? |
| frontend |
1 |
-0.010 |
0/1/0 |
- |
- |
0.49 |
- |
? |
? |
? |
? |
| performance |
1 |
+0.025 |
1/0/0 |
1.000 |
- |
0.61 |
- |
? |
? |
? |
? |
| planning |
1 |
+0.000 |
0/1/0 |
- |
- |
- |
- |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
nano vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
-0.162 [-0.424, +0.060] |
2/4/3 |
1.000 |
-0.41 |
2.00 [0.67, 7.41] |
2.13 [1.58, 2.76] |
? |
? |
? |
? |
| architecture |
1 |
+0.333 |
1/0/0 |
1.000 |
- |
- |
2.47 |
? |
? |
? |
? |
| code_review |
2 |
-0.442 [-0.467, -0.417] |
0/0/2 |
0.500 |
-12.49 |
- |
2.89 |
? |
? |
? |
? |
| coding |
2 |
-0.460 [-1.000, +0.080] |
1/0/1 |
1.000 |
-0.60 |
1.76 |
- |
? |
? |
? |
? |
| debugging |
1 |
+0.000 |
0/1/0 |
- |
- |
5.10 |
2.21 |
? |
? |
? |
? |
| frontend |
1 |
-0.008 |
0/1/0 |
- |
- |
0.43 |
- |
? |
? |
? |
? |
| performance |
1 |
+0.018 |
0/1/0 |
- |
- |
1.30 |
- |
? |
? |
? |
? |
| planning |
1 |
+0.000 |
0/1/0 |
- |
- |
- |
1.37 |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
mid vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
-0.166 [-0.371, +0.013] |
2/4/3 |
1.000 |
-0.52 |
16.70 [6.72, 54.93] |
2.80 [2.38, 3.25] |
? |
? |
? |
? |
| architecture |
1 |
+0.000 |
0/1/0 |
- |
- |
- |
2.83 |
? |
? |
? |
? |
| code_review |
2 |
-0.354 [-0.375, -0.333] |
0/0/2 |
0.500 |
-12.02 |
- |
3.25 |
? |
? |
? |
? |
| coding |
2 |
+0.040 [+0.000, +0.080] |
1/1/0 |
1.000 |
+0.71 |
4.59 |
- |
? |
? |
? |
? |
| debugging |
1 |
+0.000 |
0/1/0 |
- |
- |
49.68 |
2.38 |
? |
? |
? |
? |
| frontend |
1 |
-0.902 |
0/0/1 |
1.000 |
- |
- |
- |
? |
? |
? |
? |
| performance |
1 |
+0.036 |
1/0/0 |
1.000 |
- |
10.62 |
- |
? |
? |
? |
? |
| planning |
1 |
+0.000 |
0/1/0 |
- |
- |
- |
- |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
cascade-0.5 vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
+0.015 [-0.043, +0.066] |
3/5/1 |
0.625 |
+0.17 |
6.37 [3.52, 11.42] |
- |
? |
? |
? |
? |
| architecture |
1 |
+0.000 |
0/1/0 |
- |
- |
- |
- |
? |
? |
? |
? |
| code_review |
2 |
-0.083 [-0.167, +0.000] |
0/1/1 |
1.000 |
-0.71 |
10.77 |
- |
? |
? |
? |
? |
| coding |
2 |
+0.040 [+0.000, +0.080] |
1/1/0 |
1.000 |
+0.71 |
2.50 |
- |
? |
? |
? |
? |
| debugging |
1 |
+0.000 |
0/1/0 |
- |
- |
24.74 |
- |
? |
? |
? |
? |
| frontend |
1 |
+0.018 |
0/1/0 |
- |
- |
6.01 |
- |
? |
? |
? |
? |
| performance |
1 |
+0.036 |
1/0/0 |
1.000 |
- |
7.41 |
- |
? |
? |
? |
? |
| planning |
1 |
+0.167 |
1/0/0 |
1.000 |
- |
- |
- |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
auto-route vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
+0.035 [-0.096, +0.149] |
5/2/2 |
0.453 |
+0.17 |
3.61 [0.70, 9.11] |
1.28 [0.43, 2.54] |
? |
? |
? |
? |
| architecture |
1 |
+0.167 |
1/0/0 |
1.000 |
- |
- |
2.54 |
? |
? |
? |
? |
| code_review |
2 |
-0.033 [-0.400, +0.333] |
1/0/1 |
1.000 |
-0.06 |
0.99 |
0.43 |
? |
? |
? |
? |
| coding |
2 |
+0.040 [+0.000, +0.080] |
1/1/0 |
1.000 |
+0.71 |
2.72 |
- |
? |
? |
? |
? |
| debugging |
1 |
+0.000 |
0/1/0 |
- |
- |
0.51 |
0.52 |
? |
? |
? |
? |
| frontend |
1 |
-0.070 |
0/0/1 |
1.000 |
- |
1.16 |
- |
? |
? |
? |
? |
| performance |
1 |
+0.036 |
1/0/0 |
1.000 |
- |
0.15 |
- |
? |
? |
? |
? |
| planning |
1 |
+0.167 |
1/0/0 |
1.000 |
- |
- |
- |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
cascade-0.9 vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
+0.066 [-0.047, +0.185] |
4/4/1 |
0.375 |
+0.34 |
5.97 [3.20, 10.54] |
- |
? |
? |
? |
? |
| architecture |
1 |
+0.000 |
0/1/0 |
- |
- |
- |
- |
? |
? |
? |
? |
| code_review |
2 |
+0.042 [-0.250, +0.333] |
1/0/1 |
1.000 |
+0.10 |
10.53 |
- |
? |
? |
? |
? |
| coding |
2 |
+0.040 [+0.000, +0.080] |
1/1/0 |
1.000 |
+0.71 |
2.24 |
- |
? |
? |
? |
? |
| debugging |
1 |
+0.000 |
0/1/0 |
- |
- |
22.34 |
- |
? |
? |
? |
? |
| frontend |
1 |
-0.001 |
0/1/0 |
- |
- |
5.97 |
- |
? |
? |
? |
? |
| performance |
1 |
+0.036 |
1/0/0 |
1.000 |
- |
7.61 |
- |
? |
? |
? |
? |
| planning |
1 |
+0.400 |
1/0/0 |
1.000 |
- |
- |
- |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
auto-route-frontier vs vote
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
9 |
-0.019 [-0.320, +0.246] |
5/2/2 |
0.453 |
-0.04 |
5.96 [0.40, 23.54] |
1.49 [0.64, 2.42] |
? |
? |
? |
? |
| architecture |
1 |
+0.667 |
1/0/0 |
1.000 |
- |
- |
2.42 |
? |
? |
? |
? |
| code_review |
2 |
-0.200 [-0.400, +0.000] |
0/1/1 |
1.000 |
-0.71 |
1.57 |
1.21 |
? |
? |
? |
? |
| coding |
2 |
-0.460 [-1.000, +0.080] |
1/0/1 |
1.000 |
-0.60 |
11.89 |
- |
? |
? |
? |
? |
| debugging |
1 |
+0.000 |
0/1/0 |
- |
- |
0.63 |
0.64 |
? |
? |
? |
? |
| frontend |
1 |
+0.043 |
1/0/0 |
1.000 |
- |
1.07 |
- |
? |
? |
? |
? |
| performance |
1 |
+0.036 |
1/0/0 |
1.000 |
- |
0.15 |
- |
? |
? |
? |
? |
| planning |
1 |
+0.400 |
1/0/0 |
1.000 |
- |
- |
- |
? |
? |
? |
? |
- architecture (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- coding (2 tasks): only 2 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- frontend (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- performance (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (1 tasks): only 1 task(s) answered by both arms; at least 10 are needed to claim anything.
Δ quality is the arm minus the baseline, paired by task. Ratios are the arm over the baseline (below 1 is better). W/T/L counts tasks the arm won, tied (within 0.02) or lost. Effect d is the mean paired difference over its spread across tasks.
By category
architecture
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| vote |
1 |
0.000 |
0% |
$0.0046 |
- |
10.4s |
10.4s |
| lenses-off |
1 |
0.167 |
0% |
$0.0063 |
- |
18.6s |
18.6s |
| digest |
1 |
0.333 |
0% |
$0.0050 |
- |
11.5s |
11.5s |
| select |
1 |
0.167 |
0% |
$0.0021 |
- |
16.5s |
16.5s |
| mixed |
1 |
0.000 |
0% |
$0.0035 |
- |
29.9s |
29.9s |
| verified-synth |
1 |
0.167 |
0% |
$0.0022 |
- |
- |
- |
| open |
1 |
0.000 |
0% |
$0.0048 |
- |
38.4s |
38.4s |
| nano |
1 |
0.333 |
0% |
$0.0066 |
- |
25.6s |
25.6s |
| mid |
1 |
0.000 |
0% |
$0.0714 |
- |
29.3s |
29.3s |
| cascade-0.5 |
1 |
0.000 |
0% |
$0.0494 |
- |
- |
- |
| auto-route |
1 |
0.167 |
0% |
$0.0063 |
- |
26.3s |
26.3s |
| cascade-0.9 |
1 |
0.000 |
0% |
$0.0413 |
- |
- |
- |
| auto-route-frontier |
1 |
0.667 |
100% |
$0.0065 |
0.0065 |
25.1s |
25.1s |
code_review
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| vote |
2 |
0.583 [0.500, 0.667] |
50% |
$0.0025 |
0.0050 |
7.5s |
7.5s |
| lenses-off |
2 |
0.183 [0.167, 0.200] |
0% |
$0.0057 |
- |
15.6s |
21.9s |
| digest |
2 |
0.064 [0.061, 0.067] |
0% |
$0.0047 |
- |
- |
- |
| select |
2 |
0.625 [0.250, 1.000] |
50% |
$0.0013 |
0.0026 |
9.1s |
10.7s |
| mixed |
2 |
0.133 [0.083, 0.182] |
0% |
$0.0050 |
- |
- |
- |
| verified-synth |
2 |
0.625 [0.250, 1.000] |
50% |
$0.0046 |
0.0093 |
- |
- |
| open |
2 |
0.106 [0.059, 0.154] |
0% |
$0.0078 |
- |
29.8s |
32.8s |
| nano |
2 |
0.142 [0.083, 0.200] |
0% |
$0.0062 |
- |
20.3s |
21.7s |
| mid |
2 |
0.229 [0.125, 0.333] |
0% |
$0.0610 |
- |
24.4s |
30.0s |
| cascade-0.5 |
2 |
0.500 [0.333, 0.667] |
50% |
$0.0270 |
0.0541 |
- |
- |
| auto-route |
2 |
0.550 [0.100, 1.000] |
50% |
$0.0025 |
0.0050 |
3.2s |
13.0s |
| cascade-0.9 |
2 |
0.625 [0.250, 1.000] |
50% |
$0.0264 |
0.0528 |
- |
- |
| auto-route-frontier |
2 |
0.383 [0.100, 0.667] |
50% |
$0.0039 |
0.0079 |
9.1s |
20.6s |
coding
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| vote |
2 |
0.960 [0.920, 1.000] |
50% |
$0.0102 |
0.0204 |
- |
- |
| lenses-off |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0072 |
0.0072 |
21.0s |
21.0s |
| digest |
2 |
0.000 [0.000, 0.000] |
0% |
$0.0150 |
- |
- |
- |
| select |
2 |
0.980 [0.960, 1.000] |
50% |
$0.0160 |
0.0320 |
13.2s |
17.6s |
| mixed |
2 |
0.500 [0.000, 1.000] |
50% |
$0.0041 |
0.0083 |
20.2s |
20.2s |
| verified-synth |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0175 |
0.0175 |
12.7s |
19.8s |
| open |
2 |
0.500 [0.000, 1.000] |
50% |
$0.0131 |
0.0261 |
16.6s |
16.6s |
| nano |
2 |
0.500 [0.000, 1.000] |
50% |
$0.0180 |
0.0359 |
20.3s |
28.5s |
| mid |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0936 |
0.0936 |
28.5s |
28.5s |
| cascade-0.5 |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0510 |
0.0510 |
- |
- |
| auto-route |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0556 |
0.0556 |
16.9s |
16.9s |
| cascade-0.9 |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0458 |
0.0458 |
- |
- |
| auto-route-frontier |
2 |
0.500 [0.000, 1.000] |
50% |
$0.1213 |
0.2427 |
25.9s |
25.9s |
debugging
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| vote |
1 |
1.000 |
100% |
$0.0012 |
0.0012 |
8.8s |
8.8s |
| lenses-off |
1 |
1.000 |
100% |
$0.0022 |
0.0022 |
16.3s |
16.3s |
| digest |
1 |
1.000 |
100% |
$0.0046 |
0.0046 |
- |
- |
| select |
1 |
1.000 |
100% |
$0.0015 |
0.0015 |
12.1s |
12.1s |
| mixed |
1 |
1.000 |
100% |
$0.0037 |
0.0037 |
- |
- |
| verified-synth |
1 |
1.000 |
100% |
$0.0015 |
0.0015 |
- |
- |
| open |
1 |
0.850 |
100% |
$0.0050 |
0.0050 |
- |
- |
| nano |
1 |
1.000 |
100% |
$0.0060 |
0.0060 |
19.4s |
19.4s |
| mid |
1 |
1.000 |
100% |
$0.0583 |
0.0583 |
20.9s |
20.9s |
| cascade-0.5 |
1 |
1.000 |
100% |
$0.0290 |
0.0290 |
- |
- |
| auto-route |
1 |
1.000 |
100% |
$0.0006 |
0.0006 |
4.6s |
4.6s |
| cascade-0.9 |
1 |
1.000 |
100% |
$0.0262 |
0.0262 |
- |
- |
| auto-route-frontier |
1 |
1.000 |
100% |
$0.0007 |
0.0007 |
5.6s |
5.6s |
frontend
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| vote |
1 |
0.902 |
100% |
$0.0113 |
0.0113 |
- |
- |
| lenses-off |
1 |
0.885 |
100% |
$0.0041 |
0.0041 |
29.1s |
29.1s |
| digest |
1 |
0.000 |
0% |
$0.0025 |
- |
- |
- |
| select |
1 |
0.878 |
100% |
$0.0031 |
0.0031 |
23.7s |
23.7s |
| mixed |
1 |
0.881 |
100% |
$0.0057 |
0.0057 |
- |
- |
| verified-synth |
1 |
0.902 |
100% |
$0.0026 |
0.0026 |
17.6s |
17.6s |
| open |
1 |
0.892 |
100% |
$0.0055 |
0.0055 |
- |
- |
| nano |
1 |
0.894 |
100% |
$0.0049 |
0.0049 |
33.1s |
33.1s |
| mid |
1 |
0.000 |
0% |
$0.0835 |
- |
41.5s |
41.5s |
| cascade-0.5 |
1 |
0.920 |
100% |
$0.0676 |
0.0676 |
- |
- |
| auto-route |
1 |
0.832 |
100% |
$0.0130 |
0.0130 |
29.5s |
29.5s |
| cascade-0.9 |
1 |
0.900 |
100% |
$0.0673 |
0.0673 |
- |
- |
| auto-route-frontier |
1 |
0.945 |
100% |
$0.0121 |
0.0121 |
30.9s |
30.9s |
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| vote |
1 |
0.964 |
100% |
$0.0050 |
0.0050 |
- |
- |
| lenses-off |
1 |
0.989 |
100% |
$0.0066 |
0.0066 |
- |
- |
| digest |
1 |
0.000 |
0% |
$0.0050 |
- |
- |
- |
| select |
1 |
1.000 |
100% |
$0.0047 |
0.0047 |
12.6s |
12.6s |
| mixed |
1 |
1.000 |
100% |
$0.0022 |
0.0022 |
- |
- |
| verified-synth |
1 |
1.000 |
100% |
$0.0063 |
0.0063 |
15.0s |
15.0s |
| open |
1 |
0.989 |
100% |
$0.0030 |
0.0030 |
17.6s |
17.6s |
| nano |
1 |
0.982 |
100% |
$0.0065 |
0.0065 |
22.6s |
22.6s |
| mid |
1 |
1.000 |
100% |
$0.0534 |
0.0534 |
30.9s |
30.9s |
| cascade-0.5 |
1 |
1.000 |
100% |
$0.0372 |
0.0372 |
- |
- |
| auto-route |
1 |
1.000 |
100% |
$0.0007 |
0.0007 |
- |
- |
| cascade-0.9 |
1 |
1.000 |
100% |
$0.0383 |
0.0383 |
- |
- |
| auto-route-frontier |
1 |
1.000 |
100% |
$0.0007 |
0.0007 |
- |
- |
planning
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| vote |
1 |
0.000 |
0% |
$0.0017 |
- |
13.3s |
13.3s |
| lenses-off |
1 |
0.167 |
0% |
$0.0032 |
- |
21.5s |
21.5s |
| digest |
1 |
0.333 |
0% |
$0.0049 |
- |
- |
- |
| select |
1 |
0.167 |
0% |
$0.0022 |
- |
13.3s |
13.3s |
| mixed |
1 |
0.000 |
0% |
$0.0025 |
- |
- |
- |
| verified-synth |
1 |
0.167 |
0% |
$0.0022 |
- |
- |
- |
| open |
1 |
0.000 |
0% |
$0.0025 |
- |
- |
- |
| nano |
1 |
0.000 |
0% |
$0.0064 |
- |
18.2s |
18.2s |
| mid |
1 |
0.000 |
0% |
$0.0812 |
- |
- |
- |
| cascade-0.5 |
1 |
0.167 |
0% |
$0.0365 |
- |
- |
- |
| auto-route |
1 |
0.167 |
0% |
$0.0759 |
- |
- |
- |
| cascade-0.9 |
1 |
0.400 |
0% |
$0.0347 |
- |
- |
- |
| auto-route-frontier |
1 |
0.400 |
0% |
$0.0810 |
- |
- |
- |
By difficulty
easy
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| vote |
1 |
1.000 |
100% |
$0.0107 |
0.0107 |
- |
- |
| lenses-off |
1 |
1.000 |
100% |
$0.0032 |
0.0032 |
21.0s |
21.0s |
| digest |
1 |
0.000 |
0% |
$0.0216 |
- |
- |
- |
| select |
1 |
1.000 |
100% |
$0.0208 |
0.0208 |
13.2s |
13.2s |
| mixed |
1 |
0.000 |
0% |
$0.0031 |
- |
20.2s |
20.2s |
| verified-synth |
1 |
1.000 |
100% |
$0.0224 |
0.0224 |
12.7s |
12.7s |
| open |
1 |
1.000 |
100% |
$0.0216 |
0.0216 |
16.6s |
16.6s |
| nano |
1 |
0.000 |
0% |
$0.0233 |
- |
20.3s |
20.3s |
| mid |
1 |
1.000 |
100% |
$0.1009 |
0.1009 |
28.5s |
28.5s |
| cascade-0.5 |
1 |
1.000 |
100% |
$0.0436 |
0.0436 |
- |
- |
| auto-route |
1 |
1.000 |
100% |
$0.0993 |
0.0993 |
- |
- |
| cascade-0.9 |
1 |
1.000 |
100% |
$0.0381 |
0.0381 |
- |
- |
| auto-route-frontier |
1 |
0.000 |
0% |
$0.2387 |
- |
- |
- |
hard
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| vote |
3 |
0.488 [0.000, 0.964] |
33% |
$0.0046 |
0.0139 |
10.4s |
10.4s |
| lenses-off |
3 |
0.441 [0.167, 0.989] |
33% |
$0.0061 |
0.0183 |
15.6s |
18.6s |
| digest |
3 |
0.133 [0.000, 0.333] |
0% |
$0.0049 |
- |
11.5s |
11.5s |
| select |
3 |
0.472 [0.167, 1.000] |
33% |
$0.0028 |
0.0085 |
12.6s |
16.5s |
| mixed |
3 |
0.361 [0.000, 1.000] |
33% |
$0.0039 |
0.0117 |
29.9s |
29.9s |
| verified-synth |
3 |
0.472 [0.167, 1.000] |
33% |
$0.0045 |
0.0135 |
15.0s |
15.0s |
| open |
3 |
0.349 [0.000, 0.989] |
33% |
$0.0049 |
0.0146 |
32.8s |
38.4s |
| nano |
3 |
0.466 [0.083, 0.982] |
33% |
$0.0064 |
0.0191 |
22.6s |
25.6s |
| mid |
3 |
0.375 [0.000, 1.000] |
33% |
$0.0639 |
0.1916 |
30.0s |
30.9s |
| cascade-0.5 |
3 |
0.444 [0.000, 1.000] |
33% |
$0.0387 |
0.1160 |
- |
- |
| auto-route |
3 |
0.422 [0.100, 1.000] |
33% |
$0.0039 |
0.0116 |
13.0s |
26.3s |
| cascade-0.9 |
3 |
0.417 [0.000, 1.000] |
33% |
$0.0363 |
0.1088 |
- |
- |
| auto-route-frontier |
3 |
0.589 [0.100, 1.000] |
67% |
$0.0039 |
0.0058 |
20.6s |
25.1s |
medium
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| vote |
5 |
0.698 [0.333, 0.948] |
60% |
$0.0049 |
0.0082 |
8.8s |
13.3s |
| lenses-off |
5 |
0.650 [0.330, 0.977] |
60% |
$0.0053 |
0.0089 |
21.5s |
29.1s |
| digest |
5 |
0.279 [0.012, 0.667] |
20% |
$0.0051 |
0.0253 |
- |
- |
| select |
5 |
0.801 [0.476, 0.992] |
60% |
$0.0038 |
0.0063 |
13.3s |
23.7s |
| mixed |
5 |
0.613 [0.236, 0.976] |
60% |
$0.0042 |
0.0070 |
- |
- |
| verified-synth |
5 |
0.814 [0.480, 1.000] |
80% |
$0.0046 |
0.0058 |
17.6s |
19.8s |
| open |
5 |
0.379 [0.031, 0.727] |
40% |
$0.0053 |
0.0132 |
29.8s |
29.8s |
| nano |
5 |
0.619 [0.219, 0.979] |
60% |
$0.0073 |
0.0121 |
21.7s |
33.1s |
| mid |
5 |
0.467 [0.067, 0.867] |
40% |
$0.0729 |
0.1822 |
24.4s |
41.5s |
| cascade-0.5 |
5 |
0.751 [0.433, 0.984] |
80% |
$0.0433 |
0.0541 |
- |
- |
| auto-route |
5 |
0.800 [0.466, 1.000] |
80% |
$0.0204 |
0.0255 |
4.6s |
29.5s |
| cascade-0.9 |
5 |
0.860 [0.620, 1.000] |
80% |
$0.0410 |
0.0513 |
- |
- |
| auto-route-frontier |
5 |
0.802 [0.573, 0.989] |
80% |
$0.0202 |
0.0253 |
9.1s |
30.9s |
Judge reliability
Accuracy floor 80%: mimo-pro 100%, muse-ember 88%.
Most different answers, with evidence
- mid (repeat 1): completion 0.00
- gate tests-pass: FAIL
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: 5/6 hidden tests pass; failing: HiddenTests.test_form_fits_small_screens
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +184 -2 lines in 3 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: Output A implements the requested structure: name/email inputs with autocomplete and required, a fieldset+legend grouping the three radios under 'What is this about?', a textarea with maxlength=1000 and hint, and a 'Send message' submit button (index.html lines 12-44). Semantic HTML with lang, title, viewport meta, a single h1, and every control labelled; E5 reports no contrast/focus/a11y violations and E3 reports clean static analysis with no external requests, satisfying offline and contrast basics. However, E1 shows 5/6 hidden tests pass with HiddenTests.test_form_fits_small_screens failing (pass_fraction 0.833), which directly undermines the responsiveness criterion. Also, design_fidelity is weakened because the brief asks for radios grouped in a bordered set but the CSS resets fieldset borders to 0 (style.css lines 51-56), and the radio choices never stack/row-switch at wider viewports since .radio-group is fixed flex-direction: column (lines 83-87) with no media query; the form is otherwise narrow single-column with a roomy 9rem textarea. Focus style (:focus-visible outline) is present. Overall it is a clean, largely correct implementation but fails the small-screen test and
- vote (repeat 1): completion 0.90
- gate tests-pass: pass
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: all 6 hidden tests pass
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +88 -2 lines in 1 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: A delivers a single self-contained index.html with all requested structure: name and email inputs with autocomplete and required, a fieldset/legend radio group 'What is this about?' (Sales/Support/Press) sharing name="topic" so the question is announced, a textarea with maxlength=1000 plus hint and required, and a 'Send message' submit button (E1 tests PASS, E2/E3 build and static PASS, E5 a11y PASS with 0 contrast failures). lang, title, viewport meta, one h1 in order, labelled controls, and a :focus-visible outline are all present; layout uses width:min(100% - 2rem, 42rem) so it fits 390px through 1440px. Deductions: the radio group is in a fieldset with border:0, so it is not the bordered set the design_fidelity brief specifies; there are no explicit responsive breakpoints, only flexible widths; and the textarea min-height 9rem is roomy but modest. Code is clean and minimal but all CSS/JS is inline rather than split into style.css/app.js (acceptable per brief, still slightly less tidy). | Evidence E1 shows 6/6 hidden tests pass, E3 confirms viewport and no external requests, and E5 shows no a11y violations. Index.html implements name/email with autocomplete and required, fieldse
- auto-route (repeat 1): completion 0.83
- gate tests-pass: pass
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: all 6 hidden tests pass
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +43 -2 lines in 1 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: All 6 hidden tests pass (E1), static and build checks are clean (E2, E3), and the a11y scanner reports zero violations (E5). The page (index.html) has lang, title, viewport meta, one h1, labels for every control, a fieldset/legend grouping the radios, maxlength=1000, autocomplete attributes, a visible :focus-visible outline, and a responsive max-width single-column layout with a bordered fieldset and an 8rem-tall textarea (design fidelity and responsiveness are solid). However, the form carries novalidate="" with no accompanying script, which disables the browser's native validation UI despite the brief explicitly asking to use native validation; required attributes are present so required marking is native, but the submit-time validation experience is off (E1's passing tests apparently only check attributes). Minor gaps: radio labels rely on implicit wrapping with no id/for, and there is no app.js despite it being permitted. These are deducted in correctness and ux_polish, though the tested structure and accessibility are otherwise complete. | E1 shows all 6 hidden tests pass and E5/E3 show no a11y/static violations; index.html has name/email with autocomplete, fieldset/legend rad
- vote (repeat 1): completion 0.90
- gate tests-pass: pass
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: all 6 hidden tests pass
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +88 -2 lines in 1 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: A delivers a single self-contained index.html with all requested structure: name and email inputs with autocomplete and required, a fieldset/legend radio group 'What is this about?' (Sales/Support/Press) sharing name="topic" so the question is announced, a textarea with maxlength=1000 plus hint and required, and a 'Send message' submit button (E1 tests PASS, E2/E3 build and static PASS, E5 a11y PASS with 0 contrast failures). lang, title, viewport meta, one h1 in order, labelled controls, and a :focus-visible outline are all present; layout uses width:min(100% - 2rem, 42rem) so it fits 390px through 1440px. Deductions: the radio group is in a fieldset with border:0, so it is not the bordered set the design_fidelity brief specifies; there are no explicit responsive breakpoints, only flexible widths; and the textarea min-height 9rem is roomy but modest. Code is clean and minimal but all CSS/JS is inline rather than split into style.css/app.js (acceptable per brief, still slightly less tidy). | Evidence E1 shows 6/6 hidden tests pass, E3 confirms viewport and no external requests, and E5 shows no a11y violations. Index.html implements name/email with autocomplete and required, fieldse
- auto-route-frontier (repeat 1): completion 0.94
- gate tests-pass: pass
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: all 6 hidden tests pass
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +50 -2 lines in 1 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: E1 (all 6 hidden tests pass), E2 (build clean), E3 (static: viewport meta, no external requests, no lint issues) and E5 (a11y: zero violations, no contrast failures) confirm the brief is met. index.html has lang/title/viewport (lines 2-6), one h1 (line 31), labelled name/email inputs with autocomplete and required (lines 35-40), radio group in a fieldset with legend 'What is this about?' (lines 42-49), a textarea with maxlength=1000 (line 52) and a submit button (line 54). Focus-visible outline (line 25) and 48px min-height button aid keyboard/mobile use. Layout is a single-column
min(100%, 640px) card with clamp padding and flexible choices (lines 11, 21) which fits 390px-1440px. Minor design gaps: the form lacks a live submit handler, has no explicit 'required' asterisk/native-required marking visible to sighted users, and the radio choices wrap rather than stack at phone widths; the message box is roomy (min-height 150px) but the card is 640px which slightly exceeds the strict 'narrow' form width ideal. | All 6 hidden tests pass (E1) with no a11y or static violations (E3,E5); index.html shows native required/type=email validation, autocomplete name/email, fieldset/legend aroun
- vote (repeat 1): completion 0.90
- gate tests-pass: pass
- gate works-offline: pass
- gate no-console-errors: unverified
- evidence E1: tests measured: all 6 hidden tests pass
- evidence E2: build measured: everything compiles
- evidence E3: static measured: clean
- evidence E4: diff_stats measured: +88 -2 lines in 1 files
- evidence E5: a11y measured: no violations (static source rules only)
- evidence E6: screenshot skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- evidence E7: console skipped: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- judge: A delivers a single self-contained index.html with all requested structure: name and email inputs with autocomplete and required, a fieldset/legend radio group 'What is this about?' (Sales/Support/Press) sharing name="topic" so the question is announced, a textarea with maxlength=1000 plus hint and required, and a 'Send message' submit button (E1 tests PASS, E2/E3 build and static PASS, E5 a11y PASS with 0 contrast failures). lang, title, viewport meta, one h1 in order, labelled controls, and a :focus-visible outline are all present; layout uses width:min(100% - 2rem, 42rem) so it fits 390px through 1440px. Deductions: the radio group is in a fieldset with border:0, so it is not the bordered set the design_fidelity brief specifies; there are no explicit responsive breakpoints, only flexible widths; and the textarea min-height 9rem is roomy but modest. Code is clean and minimal but all CSS/JS is inline rather than split into style.css/app.js (acceptable per brief, still slightly less tidy). | Evidence E1 shows 6/6 hidden tests pass, E3 confirms viewport and no external requests, and E5 shows no a11y violations. Index.html implements name/email with autocomplete and required, fieldse
Methodology
- Run: v3-ablation (completed), created 2026-11-19 10:06:52
- Dataset: evals/datasets/v2, split dev, 9 tasks, hash abf13a9e14e00523
- Arm nano: strategy panel-nano: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
- Arm open: strategy panel-open: panel, models deepseek-flash, minimax-flash, qwen-flash, glm, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
- Arm mixed: strategy panel-mixed: panel, models claude-haiku, gpt-luna, deepseek-flash, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
- Arm mid: strategy panel-mid: panel, models gpt-terra, gemini-hydrogen, claude-sonnet, synthesizer claude-sonnet, aggregator llm, rounds 1, judge off
- Arm lenses-off: strategy panel-nano, overrides {'lenses': False}: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
- Arm select: strategy panel-nano-select, overrides {'aggregator': 'select'}: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator select, rounds 1, judge off
- Arm vote: strategy panel-vote: panel, models claude-haiku, gpt-luna, gemini-flash, aggregator vote, rounds 1, judge off
- Arm digest: strategy panel-digest: panel, models claude-haiku, gpt-luna, gemini-flash, aggregator digest, rounds 1, judge off
- Arm cascade-0.5: strategy panel-cascade, overrides {'cascade': {'agreement_threshold': 0.5}}: cascade, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-sonnet, aggregator llm, rounds 1, judge off
- Arm cascade-0.9: strategy panel-cascade, overrides {'cascade': {'agreement_threshold': 0.9}}: cascade, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-sonnet, aggregator llm, rounds 1, judge off
- Arm verified-synth: strategy verified-synth: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator verified-synth, rounds 1, judge off
- Arm auto-route: strategy auto: auto, models claude-haiku, gpt-luna, gemini-flash, gpt-terra, gemini-hydrogen, claude-sonnet, aggregator auto, rounds 1, judge off
- Arm auto-route-frontier: strategy auto-frontier: auto, models claude-haiku, gpt-luna, gemini-flash, gpt-terra, gemini-hydrogen, claude-sonnet, claude-opus, aggregator auto, rounds 1, judge off
- Model claude-haiku: anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Model gpt-luna: openai gpt-6-luna: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Model gemini-flash: google gemini-3.8-flash: $0.75 in / $3.75 out per 1M tokens, verified 2026-11-17
- Model deepseek-flash: openrouter deepseek/deepseek-v4.1-flash-1017: $0.036 in / $0.5 out per 1M tokens, verified 2026-11-17
- Model minimax-flash: openrouter minimax/minimax-m3.1-flash: $0.15 in / $0.6 out per 1M tokens, verified 2026-11-17
- Model qwen-flash: openrouter qwen/qwen3.8-flash: $0.15 in / $0.47 out per 1M tokens, verified 2026-11-17
- Model glm: openrouter z-ai/glm-5.3: $0.0699 in / $1.4 out per 1M tokens, verified 2026-11-17
- Model gpt-terra: openai gpt-6.1-terra: $1 in / $5 out per 1M tokens, verified 2026-11-17
- Model gemini-hydrogen: google gemini-4-hydrogen: $1.5 in / $7.5 out per 1M tokens, verified 2026-11-17
- Model claude-sonnet: anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
- Model claude-opus: anthropic claude-opus-5-5: $4 in / $20 out per 1M tokens, verified 2026-11-17
- Model mimo-pro: openrouter xiaomi/mimo-v2.6-pro: $0.435 in / $0.87 out per 1M tokens, verified 2026-11-17
- Model muse-ember: openrouter meta/muse-ember: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Judge models: mimo-pro, muse-ember
- Repeats / seed: 1 per task / seed 0
- Intervals: 95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
- Verdict rules: non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline vote
- Spend: $1.96 on the arms + $0.1240 on scoring and judging (cap $3.40)
- Fusion version: 0.2.0 (git 5abab47, with uncommitted changes)
Supporting study: v3-route
Arms: rung0, rung1, rung2. Baseline for the comparisons: rung0.
Some prices were unknown for rung1, rung2: its costs are a lower bound.
1 repeat(s) per task: variance across repeats is poorly estimated.
Headline
Each arm against rung0, on all tasks. ✓ yes, ✗ no, ? inconclusive, ⊘ blocked. The reasons are listed below the table.
| Arm |
Paired tasks |
Cheaper (per solved task) |
Faster |
Not worse |
Better |
| rung1 |
20 |
✗ no |
✗ no |
✓ yes |
✓ yes |
| rung2 |
20 |
✗ no |
✗ no |
✓ yes |
✓ yes |
- rung1 vs rung0 on all (20 tasks): cheaper (per solved task): no. cost per solved task ratio 5.16 [2.57, 9.97] (below 1 is better): the whole interval is at or above 1.
- rung1 vs rung0 on all (20 tasks): faster: no. seconds per task ratio 1.86 [1.57, 2.15] (below 1 is better): the whole interval is at or above 1.
- rung1 vs rung0 on all (20 tasks): not worse: yes. quality difference +0.097 [+0.000, +0.189] over 20 tasks: the lower bound is above -0.03, the non-inferiority margin.
- rung1 vs rung0 on all (20 tasks): better: yes. quality difference +0.097 [+0.000, +0.189] over 20 tasks: the interval is above 0.
- rung2 vs rung0 on all (20 tasks): cheaper (per solved task): no. cost per solved task ratio 81.23 [49.46, 129.12] (below 1 is better): the whole interval is at or above 1.
- rung2 vs rung0 on all (20 tasks): faster: no. seconds per task ratio 4.96 [4.30, 5.70] (below 1 is better): the whole interval is at or above 1.
- rung2 vs rung0 on all (20 tasks): not worse: yes. quality difference +0.144 [+0.064, +0.243] over 20 tasks: the lower bound is above -0.03, the non-inferiority margin.
- rung2 vs rung0 on all (20 tasks): better: yes. quality difference +0.144 [+0.064, +0.243] over 20 tasks: the interval is above 0.
Arms
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
Frontier |
| rung0 |
20 |
0.464 [0.346, 0.592] |
40% |
$0.0008 |
0.0019 |
5.6s |
7.2s |
$ s |
| rung1 |
20 |
0.560 [0.434, 0.692] |
45% |
$0.0044 |
0.0099 |
10.7s |
14.7s |
$ s |
| rung2 |
20 |
0.608 [0.469, 0.750] |
45% |
$0.0700 |
0.1556 |
27.2s |
36.3s |
$ s |
Frontier: $ = on the cost-quality frontier (no arm is cheaper per task and at least as good), s = on the latency-quality frontier (median seconds).
Efficiency and speed
| Arm |
Out tok/s (effective) |
Tokens / solved |
Quality / $ |
Quality / min |
Calls |
Errors |
| rung0 |
198 |
8,740 |
605.0 |
4.877 |
1.0 |
0 |
| rung1 |
234 |
19,147 |
125.9 |
3.175 |
3.5 |
0 |
| rung2 |
208 |
36,888 |
8.7 |
1.288 |
4.2 |
0 |
Streamed speed per model:
| Arm |
Model |
Decode tok/s |
TTFT ms |
| rung0 |
claude-haiku |
284 |
1493 |
| rung1 |
claude-haiku |
276 |
1560 |
| rung1 |
gemini-flash |
165 |
3827 |
| rung1 |
gpt-luna |
306 |
4135 |
| rung2 |
claude-sonnet |
175 |
1934 |
| rung2 |
gemini-hydrogen |
4648 |
5954 |
| rung2 |
gpt-terra |
115 |
2729 |
Cost of measuring
Scoring and judging cost and time, kept apart: never part of an arm's cost or latency.
| Arm |
Eval $ total |
Eval $ / item |
Eval s / item |
| rung0 |
$0.0217 |
$0.0011 |
0.0s |
| rung1 |
$0.0238 |
$0.0012 |
0.0s |
| rung2 |
$0.0306 |
$0.0015 |
0.0s |
Variance across repeats
| Arm |
Mean sd of quality across repeats |
Tasks whose solved flips |
| rung0 |
- |
- |
| rung1 |
- |
- |
| rung2 |
- |
- |
Comparisons against the baseline
rung1 vs rung0
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
20 |
+0.097 [+0.000, +0.189] |
11/6/3 |
0.057 |
+0.45 |
5.16 [2.57, 9.97] |
1.86 [1.57, 2.15] |
✗ |
✗ |
✓ |
✓ |
| architecture |
5 |
+0.073 [-0.200, +0.360] |
2/1/2 |
1.000 |
+0.20 |
4.05 |
2.07 [1.74, 2.46] |
? |
? |
? |
? |
| code_review |
5 |
+0.055 [-0.122, +0.200] |
3/1/1 |
0.625 |
+0.27 |
5.47 |
1.50 [0.90, 2.72] |
? |
? |
? |
? |
| debugging |
5 |
+0.072 [+0.000, +0.156] |
2/3/0 |
0.500 |
+0.71 |
6.07 [5.05, 7.47] |
1.75 [1.36, 2.29] |
? |
? |
? |
? |
| planning |
5 |
+0.187 [+0.086, +0.288] |
4/1/0 |
0.125 |
+1.35 |
- |
2.06 [1.89, 2.34] |
? |
? |
? |
? |
- architecture (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
rung2 vs rung0
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
20 |
+0.144 [+0.064, +0.243] |
12/7/1 |
0.003 |
+0.70 |
81.23 [49.46, 129.12] |
4.96 [4.30, 5.70] |
✗ |
✗ |
✓ |
✓ |
| architecture |
5 |
+0.127 [+0.013, +0.273] |
3/2/0 |
0.250 |
+0.76 |
100.40 [89.73, 111.07] |
5.33 [4.77, 5.93] |
? |
? |
? |
? |
| code_review |
5 |
+0.126 [-0.072, +0.383] |
4/0/1 |
0.375 |
+0.44 |
73.94 |
4.71 [2.92, 7.56] |
? |
? |
? |
? |
| debugging |
5 |
+0.114 [+0.000, +0.258] |
2/3/0 |
0.500 |
+0.69 |
87.84 [77.63, 101.33] |
4.25 [3.85, 4.69] |
? |
? |
? |
? |
| planning |
5 |
+0.210 [+0.029, +0.410] |
3/2/0 |
0.250 |
+0.86 |
- |
5.48 [4.32, 6.79] |
? |
? |
? |
? |
- architecture (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (5 tasks): only 5 task(s) answered by both arms; at least 10 are needed to claim anything.
Δ quality is the arm minus the baseline, paired by task. Ratios are the arm over the baseline (below 1 is better). W/T/L counts tasks the arm won, tied (within 0.02) or lost. Effect d is the mean paired difference over its spread across tasks.
By category
architecture
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| rung0 |
5 |
0.467 [0.347, 0.587] |
40% |
$0.0008 |
0.0019 |
6.1s |
6.6s |
| rung1 |
5 |
0.540 [0.347, 0.733] |
40% |
$0.0031 |
0.0078 |
12.3s |
14.9s |
| rung2 |
5 |
0.593 [0.420, 0.813] |
40% |
$0.0772 |
0.1929 |
32.1s |
37.5s |
code_review
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| rung0 |
5 |
0.304 [0.148, 0.509] |
20% |
$0.0008 |
0.0038 |
4.9s |
7.7s |
| rung1 |
5 |
0.359 [0.224, 0.517] |
20% |
$0.0041 |
0.0206 |
6.9s |
14.9s |
| rung2 |
5 |
0.430 [0.210, 0.740] |
20% |
$0.0558 |
0.2790 |
23.8s |
37.8s |
debugging
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| rung0 |
5 |
0.856 [0.742, 0.970] |
100% |
$0.0008 |
0.0008 |
5.4s |
7.3s |
| rung1 |
5 |
0.928 [0.844, 1.000] |
100% |
$0.0047 |
0.0047 |
9.4s |
11.5s |
| rung2 |
5 |
0.970 [0.910, 1.000] |
100% |
$0.0683 |
0.0683 |
24.5s |
26.7s |
planning
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| rung0 |
5 |
0.228 [0.113, 0.331] |
0% |
$0.0008 |
- |
5.8s |
7.2s |
| rung1 |
5 |
0.414 [0.186, 0.600] |
20% |
$0.0058 |
0.0292 |
11.5s |
14.7s |
| rung2 |
5 |
0.437 [0.186, 0.686] |
20% |
$0.0788 |
0.3941 |
34.6s |
36.3s |
By difficulty
easy
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| rung0 |
9 |
0.491 [0.373, 0.619] |
44% |
$0.0007 |
0.0017 |
6.0s |
6.6s |
| rung1 |
9 |
0.628 [0.487, 0.772] |
56% |
$0.0041 |
0.0074 |
11.1s |
14.9s |
| rung2 |
9 |
0.652 [0.493, 0.830] |
44% |
$0.0726 |
0.1633 |
29.9s |
37.8s |
hard
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| rung0 |
7 |
0.383 [0.218, 0.576] |
29% |
$0.0007 |
0.0026 |
5.6s |
7.2s |
| rung1 |
7 |
0.472 [0.320, 0.656] |
29% |
$0.0049 |
0.0170 |
11.5s |
14.7s |
| rung2 |
7 |
0.584 [0.366, 0.795] |
43% |
$0.0660 |
0.1540 |
27.2s |
36.3s |
medium
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| rung0 |
4 |
0.542 [0.083, 1.000] |
50% |
$0.0009 |
0.0017 |
5.4s |
7.7s |
| rung1 |
4 |
0.562 [0.125, 1.000] |
50% |
$0.0045 |
0.0090 |
8.5s |
10.4s |
| rung2 |
4 |
0.550 [0.100, 1.000] |
50% |
$0.0713 |
0.1425 |
26.4s |
35.3s |
Methodology
- Run: v3-route (completed), created 2026-11-19 10:22:39
- Dataset: evals/datasets/v2, split dev, 20 tasks, hash cc49b32c66c27cbd
- Arm rung0: strategy solo-haiku-check: solo, models claude-haiku, aggregator auto, rounds 1, judge off
- Arm rung1: strategy panel-nano-auto: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator auto, rounds 1, judge off
- Arm rung2: strategy panel-mid: panel, models gpt-terra, gemini-hydrogen, claude-sonnet, synthesizer claude-sonnet, aggregator llm, rounds 1, judge off
- Model claude-haiku: anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Model gpt-luna: openai gpt-6-luna: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Model gemini-flash: google gemini-3.8-flash: $0.75 in / $3.75 out per 1M tokens, verified 2026-11-17
- Model gpt-terra: openai gpt-6.1-terra: $1 in / $5 out per 1M tokens, verified 2026-11-17
- Model gemini-hydrogen: google gemini-4-hydrogen: $1.5 in / $7.5 out per 1M tokens, verified 2026-11-17
- Model claude-sonnet: anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
- Model mimo-pro: openrouter xiaomi/mimo-v2.6-pro: $0.435 in / $0.87 out per 1M tokens, verified 2026-11-17
- Model muse-ember: openrouter meta/muse-ember: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Judge models: mimo-pro, muse-ember
- Repeats / seed: 1 per task / seed 7
- Intervals: 95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
- Verdict rules: non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline rung0
- Spend: $1.50 on the arms + $0.0761 on scoring and judging (cap $2.40)
- Fusion version: 0.2.0 (git 5abab47, with uncommitted changes)
Supporting study: v3-latency
Arms: luna-none, solo-opus, panel-duo, auto-route. Baseline for the comparisons: solo-opus.
1 job(s) halted on their first attempt (a call timed out or the panel had no quorum) and were run again, with longer timeouts where it was a timeout: solo-opus (1). Only the last attempt is scored; the first attempt's cost is in the totals. A halted job scores 0, so without the retry these arms would be understated by the harness's limit, not by their answers.
Some prices were unknown for auto-route: its costs are a lower bound.
1 repeat(s) per task: variance across repeats is poorly estimated.
Headline
Each arm against solo-opus, on all tasks. ✓ yes, ✗ no, ? inconclusive, ⊘ blocked. The reasons are listed below the table.
| Arm |
Paired tasks |
Cheaper (per solved task) |
Faster |
Not worse |
Better |
| luna-none |
30 |
✓ yes |
✓ yes |
? inconclusive |
? inconclusive |
| panel-duo |
30 |
✓ yes |
✓ yes |
? inconclusive |
? inconclusive |
| auto-route |
30 |
✓ yes |
✓ yes |
? inconclusive |
✗ no |
- luna-none vs solo-opus on all (30 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.04 [0.03, 0.06] (below 1 is better): the whole interval is below 1.
- luna-none vs solo-opus on all (30 tasks): faster: yes. seconds per task ratio 0.70 [0.63, 0.78] (below 1 is better): the whole interval is below 1.
- luna-none vs solo-opus on all (30 tasks): not worse: inconclusive. quality difference -0.052 [-0.182, +0.075] over 30 tasks: the interval straddles the -0.03 margin.
- luna-none vs solo-opus on all (30 tasks): better: inconclusive. quality difference -0.052 [-0.182, +0.075] over 30 tasks: the interval includes 0.
- panel-duo vs solo-opus on all (30 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.05 [0.04, 0.06] (below 1 is better): the whole interval is below 1.
- panel-duo vs solo-opus on all (30 tasks): faster: yes. seconds per task ratio 0.82 [0.74, 0.91] (below 1 is better): the whole interval is below 1.
- panel-duo vs solo-opus on all (30 tasks): not worse: inconclusive. quality difference -0.065 [-0.178, +0.043] over 30 tasks: the interval straddles the -0.03 margin.
- panel-duo vs solo-opus on all (30 tasks): better: inconclusive. quality difference -0.065 [-0.178, +0.043] over 30 tasks: the interval includes 0.
- auto-route vs solo-opus on all (30 tasks): cheaper (per solved task): yes. cost per solved task ratio 0.11 [0.08, 0.15] (below 1 is better): the whole interval is below 1.
- auto-route vs solo-opus on all (30 tasks): faster: yes. seconds per task ratio 0.74 [0.64, 0.83] (below 1 is better): the whole interval is below 1.
- auto-route vs solo-opus on all (30 tasks): not worse: inconclusive. quality difference -0.134 [-0.259, -0.009] over 30 tasks: the interval straddles the -0.03 margin.
- auto-route vs solo-opus on all (30 tasks): better: no. quality difference -0.134 [-0.259, -0.009] over 30 tasks: the interval is at or below 0.
Arms
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
Frontier |
| luna-none |
30 |
0.743 [0.635, 0.845] |
73% |
$0.0021 |
0.0028 |
14.2s |
17.4s |
$ s |
| solo-opus |
30 |
0.794 [0.684, 0.890] |
77% |
$0.0486 |
0.0634 |
18.5s |
25.0s |
$ s |
| panel-duo |
30 |
0.730 [0.628, 0.829] |
73% |
$0.0022 |
0.0030 |
16.8s |
20.5s |
- |
| auto-route |
30 |
0.660 [0.544, 0.772] |
63% |
$0.0044 |
0.0069 |
12.6s |
22.9s |
s |
Frontier: $ = on the cost-quality frontier (no arm is cheaper per task and at least as good), s = on the latency-quality frontier (median seconds).
Efficiency and speed
| Arm |
Out tok/s (effective) |
Tokens / solved |
Quality / $ |
Quality / min |
Calls |
Errors |
| luna-none |
227 |
12,957 |
355.6 |
3.313 |
2.8 |
0 |
| solo-opus |
106 |
5,660 |
16.3 |
2.478 |
1.0 |
0 |
| panel-duo |
210 |
13,345 |
328.8 |
2.767 |
2.8 |
0 |
| auto-route |
192 |
13,222 |
151.6 |
2.792 |
3.1 |
0 |
Streamed speed per model:
| Arm |
Model |
Decode tok/s |
TTFT ms |
| luna-none |
claude-haiku |
275 |
1850 |
| luna-none |
gpt-luna |
126 |
776 |
| solo-opus |
claude-opus |
142 |
3212 |
| panel-duo |
claude-haiku |
284 |
1733 |
| panel-duo |
gpt-luna |
240 |
4544 |
| auto-route |
claude-haiku |
262 |
1360 |
| auto-route |
gemini-flash |
143 |
4091 |
| auto-route |
gpt-luna |
223 |
4133 |
Cost of measuring
Scoring and judging cost and time, kept apart: never part of an arm's cost or latency.
| Arm |
Eval $ total |
Eval $ / item |
Eval s / item |
| luna-none |
$0 |
$0 |
0.0s |
| solo-opus |
$0 |
$0 |
0.0s |
| panel-duo |
$0 |
$0 |
0.0s |
| auto-route |
$0 |
$0 |
0.0s |
Variance across repeats
| Arm |
Mean sd of quality across repeats |
Tasks whose solved flips |
| luna-none |
- |
- |
| solo-opus |
- |
- |
| panel-duo |
- |
- |
| auto-route |
- |
- |
Comparisons against the baseline
luna-none vs solo-opus
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
30 |
-0.052 [-0.182, +0.075] |
7/10/13 |
0.263 |
-0.14 |
0.04 [0.03, 0.06] |
0.70 [0.63, 0.78] |
✓ |
✓ |
? |
? |
| architecture |
7 |
-0.119 [-0.238, +0.024] |
1/2/4 |
0.375 |
-0.64 |
0.04 [0.04, 0.05] |
0.64 [0.55, 0.77] |
? |
? |
? |
? |
| code_review |
8 |
-0.231 [-0.491, -0.006] |
2/2/4 |
0.688 |
-0.63 |
0.15 [0.06, 0.37] |
0.85 [0.67, 1.02] |
? |
? |
? |
? |
| debugging |
8 |
+0.162 [-0.175, +0.500] |
3/3/2 |
1.000 |
+0.32 |
0.03 [0.01, 0.05] |
0.72 [0.56, 0.88] |
? |
? |
? |
? |
| planning |
7 |
-0.024 [-0.122, +0.099] |
1/3/3 |
0.625 |
-0.15 |
0.03 [0.02, 0.04] |
0.65 [0.57, 0.73] |
? |
? |
? |
? |
- architecture (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.
panel-duo vs solo-opus
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
30 |
-0.065 [-0.178, +0.043] |
5/15/10 |
0.302 |
-0.20 |
0.05 [0.04, 0.06] |
0.82 [0.74, 0.91] |
✓ |
✓ |
? |
? |
| architecture |
7 |
-0.119 [-0.238, -0.024] |
0/4/3 |
0.250 |
-0.75 |
0.04 [0.04, 0.05] |
0.78 [0.64, 0.92] |
? |
? |
? |
? |
| code_review |
8 |
-0.201 [-0.485, +0.040] |
2/3/3 |
1.000 |
-0.52 |
0.10 [0.05, 0.35] |
0.85 [0.65, 1.05] |
? |
? |
? |
? |
| debugging |
8 |
+0.099 [-0.206, +0.366] |
3/3/2 |
1.000 |
+0.23 |
0.03 [0.01, 0.07] |
0.94 [0.71, 1.17] |
? |
? |
? |
? |
| planning |
7 |
-0.041 [-0.082, +0.000] |
0/5/2 |
0.500 |
-0.59 |
0.04 [0.04, 0.05] |
0.76 [0.70, 0.81] |
? |
? |
? |
? |
- architecture (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.
auto-route vs solo-opus
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
30 |
-0.134 [-0.259, -0.009] |
5/11/14 |
0.064 |
-0.38 |
0.11 [0.08, 0.15] |
0.74 [0.64, 0.83] |
✓ |
✓ |
? |
✗ |
| architecture |
7 |
-0.152 [-0.286, -0.048] |
0/3/4 |
0.125 |
-0.89 |
0.12 [0.09, 0.20] |
0.87 [0.74, 1.04] |
? |
? |
? |
? |
| code_review |
8 |
-0.275 [-0.554, -0.004] |
2/1/5 |
0.453 |
-0.62 |
0.23 |
0.69 [0.48, 0.97] |
? |
? |
? |
? |
| debugging |
8 |
+0.024 [-0.314, +0.297] |
3/4/1 |
0.625 |
+0.05 |
0.05 [0.02, 0.12] |
0.45 [0.35, 0.58] |
? |
? |
? |
? |
| planning |
7 |
-0.137 [-0.235, -0.048] |
0/3/4 |
0.125 |
-0.97 |
0.11 [0.11, 0.12] |
0.87 [0.76, 0.96] |
? |
? |
? |
? |
- architecture (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.
- code_review (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
- debugging (8 tasks): only 8 task(s) answered by both arms; at least 10 are needed to claim anything.
- planning (7 tasks): only 7 task(s) answered by both arms; at least 10 are needed to claim anything.
Δ quality is the arm minus the baseline, paired by task. Ratios are the arm over the baseline (below 1 is better). W/T/L counts tasks the arm won, tied (within 0.02) or lost. Effect d is the mean paired difference over its spread across tasks.
By category
architecture
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| luna-none |
7 |
0.748 [0.600, 0.881] |
86% |
$0.0025 |
0.0029 |
15.6s |
18.6s |
| solo-opus |
7 |
0.867 [0.695, 0.976] |
86% |
$0.0594 |
0.0693 |
23.1s |
39.9s |
| panel-duo |
7 |
0.748 [0.614, 0.881] |
86% |
$0.0026 |
0.0030 |
19.1s |
22.9s |
| auto-route |
7 |
0.714 [0.548, 0.881] |
71% |
$0.0060 |
0.0084 |
21.0s |
27.9s |
code_review
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| luna-none |
8 |
0.430 [0.264, 0.635] |
25% |
$0.0017 |
0.0067 |
12.1s |
14.4s |
| solo-opus |
8 |
0.661 [0.427, 0.875] |
75% |
$0.0338 |
0.0451 |
11.9s |
16.6s |
| panel-duo |
8 |
0.460 [0.281, 0.654] |
38% |
$0.0018 |
0.0047 |
11.5s |
15.2s |
| auto-route |
8 |
0.387 [0.211, 0.607] |
25% |
$0.0026 |
0.0105 |
9.0s |
15.8s |
debugging
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| luna-none |
8 |
0.900 [0.725, 1.000] |
88% |
$0.0019 |
0.0022 |
13.2s |
17.4s |
| solo-opus |
8 |
0.738 [0.475, 0.912] |
62% |
$0.0441 |
0.0705 |
16.0s |
22.0s |
| panel-duo |
8 |
0.836 [0.650, 0.975] |
88% |
$0.0020 |
0.0023 |
15.5s |
23.5s |
| auto-route |
8 |
0.761 [0.537, 0.948] |
75% |
$0.0026 |
0.0034 |
6.5s |
10.9s |
planning
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| luna-none |
7 |
0.915 [0.867, 0.976] |
100% |
$0.0024 |
0.0024 |
14.9s |
18.9s |
| solo-opus |
7 |
0.939 [0.816, 1.000] |
86% |
$0.0599 |
0.0698 |
23.0s |
26.3s |
| panel-duo |
7 |
0.898 [0.776, 0.980] |
86% |
$0.0026 |
0.0030 |
18.2s |
20.2s |
| auto-route |
7 |
0.802 [0.645, 0.952] |
86% |
$0.0067 |
0.0078 |
22.6s |
24.3s |
By difficulty
easy
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| luna-none |
7 |
0.881 [0.762, 0.976] |
100% |
$0.0021 |
0.0021 |
14.7s |
16.7s |
| solo-opus |
7 |
0.729 [0.505, 0.929] |
71% |
$0.0482 |
0.0675 |
18.7s |
26.1s |
| panel-duo |
7 |
0.811 [0.730, 0.891] |
100% |
$0.0023 |
0.0023 |
19.0s |
22.9s |
| auto-route |
7 |
0.805 [0.647, 0.940] |
71% |
$0.0043 |
0.0061 |
18.1s |
27.9s |
hard
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| luna-none |
7 |
0.711 [0.457, 0.932] |
71% |
$0.0023 |
0.0032 |
14.4s |
18.9s |
| solo-opus |
7 |
0.571 [0.314, 0.820] |
43% |
$0.0462 |
0.1079 |
18.5s |
23.3s |
| panel-duo |
7 |
0.658 [0.448, 0.867] |
57% |
$0.0022 |
0.0038 |
16.8s |
23.5s |
| auto-route |
7 |
0.545 [0.361, 0.714] |
57% |
$0.0039 |
0.0069 |
12.6s |
20.5s |
medium
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| luna-none |
16 |
0.696 [0.545, 0.832] |
62% |
$0.0020 |
0.0032 |
12.4s |
16.6s |
| solo-opus |
16 |
0.921 [0.825, 1.000] |
94% |
$0.0498 |
0.0531 |
17.5s |
26.3s |
| panel-duo |
16 |
0.725 [0.575, 0.880] |
69% |
$0.0022 |
0.0032 |
15.5s |
20.5s |
| auto-route |
16 |
0.647 [0.456, 0.817] |
62% |
$0.0045 |
0.0073 |
11.0s |
24.3s |
Methodology
- Run: v3-latency (completed), created 2026-11-19 10:33:44
- Dataset: evals/datasets/v2, split dev, 30 tasks, hash de8a9a49c422425a
- Arm solo-opus: strategy solo-opus: solo, models claude-opus, aggregator auto, rounds 1, judge off
- Arm auto-route: strategy auto: auto, models claude-haiku, gpt-luna, gemini-flash, gpt-terra, gemini-hydrogen, claude-sonnet, aggregator auto, rounds 1, judge off
- Arm panel-duo: strategy panel-duo: panel, models claude-haiku, gpt-luna, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
- Arm luna-none: strategy panel-duo, overrides {'members': [{'model': 'claude-haiku'}, {'model': 'gpt-luna', 'reasoning_effort': 'none'}]}: panel, models claude-haiku, gpt-luna, synthesizer claude-haiku, aggregator llm, rounds 1, judge off
- Model claude-opus: anthropic claude-opus-5-5: $4 in / $20 out per 1M tokens, verified 2026-11-17
- Model claude-haiku: anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Model gpt-luna: openai gpt-6-luna: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Model gemini-flash: google gemini-3.8-flash: $0.75 in / $3.75 out per 1M tokens, verified 2026-11-17
- Model gpt-terra: openai gpt-6.1-terra: $1 in / $5 out per 1M tokens, verified 2026-11-17
- Model gemini-hydrogen: google gemini-4-hydrogen: $1.5 in / $7.5 out per 1M tokens, verified 2026-11-17
- Model claude-sonnet: anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
- Judge models: none (deterministic scorers only)
- Repeats / seed: 1 per task / seed 0
- Intervals: 95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
- Verdict rules: non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline solo-opus
- Spend: $1.72 on the arms + $0 on scoring and judging (cap $2.35)
- Fusion version: 0.2.0 (git 5abab47, with uncommitted changes)
Supporting study: v3-e2e
Arms: sonnet, opus, sonnet+fusion, opus+fusion, haiku, haiku+fusion. Baseline for the comparisons: sonnet.
1 repeat(s) per task: variance across repeats is poorly estimated.
Fewer than 10 tasks are answered by both an arm and the baseline: no verdict is possible for it.
Headline
Each arm against sonnet, on all tasks. ✓ yes, ✗ no, ? inconclusive, ⊘ blocked. The reasons are listed below the table.
| Arm |
Paired tasks |
Cheaper (per solved task) |
Faster |
Not worse |
Better |
| opus |
4 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| sonnet+fusion |
4 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| opus+fusion |
4 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| haiku |
4 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
| haiku+fusion |
4 |
? inconclusive |
? inconclusive |
? inconclusive |
? inconclusive |
- opus vs sonnet (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
- sonnet+fusion vs sonnet (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
- opus+fusion vs sonnet (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
- haiku vs sonnet (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
- haiku+fusion vs sonnet (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
Arms
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
Frontier |
| sonnet |
4 |
1.000 [1.000, 1.000] |
100% |
$0.0256 |
0.0256 |
12.9s |
18.9s |
s |
| opus |
4 |
1.000 [1.000, 1.000] |
100% |
$0.0678 |
0.0678 |
25.6s |
27.4s |
- |
| sonnet+fusion |
4 |
1.000 [1.000, 1.000] |
100% |
$0.0504 |
0.0504 |
13.3s |
24.6s |
- |
| opus+fusion |
4 |
1.000 [1.000, 1.000] |
100% |
$0.1221 |
0.1221 |
23.1s |
43.0s |
- |
| haiku |
4 |
0.992 [0.976, 1.000] |
75% |
$0.0027 |
0.0035 |
17.2s |
22.7s |
$ |
| haiku+fusion |
4 |
1.000 [1.000, 1.000] |
100% |
$0.0058 |
0.0058 |
21.5s |
34.8s |
$ |
Frontier: $ = on the cost-quality frontier (no arm is cheaper per task and at least as good), s = on the latency-quality frontier (median seconds).
Efficiency and speed
| Arm |
Out tok/s (effective) |
Tokens / solved |
Quality / $ |
Quality / min |
Calls |
Errors |
| sonnet |
- |
21,748 |
39.0 |
3.834 |
5.2 |
0 |
| opus |
- |
21,376 |
14.7 |
2.361 |
5.8 |
0 |
| sonnet+fusion |
- |
96,689 |
19.9 |
3.545 |
4.8 |
0 |
| opus+fusion |
- |
105,501 |
8.2 |
2.028 |
5.8 |
0 |
| haiku |
- |
44,076 |
374.1 |
3.187 |
6.0 |
0 |
| haiku+fusion |
- |
169,798 |
173.4 |
2.216 |
7.8 |
0 |
Cost of measuring
Scoring and judging cost and time, kept apart: never part of an arm's cost or latency.
| Arm |
Eval $ total |
Eval $ / item |
Eval s / item |
| sonnet |
$0 |
$0 |
0.7s |
| opus |
$0 |
$0 |
0.7s |
| sonnet+fusion |
$0 |
$0 |
0.6s |
| opus+fusion |
$0 |
$0 |
1.4s |
| haiku |
$0 |
$0 |
1.2s |
| haiku+fusion |
$0 |
$0 |
0.6s |
Variance across repeats
| Arm |
Mean sd of quality across repeats |
Tasks whose solved flips |
| sonnet |
- |
- |
| opus |
- |
- |
| sonnet+fusion |
- |
- |
| opus+fusion |
- |
- |
| haiku |
- |
- |
| haiku+fusion |
- |
- |
Claude Code end to end
Each run is headless Claude Code on a temp copy of the task; the hidden tests decide solved. Costs are means per run: the host's from Claude Code's own JSON, Fusion's from its ledger.
| Arm |
Executor |
Fusion |
Runs |
Solved |
Host $ |
Fusion $ |
Total $ |
$/solved |
p50 |
Turns |
Tool calls |
Fusion calls |
| sonnet |
sonnet |
off |
4 |
100% |
$0.0256 |
$0 |
$0.0256 |
$0.0256 |
12.9s |
5.2 |
4.2 |
0.0 |
| opus |
opus |
off |
4 |
100% |
$0.0678 |
$0 |
$0.0678 |
$0.0678 |
25.6s |
5.8 |
4.8 |
0.0 |
| sonnet+fusion |
sonnet |
on |
4 |
100% |
$0.0504 |
$0 |
$0.0504 |
$0.0504 |
13.3s |
4.8 |
3.8 |
0.0 |
| opus+fusion |
opus |
on |
4 |
100% |
$0.1218 |
$0.0003 |
$0.1221 |
$0.1221 |
23.1s |
5.5 |
4.5 |
0.2 |
| haiku |
haiku |
off |
4 |
75% |
$0.0027 |
$0 |
$0.0027 |
$0.0035 |
17.2s |
6.0 |
5.0 |
0.0 |
| haiku+fusion |
haiku |
on |
4 |
100% |
$0.0055 |
$0.0003 |
$0.0058 |
$0.0058 |
21.5s |
7.5 |
6.5 |
0.2 |
Comparisons against the baseline
opus vs sonnet
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
4 |
+0.000 [+0.000, +0.000] |
0/4/0 |
- |
- |
2.65 [2.26, 3.31] |
1.62 [1.38, 2.00] |
? |
? |
? |
? |
| coding |
4 |
+0.000 [+0.000, +0.000] |
0/4/0 |
- |
- |
2.65 [2.26, 3.31] |
1.62 [1.41, 2.00] |
? |
? |
? |
? |
- coding (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
sonnet+fusion vs sonnet
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
4 |
+0.000 [+0.000, +0.000] |
0/4/0 |
- |
- |
1.97 [1.06, 2.45] |
1.08 [0.94, 1.23] |
? |
? |
? |
? |
| coding |
4 |
+0.000 [+0.000, +0.000] |
0/4/0 |
- |
- |
1.97 [1.14, 2.45] |
1.08 [0.94, 1.23] |
? |
? |
? |
? |
- coding (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
opus+fusion vs sonnet
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
4 |
+0.000 [+0.000, +0.000] |
0/4/0 |
- |
- |
4.76 [3.17, 6.17] |
1.89 [1.67, 2.19] |
? |
? |
? |
? |
| coding |
4 |
+0.000 [+0.000, +0.000] |
0/4/0 |
- |
- |
4.76 [3.17, 6.17] |
1.89 [1.67, 2.19] |
? |
? |
? |
? |
- coding (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
haiku vs sonnet
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
4 |
-0.008 [-0.024, +0.000] |
0/3/1 |
1.000 |
-0.50 |
0.14 [0.10, 0.40] |
1.19 [1.10, 1.35] |
? |
? |
? |
? |
| coding |
4 |
-0.008 [-0.024, +0.000] |
0/3/1 |
1.000 |
-0.50 |
0.14 [0.10, 0.40] |
1.19 [1.10, 1.35] |
? |
? |
? |
? |
- coding (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
haiku+fusion vs sonnet
| Scope |
Tasks |
Δ quality (95% CI) |
W/T/L |
Sign test p |
Effect d |
$/solved ratio |
Seconds ratio |
Cheaper |
Faster |
Not worse |
Better |
| all |
4 |
+0.000 [+0.000, +0.000] |
0/4/0 |
- |
- |
0.23 [0.17, 0.27] |
1.73 [1.56, 1.83] |
? |
? |
? |
? |
| coding |
4 |
+0.000 [+0.000, +0.000] |
0/4/0 |
- |
- |
0.23 [0.17, 0.27] |
1.73 [1.56, 1.83] |
? |
? |
? |
? |
- coding (4 tasks): only 4 task(s) answered by both arms; at least 10 are needed to claim anything.
Δ quality is the arm minus the baseline, paired by task. Ratios are the arm over the baseline (below 1 is better). W/T/L counts tasks the arm won, tied (within 0.02) or lost. Effect d is the mean paired difference over its spread across tasks.
By category
coding
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| sonnet |
4 |
1.000 [1.000, 1.000] |
100% |
$0.0256 |
0.0256 |
12.9s |
18.9s |
| opus |
4 |
1.000 [1.000, 1.000] |
100% |
$0.0678 |
0.0678 |
25.6s |
27.4s |
| sonnet+fusion |
4 |
1.000 [1.000, 1.000] |
100% |
$0.0504 |
0.0504 |
13.3s |
24.6s |
| opus+fusion |
4 |
1.000 [1.000, 1.000] |
100% |
$0.1221 |
0.1221 |
23.1s |
43.0s |
| haiku |
4 |
0.992 [0.976, 1.000] |
75% |
$0.0027 |
0.0035 |
17.2s |
22.7s |
| haiku+fusion |
4 |
1.000 [1.000, 1.000] |
100% |
$0.0058 |
0.0058 |
21.5s |
34.8s |
By difficulty
easy
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| sonnet |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0240 |
0.0240 |
12.9s |
18.9s |
| opus |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0635 |
0.0635 |
21.3s |
25.6s |
| sonnet+fusion |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0533 |
0.0533 |
11.7s |
24.6s |
| opus+fusion |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0763 |
0.0763 |
23.1s |
31.1s |
| haiku |
2 |
0.984 [0.968, 1.000] |
50% |
$0.0026 |
0.0052 |
14.6s |
22.7s |
| haiku+fusion |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0064 |
0.0064 |
18.7s |
34.8s |
medium
| Arm |
Tasks |
Quality (95% CI) |
Solved |
$/task |
$/solved |
p50 |
p90 |
| sonnet |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0272 |
0.0272 |
12.0s |
18.8s |
| opus |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0722 |
0.0722 |
27.4s |
27.4s |
| sonnet+fusion |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0474 |
0.0474 |
13.3s |
18.1s |
| opus+fusion |
2 |
1.000 [1.000, 1.000] |
100% |
$0.1678 |
0.1678 |
21.2s |
43.0s |
| haiku |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0027 |
0.0027 |
17.2s |
20.2s |
| haiku+fusion |
2 |
1.000 [1.000, 1.000] |
100% |
$0.0052 |
0.0052 |
21.5s |
33.3s |
Methodology
- Run: v3-e2e (completed), created 2026-11-19 10:59:06
- Dataset: evals/datasets/v2, split test, 4 tasks, hash a20ac5229b42b8bb
- Arm opus: claude-code opus, model claude-opus
- Arm opus+fusion: claude-code opus + fusion, model claude-opus
- Arm sonnet: claude-code sonnet, model claude-sonnet
- Arm sonnet+fusion: claude-code sonnet + fusion, model claude-sonnet
- Arm haiku: claude-code haiku, model claude-haiku
- Arm haiku+fusion: claude-code haiku + fusion, model claude-haiku
- Model claude-opus: anthropic claude-opus-5-5: $4 in / $20 out per 1M tokens, verified 2026-11-17
- Model claude-sonnet: anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
- Model claude-haiku: anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Judge models: none (deterministic scorers only)
- Repeats / seed: 1 per task / seed 0
- Intervals: 95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
- Verdict rules: non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline sonnet
- Spend: $1.10 on the arms + $0 on scoring and judging (cap $4.00)
- Fusion version: 0.2.0 (git 5abab47, with uncommitted changes)
Spend
| Run |
Live spend |
| bench-e2e |
$1.20 |
| calibration |
$0.1968 |
| task-4 |
$0.0000 |
| tune |
$0.0005 |
| tune-20261118-183817-01-tune-cc96c980 |
$0.2920 |
| tune-20261118-183817-02-tune-6ffc6f08 |
$0.0727 |
| tune-20261118-183817-03-tune-237ca193 |
$0.0045 |
| v3-ablation |
$2.08 |
| v3-headline |
$4.42 |
| v3-latency |
$1.72 |
| v3-route |
$1.58 |
| v3-route-plus |
$0.1140 |
| Total |
$11.68 of the $15 cap |
Limitations
- Most tasks are synthetic, and no person has reviewed them yet. An LLM drafted the text,
review and debugging tasks; 34 coding tasks are real upstream bugs of permissively licensed
projects, with prompts written for the study. See
evals/datasets/README.md.
They test whether an answer finds the seeded defect, names the root cause or passes the hidden
tests, not how an answer feels in a real project.
- It is a small study. The $15 budget caps it at a few dozen tasks with one repeat, and the
executable categories (coding, frontend, performance) at two or three tasks each in the
headline, so the intervals are wide and many verdicts are inconclusive. That is a result: the
study could not tell, which is not the same as the arms being equal.
- The scorers decide the numbers. Review, debugging and rubric scores come from keyword and
structure matching, with neutral LLM judges (from vendors on no arm) for what those cannot
settle; a judge may still favour a style. Frontend and performance scores are measured (tests,
timings, accessibility), not judged by a model, and they check structure and speed, not intent
or taste; browser checks that need Playwright were skipped where it was not installed, the same
for every arm.
- Solo arms are not always one model. With hedging on (the shipped default), a solo arm whose
model is slow also asks a same-tier model and keeps the first answer; the arms' warnings and
call ledgers say when that happened.
- Prices and speeds are those of the day. Costs use the catalog's prices (dates in the
methodology); latencies depend on provider load at the time of the run, and a job that timed
out was run again with a longer timeout (only the last attempt is scored).
- One dataset, one set of models. A different task mix, a newer model or another judge can
change a verdict. The tuning used the
dev split and this report is from the held-out test
split, but both come from the same author and generator.
- The end-to-end runs are few. The Claude Code A/B covers four coding tasks per arm: enough
to see costs, turns and tool calls inside a real session, not to rank the arms.
Reproduce
uv run fusion bench report v3-headline --format html
uv run fusion bench results v3-headline --also v3-ablation --also v3-route --also v3-latency --also v3-e2e --html docs/benchmark-report.html --svg docs/assets/benchmark-cost-quality.svg