Benchmark report
v3-headline · evals/datasets/v2 · 6 arms · baseline solo-opus
Headline
Each arm against the baseline on all tasks. A verdict is yes, no or inconclusive; inconclusive means the study cannot tell.
solo-haiku vs solo-opus (47 paired tasks)
cost per solved task ratio 0.01 [0.01, 0.02] (below 1 is better): the whole interval is below 1
seconds per task ratio 0.32 [0.29, 0.35] (below 1 is better): the whole interval is below 1
quality difference +0.053 [-0.052, +0.147] over 47 tasks: the interval straddles the -0.03 margin
quality difference +0.053 [-0.052, +0.147] over 47 tasks: the interval includes 0
solo-luna vs solo-opus (47 paired tasks)
cost per solved task ratio 0.01 [0.01, 0.02] (below 1 is better): the whole interval is below 1
seconds per task ratio 0.48 [0.44, 0.52] (below 1 is better): the whole interval is below 1
quality difference -0.053 [-0.144, +0.047] over 47 tasks: the interval straddles the -0.03 margin
quality difference -0.053 [-0.144, +0.047] over 47 tasks: the interval includes 0
solo-sol vs solo-opus (47 paired tasks)
cost per solved task ratio 0.37 [0.25, 0.53] (below 1 is better): the whole interval is below 1
seconds per task ratio 0.96 [0.89, 1.03] (below 1 is better): the interval includes 1
quality difference -0.112 [-0.191, -0.034] over 47 tasks: the upper bound is below -0.03: it loses more than the margin
quality difference -0.112 [-0.191, -0.034] over 47 tasks: the interval is at or below 0
fusion-default vs solo-opus (47 paired tasks)
cost per solved task ratio 0.11 [0.05, 0.23] (below 1 is better): the whole interval is below 1
seconds per task ratio 0.69 [0.59, 0.78] (below 1 is better): the whole interval is below 1
quality difference +0.022 [-0.072, +0.117] over 47 tasks: the interval straddles the -0.03 margin
quality difference +0.022 [-0.072, +0.117] over 47 tasks: the interval includes 0
fusion-best vs solo-opus (47 paired tasks)
cost per solved task ratio 0.06 [0.04, 0.09] (below 1 is better): the whole interval is below 1
seconds per task ratio 0.50 [0.47, 0.53] (below 1 is better): the whole interval is below 1
quality difference +0.009 [-0.079, +0.098] over 47 tasks: the interval straddles the -0.03 margin
quality difference +0.009 [-0.079, +0.098] over 47 tasks: the interval includes 0
Arms
Quality and cost
Quality and speed
Quality by category
Where the time goes
One run per arm (the one with the median latency): each bar is a model call. Calls that overlap ran at the same time; a gap is waiting.
| Arm | Tasks | Quality (95% CI) | Solved | $/task | $/solved | p50 | p90 | Out tok/s | Tokens/solved | Quality/$ | Quality/min | Eval $/item | Eval s/item | Frontier |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| solo-haiku | 47 | 0.643 [0.548, 0.731] | 57% | $0.0009 | $0.0016 | 6.7s | 8.8s | 208 | 6,805 | 707.7 | 5.834 | $0.0014 | 4.3s | cost frontier speed frontier |
| solo-luna | 47 | 0.537 [0.432, 0.648] | 49% | $0.0007 | $0.0014 | 9.7s | 14.0s | 103 | 5,052 | 803.5 | 3.342 | $0.0013 | 4.2s | cost frontier |
| solo-opus | 47 | 0.590 [0.500, 0.679] | 43% | $0.0544 | $0.1279 | 20.7s | 29.9s | 110 | 11,327 | 10.8 | 1.690 | $0.0016 | 4.7s | - |
| solo-sol | 47 | 0.478 [0.380, 0.580] | 40% | $0.0191 | $0.0472 | 20.7s | 28.9s | 81 | 7,420 | 25.1 | 1.430 | $0.0015 | 4.0s | - |
| fusion-default | 47 | 0.612 [0.519, 0.702] | 53% | $0.0073 | $0.0137 | 13.2s | 23.6s | 207 | 17,310 | 83.8 | 2.580 | $0.0013 | 3.3s | - |
| fusion-best | 47 | 0.599 [0.492, 0.705] | 53% | $0.0042 | $0.0078 | 11.2s | 13.7s | 237 | 17,434 | 144.1 | 3.202 | $0.0013 | 3.2s | - |
Comparisons by category
| Scope | Tasks | Δ quality (95% CI) | W/T/L | Sign test p | Effect d | $/solved ratio | Seconds ratio | Cheaper | Faster | Not worse | Better |
|---|---|---|---|---|---|---|---|---|---|---|---|
| all | 47 | +0.053 [-0.052, +0.147] | 23/12/12 | 0.090 | +0.16 | 0.01 [0.01, 0.02] | 0.32 [0.29, 0.35] | ✓ yes | ✓ yes | ? inconclusive | ? inconclusive |
| architecture | 10 | +0.130 [-0.050, +0.310] | 7/0/3 | 0.344 | +0.41 | 0.00 | 0.34 [0.31, 0.36] | ? inconclusive | ✓ yes | ? inconclusive | ? inconclusive |
| code_review | 10 | +0.179 [+0.060, +0.313] | 8/1/1 | 0.039 | +0.85 | 0.01 [0.00, 0.02] | 0.34 [0.29, 0.40] | ✓ yes | ✓ yes | ✓ yes | ✓ yes |
| coding | 3 | -0.074 [-0.222, +0.000] | 0/2/1 | 1.000 | -0.58 | 0.03 [0.02, 0.07] | 0.38 [0.25, 0.58] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| debugging | 10 | +0.000 [-0.084, +0.084] | 2/6/2 | 1.000 | +0.00 | 0.02 [0.01, 0.02] | 0.32 [0.28, 0.38] | ✓ yes | ✓ yes | ? inconclusive | ? inconclusive |
| frontend | 2 | -0.464 [-0.929, +0.000] | 0/1/1 | 1.000 | -0.71 | - | 0.27 [0.25, 0.29] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| performance | 2 | +0.016 [+0.004, +0.029] | 1/1/0 | 1.000 | +0.91 | 0.01 [0.01, 0.03] | 0.26 [0.11, 0.58] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| planning | 10 | +0.050 [-0.261, +0.353] | 5/1/4 | 1.000 | +0.10 | 0.01 [0.00, 0.03] | 0.29 [0.26, 0.33] | ✓ yes | ✓ yes | ? inconclusive | ? inconclusive |
| Scope | Tasks | Δ quality (95% CI) | W/T/L | Sign test p | Effect d | $/solved ratio | Seconds ratio | Cheaper | Faster | Not worse | Better |
|---|---|---|---|---|---|---|---|---|---|---|---|
| all | 47 | -0.053 [-0.144, +0.047] | 12/14/21 | 0.163 | -0.16 | 0.01 [0.01, 0.02] | 0.48 [0.44, 0.52] | ✓ yes | ✓ yes | ? inconclusive | ? inconclusive |
| architecture | 10 | -0.123 [-0.250, -0.017] | 1/3/6 | 0.125 | -0.65 | 0.01 | 0.45 [0.40, 0.52] | ? inconclusive | ✓ yes | ? inconclusive | ✗ no |
| code_review | 10 | +0.271 [+0.139, +0.404] | 7/3/0 | 0.016 | +1.19 | 0.00 [0.00, 0.01] | 0.39 [0.35, 0.44] | ✓ yes | ✓ yes | ✓ yes | ✓ yes |
| coding | 3 | +0.000 [+0.000, +0.000] | 0/3/0 | - | - | 0.01 [0.01, 0.01] | 0.40 [0.36, 0.44] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| debugging | 10 | -0.060 [-0.286, +0.191] | 2/2/6 | 0.289 | -0.15 | 0.01 [0.01, 0.02] | 0.60 [0.52, 0.71] | ✓ yes | ✓ yes | ? inconclusive | ? inconclusive |
| frontend | 2 | -0.464 [-0.929, +0.000] | 0/1/1 | 1.000 | -0.71 | - | - | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| performance | 2 | -0.005 [-0.007, -0.004] | 0/2/0 | - | -2.12 | 0.01 [0.01, 0.01] | 0.32 [0.27, 0.45] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| planning | 10 | -0.240 [-0.393, -0.082] | 2/0/8 | 0.109 | -0.88 | - | 0.50 [0.42, 0.57] | ? inconclusive | ✓ yes | ✗ no | ✗ no |
| Scope | Tasks | Δ quality (95% CI) | W/T/L | Sign test p | Effect d | $/solved ratio | Seconds ratio | Cheaper | Faster | Not worse | Better |
|---|---|---|---|---|---|---|---|---|---|---|---|
| all | 47 | -0.112 [-0.191, -0.034] | 8/11/28 | 0.001 | -0.41 | 0.37 [0.25, 0.53] | 0.96 [0.89, 1.03] | ✓ yes | ? inconclusive | ✗ no | ✗ no |
| architecture | 10 | -0.160 [-0.307, +0.053] | 1/0/9 | 0.021 | -0.52 | 0.40 | 1.04 [0.93, 1.14] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| code_review | 10 | -0.136 [-0.263, -0.017] | 2/1/7 | 0.180 | -0.65 | 1.08 | 0.93 [0.79, 1.07] | ? inconclusive | ? inconclusive | ? inconclusive | ✗ no |
| coding | 3 | +0.000 [+0.000, +0.000] | 0/3/0 | - | - | 0.30 [0.30, 0.32] | 1.03 [0.88, 1.21] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| debugging | 10 | -0.046 [-0.171, +0.095] | 2/3/5 | 0.453 | -0.21 | 0.19 [0.11, 0.26] | 0.66 [0.58, 0.77] | ✓ yes | ✓ yes | ? inconclusive | ? inconclusive |
| frontend | 2 | -0.005 [-0.010, +0.000] | 0/2/0 | - | -0.71 | 0.32 | 0.91 [0.79, 1.04] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| performance | 2 | -0.034 [-0.046, -0.021] | 0/0/2 | 0.500 | -1.92 | 0.45 [0.42, 0.49] | 1.17 [1.00, 1.53] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| planning | 10 | -0.178 [-0.433, +0.046] | 3/2/5 | 0.727 | -0.43 | 0.59 | 1.08 [0.96, 1.20] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| Scope | Tasks | Δ quality (95% CI) | W/T/L | Sign test p | Effect d | $/solved ratio | Seconds ratio | Cheaper | Faster | Not worse | Better |
|---|---|---|---|---|---|---|---|---|---|---|---|
| all | 47 | +0.022 [-0.072, +0.117] | 18/18/11 | 0.265 | +0.07 | 0.11 [0.05, 0.23] | 0.69 [0.59, 0.78] | ✓ yes | ✓ yes | ? inconclusive | ? inconclusive |
| architecture | 10 | +0.217 [+0.057, +0.373] | 8/1/1 | 0.039 | +0.80 | 0.02 | 0.80 [0.66, 0.93] | ? inconclusive | ✓ yes | ✓ yes | ✓ yes |
| code_review | 10 | -0.019 [-0.138, +0.130] | 3/3/4 | 1.000 | -0.08 | 0.11 | 0.48 [0.35, 0.67] | ? inconclusive | ✓ yes | ? inconclusive | ? inconclusive |
| coding | 3 | +0.000 [+0.000, +0.000] | 0/3/0 | - | - | 0.10 [0.01, 0.14] | 0.64 [0.27, 0.91] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| debugging | 10 | +0.056 [-0.042, +0.175] | 2/7/1 | 1.000 | +0.29 | 0.04 [0.02, 0.07] | 0.42 [0.32, 0.54] | ✓ yes | ✓ yes | ? inconclusive | ? inconclusive |
| frontend | 2 | -0.464 [-0.929, +0.000] | 0/1/1 | 1.000 | -0.71 | - | 0.96 | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| performance | 2 | +0.007 [+0.004, +0.011] | 0/2/0 | - | +1.41 | 0.06 [0.01, 0.08] | 0.48 [0.23, 0.59] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| planning | 10 | -0.057 [-0.326, +0.214] | 5/1/4 | 1.000 | -0.12 | 0.08 [0.02, 0.23] | 0.92 [0.75, 1.13] | ✓ yes | ? inconclusive | ? inconclusive | ? inconclusive |
| Scope | Tasks | Δ quality (95% CI) | W/T/L | Sign test p | Effect d | $/solved ratio | Seconds ratio | Cheaper | Faster | Not worse | Better |
|---|---|---|---|---|---|---|---|---|---|---|---|
| all | 47 | +0.009 [-0.079, +0.098] | 17/14/16 | 1.000 | +0.03 | 0.06 [0.04, 0.09] | 0.50 [0.47, 0.53] | ✓ yes | ✓ yes | ? inconclusive | ? inconclusive |
| architecture | 10 | -0.110 [-0.290, +0.077] | 3/2/5 | 0.727 | -0.35 | 0.07 | 0.48 [0.44, 0.52] | ? inconclusive | ✓ yes | ? inconclusive | ? inconclusive |
| code_review | 10 | +0.235 [+0.129, +0.338] | 8/1/1 | 0.039 | +1.24 | 0.02 [0.00, 0.04] | 0.51 [0.45, 0.60] | ✓ yes | ✓ yes | ✓ yes | ✓ yes |
| coding | 3 | +0.000 [+0.000, +0.000] | 0/3/0 | - | - | 0.08 [0.03, 0.12] | 0.58 [0.52, 0.66] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| debugging | 10 | -0.030 [-0.138, +0.069] | 2/5/3 | 1.000 | -0.17 | 0.11 [0.09, 0.12] | 0.55 [0.41, 0.81] | ✓ yes | ✓ yes | ? inconclusive | ? inconclusive |
| frontend | 2 | +0.524 [+0.062, +0.987] | 2/0/0 | 0.500 | +0.80 | 0.02 | 0.54 [0.53, 0.55] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| performance | 2 | -0.020 [-0.032, -0.007] | 0/1/1 | 1.000 | -1.11 | 0.03 [0.03, 0.05] | 0.41 [0.33, 0.61] | ? inconclusive | ? inconclusive | ? inconclusive | ? inconclusive |
| planning | 10 | -0.155 [-0.376, +0.071] | 2/2/6 | 0.289 | -0.41 | 0.28 | 0.49 [0.45, 0.54] | ? inconclusive | ✓ yes | ? inconclusive | ? inconclusive |
| Difficulty | Arm | Tasks | Quality (95% CI) | $/solved |
|---|---|---|---|---|
| easy | solo-haiku | 13 | 0.709 [0.544, 0.849] | $0.0015 |
| easy | solo-luna | 13 | 0.558 [0.376, 0.747] | $0.0010 |
| easy | solo-opus | 13 | 0.637 [0.474, 0.797] | $0.0740 |
| easy | solo-sol | 13 | 0.502 [0.309, 0.700] | $0.0409 |
| easy | fusion-default | 13 | 0.755 [0.616, 0.885] | $0.0055 |
| easy | fusion-best | 13 | 0.709 [0.546, 0.868] | $0.0061 |
| hard | solo-haiku | 15 | 0.613 [0.426, 0.782] | $0.0016 |
| hard | solo-luna | 15 | 0.561 [0.362, 0.757] | $0.0015 |
| hard | solo-opus | 15 | 0.490 [0.321, 0.656] | $0.2108 |
| hard | solo-sol | 15 | 0.510 [0.361, 0.645] | $0.0421 |
| hard | fusion-default | 15 | 0.523 [0.359, 0.685] | $0.0137 |
| hard | fusion-best | 15 | 0.571 [0.384, 0.753] | $0.0086 |
| medium | solo-haiku | 19 | 0.621 [0.458, 0.773] | $0.0016 |
| medium | solo-luna | 19 | 0.505 [0.335, 0.672] | $0.0016 |
| medium | solo-opus | 19 | 0.637 [0.516, 0.772] | $0.1403 |
| medium | solo-sol | 19 | 0.436 [0.266, 0.610] | $0.0567 |
| medium | fusion-default | 19 | 0.585 [0.431, 0.740] | $0.0197 |
| medium | fusion-best | 19 | 0.545 [0.369, 0.708] | $0.0086 |
| Arm | Mean sd across repeats | Tasks whose solved flips |
|---|---|---|
| solo-haiku | - | - |
| solo-luna | - | - |
| solo-opus | - | - |
| solo-sol | - | - |
| fusion-default | - | - |
| fusion-best | - | - |
Judge reliability
Headline verdict allowed. Accuracy floor 80%: mimo-pro 100%, muse-ember 88%.
Most different answers, with evidence
The tasks with evidence where an arm and the baseline differed most in completion.
fe-data-table (frontend): completion differs by +0.99
fusion-best (repeat 1): completion 0.99
- gate tests-pass: ✓ pass
- gate works-offline: ✓ pass external_requests = 0, needs <= 0
- gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E1 tests (measured): all 7 hidden tests pass
- E2 build (measured): everything compiles
- E3 static (measured): clean
- E4 diff_stats (measured): +171 -2 lines in 3 files
- E5 a11y (measured): no violations (static source rules only)
- E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
Judge: All 7 hidden tests pass (E1) and static/build/a11y checks are clean (E2, E3, E5). index.html has lang, viewport meta, a title, a single h1, a <caption>, column headers with scope=col, row headers with scope=row, and a button in each column header with data-sort (lines 14-30). app.js updates aria-sort on the active column (and removes it elsewhere) and toggles ascending/descending per column, using numeric comparison for tickets and localeCompare for text (lines 12-31). The wrapper .table-scroll has tabindex=0 + role=region + aria-label so it is keyboard-reachable, and overflow-x:auto with table min-width:36rem means the table scrolls inside its own container on narrow screens while body max-width keeps the page from breaking (style.css lines 28-47). Design matches the brief: bordered/tinted header row (#e8eef6), zebra rows (nth-child(even)), borderless text-looking header buttons, caption on top. Focus-visible outlines are defined for both the wrapper and buttons (style.css lines 36-40). E3 shows no external requests, viewport meta present, no lint issues; E4 shows a compact 171-line addition across 3 files. Minor nits: numeric detection is hardcoded to the 'tickets' key rather tha
Measuring this cost $0.0050 and 22.2s: not part of the arm's cost or time.
solo-opus (repeat 1): completion 0.00
- gate tests-pass: ✗ FAIL 6/7 hidden tests pass; failing: HiddenTests.test_exactly_one_column_reports_its_sort
- gate works-offline: ✓ pass external_requests = 0, needs <= 0
- gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E1 tests (measured): 6/7 hidden tests pass; failing: HiddenTests.test_exactly_one_column_reports_its_sort
- E2 build (measured): everything compiles
- E3 static (measured): clean
- E4 diff_stats (measured): +178 -2 lines in 3 files
- E5 a11y (measured): no violations (static source rules only)
- E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
Judge: Output A builds a real table with a caption (#people-caption), column headers with sorting buttons, row headers (scope="row" for each name), and a keyboard-reachable scroll wrapper (role=region, tabindex=0, overflow-x:auto with min-width:36rem so it scrolls on a 390px phone) — E3 static PASS, E5 a11y PASS. Sorting works for text and numeric columns via data-type, and the sorted column is toggled to aria-sort with a live status line. However, the hidden suite fails: E1 shows 6/7 with HiddenTests.test_exactly_one_column_reports_its_sort failing consistently across 3 runs, and the tests-pass hard gate is FAIL. This directly undercuts the core sort-reporting requirement even though the JS attempts to reset other headers to "none" (app.js lines 31-40), so correctness and ux_polish are reduced. The tinted header (#eef2f8), bordered wrapper, zebra-capable rows (only border-top given; no alternating row tint is actually applied — a design_fidelity gap), text-like header buttons, caption on top, visible :focus-visible outline, and viewport meta/lang/title all satisfy the design and a11y expectations (E2 build PASS, E4 diff_stats PASS), and everything is local/offline (works-offline pass). C
Measuring this cost $0.0056 and 53.9s: not part of the arm's cost or time.
fe-login-dark (frontend): completion differs by -0.93
fusion-default (repeat 1): completion 0.00
- gate tests-pass: ✗ FAIL 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
- gate works-offline: ✓ pass external_requests = 0, needs <= 0
- gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E1 tests (measured): 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
- E2 build (measured): everything compiles
- E3 static (measured): clean
- E4 diff_stats (measured): +156 -2 lines in 3 files
- E5 a11y (measured): no violations (static source rules only)
- E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
Judge: The output delivers the requested structure: index.html has lang, title, viewport meta, one h1, a labelled Username and Password field, a Show password checkbox with an 8-line app.js toggle (app.js lines 1-8), a full-width Sign in button and a quiet Forgot your password? link (E3, E5). Accessibility is strong: E5 reports zero contrast failures and a 3px yellow :focus-visible outline (style.css line 118-121); autocomplete attributes are present. Design matches the brief: centred dark card (max-width 24rem, .page flex-centred), full-width primary button and underlined secondary link (E3 max_fixed_width_px=0). However, the hidden test suite fails 6/7, specifically HiddenTests.test_fits_small_screens (E1), so the responsiveness criterion cannot be scored highly; the remaining build/static/a11y checks pass (E2, E3, E5) and the file diff is minimal and clean (E4: +156 -2 over 3 files). | E1 shows 6/7 hidden tests pass but test_fits_small_screens fails, so responsiveness is weak and tests hard gate fails despite E2 build PASS, E3 static PASS and E5 a11y PASS indicating correct structure, labels and contrast. E6 screenshot n/a and E7 console n/a limit visual verification, so design and pol
Measuring this cost $0.0046 and 48.9s: not part of the arm's cost or time.
solo-opus (repeat 1): completion 0.93
- gate tests-pass: ✓ pass
- gate works-offline: ✓ pass external_requests = 0, needs <= 0
- gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E1 tests (measured): all 7 hidden tests pass
- E2 build (measured): everything compiles
- E3 static (measured): clean
- E4 diff_stats (measured): +170 -2 lines in 3 files
- E5 a11y (measured): no violations (static source rules only)
- E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
Judge: Output A implements the brief cleanly: index.html (E1/E3) has lang, title, viewport meta, a single h1 'Sign in', a semantic <main>/<form> with properly associated <label for> on both fields and the 'Show password' checkbox, required inputs with autocomplete='username'/'current-password', a role="status" live region, and a 'Forgot your password?' link. style.css centres a 24rem dark card (flexbox + width:min(100%,24rem) and 1rem body padding) so it works from 390px to 1440px, uses #f1f5f9/#cbd5e1/#93c5fd text on #0f172a/#1e293b backgrounds and white on #1d4ed8 — all well above 4.5:1, confirmed by E5 (contrast_failures=0, 0 violations). Focus is a 3px #fbbf24 outline with offset, applied to :focus-visible plus a :focus fallback. app.js is 22 minimal lines implementing the show-password toggle by swapping input.type and a form-submit handler with a polite status message. All assets are local (E2 build PASS, hard gate works-offline PASS), and all 7 hidden tests pass (E1) with the static checks (E3) and diff stats (E4) passing. No images, so alt-text is not applicable. Minor nits: the submit handler always preventDefaults, and the card could use a tiny max-width guard at very narrow wid
Measuring this cost $0.0080 and 35.8s: not part of the arm's cost or time.
fe-login-dark (frontend): completion differs by -0.93
solo-haiku (repeat 1): completion 0.00
- gate tests-pass: ✗ FAIL 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
- gate works-offline: ✓ pass external_requests = 0, needs <= 0
- gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E1 tests (measured): 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
- E2 build (measured): everything compiles
- E3 static (measured): clean
- E4 diff_stats (measured): +44 -2 lines in 1 files
- E5 a11y (measured): no violations (static source rules only)
- E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
Judge: Output A delivers a single self-contained index.html with lang, viewport meta, title, one h1, labelled Username/Password inputs with autocomplete, a working Show password checkbox toggle (inline script lines 42-50), a full-width Sign in button and a quiet 'Forgot your password?' link, all centred in a dark card (E3, E5: 0 contrast failures, 0 lint issues, 0 external requests). However, E1 reports 6/7 hidden tests passing with HiddenTests.test_fits_small_screens failing, so the responsiveness requirement from 390px phone to 1440px desktop is not fully verified despite the CSS using width:100%, max-width:360px and flex centering; this also trips the hard gate tests-pass (FAIL, E1). Everything else builds and passes static/a11y checks (E2, E3, E5). | File shows centred dark card with h1, labelled Username/Password, Show password checkbox toggling type, full-width Sign in button, Forgot link, bright #ffd54a focus-visible ring and high-contrast colours, matching a11y PASS E5 contrast_failures=0. However hard test gate fails: E1 tests FAIL 6/7 with HiddenTests.test_fits_small_screens failing, so responsiveness penalised despite viewport meta and max-width layout; build/static/diff PASS p
Measuring this cost $0.0056 and 66.6s: not part of the arm's cost or time.
solo-opus (repeat 1): completion 0.93
- gate tests-pass: ✓ pass
- gate works-offline: ✓ pass external_requests = 0, needs <= 0
- gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E1 tests (measured): all 7 hidden tests pass
- E2 build (measured): everything compiles
- E3 static (measured): clean
- E4 diff_stats (measured): +170 -2 lines in 3 files
- E5 a11y (measured): no violations (static source rules only)
- E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
- E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
Judge: Output A implements the brief cleanly: index.html (E1/E3) has lang, title, viewport meta, a single h1 'Sign in', a semantic <main>/<form> with properly associated <label for> on both fields and the 'Show password' checkbox, required inputs with autocomplete='username'/'current-password', a role="status" live region, and a 'Forgot your password?' link. style.css centres a 24rem dark card (flexbox + width:min(100%,24rem) and 1rem body padding) so it works from 390px to 1440px, uses #f1f5f9/#cbd5e1/#93c5fd text on #0f172a/#1e293b backgrounds and white on #1d4ed8 — all well above 4.5:1, confirmed by E5 (contrast_failures=0, 0 violations). Focus is a 3px #fbbf24 outline with offset, applied to :focus-visible plus a :focus fallback. app.js is 22 minimal lines implementing the show-password toggle by swapping input.type and a form-submit handler with a polite status message. All assets are local (E2 build PASS, hard gate works-offline PASS), and all 7 hidden tests pass (E1) with the static checks (E3) and diff stats (E4) passing. No images, so alt-text is not applicable. Minor nits: the submit handler always preventDefaults, and the card could use a tiny max-width guard at very narrow wid
Measuring this cost $0.0080 and 35.8s: not part of the arm's cost or time.
Methodology
- Run
- v3-headline (completed), created 2026-11-19 10:42:16
- Dataset
- evals/datasets/v2, split test, 47 tasks, hash 84c35e7998b364bd
- Arm solo-opus
- strategy solo-opus: solo, models claude-opus, aggregator auto, rounds 1, judge off
- Arm solo-sol
- strategy solo-sol: solo, models gpt-sol, aggregator auto, rounds 1, judge off
- Arm solo-haiku
- strategy solo-haiku: solo, models claude-haiku, aggregator auto, rounds 1, judge off
- Arm solo-luna
- strategy solo-luna: solo, models gpt-luna, aggregator auto, rounds 1, judge off
- Arm fusion-default
- strategy auto: auto, models claude-haiku, gpt-luna, gemini-flash, gpt-terra, gemini-hydrogen, claude-sonnet, aggregator auto, rounds 1, judge off
- Arm fusion-best
- strategy panel-nano-select, overrides {'aggregator': 'select'}: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator select, rounds 1, judge off
- Model claude-opus
- anthropic claude-opus-5-5: $4 in / $20 out per 1M tokens, verified 2026-11-17
- Model gpt-sol
- openai gpt-6.2-sol: $2 in / $10 out per 1M tokens, verified 2026-11-17
- Model claude-haiku
- anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Model gpt-luna
- openai gpt-6-luna: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Model gemini-flash
- google gemini-3.8-flash: $0.75 in / $3.75 out per 1M tokens, verified 2026-11-17
- Model gpt-terra
- openai gpt-6.1-terra: $1 in / $5 out per 1M tokens, verified 2026-11-17
- Model gemini-hydrogen
- google gemini-4-hydrogen: $1.5 in / $7.5 out per 1M tokens, verified 2026-11-17
- Model claude-sonnet
- anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
- Model mimo-pro
- openrouter xiaomi/mimo-v2.6-pro: $0.435 in / $0.87 out per 1M tokens, verified 2026-11-17
- Model muse-ember
- openrouter meta/muse-ember: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
- Judge models
- mimo-pro, muse-ember
- Repeats / seed
- 1 per task / seed 100
- Intervals
- 95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
- Verdict rules
- non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline solo-opus
- Spend
- $4.03 on the arms + $0.3940 on scoring and judging (cap $6.00)
- Fusion version
- 0.2.0 (git 5abab47, with uncommitted changes)