Benchmark report

v3-headline · evals/datasets/v2 · 6 arms · baseline solo-opus

2 job(s) halted on their first attempt (a call timed out or the panel had no quorum) and were run again, with longer timeouts where it was a timeout: solo-opus (2). Only the last attempt is scored; the first attempt's cost is in the totals. A halted job scores 0, so without the retry these arms would be understated by the harness's limit, not by their answers.
Some prices were unknown for solo-luna, solo-sol, fusion-default, fusion-best: its costs are a lower bound.
Items replayed from the response cache are left out of the latency figures (solo-luna, fusion-default, fusion-best): their wall time is not a measurement.
1 repeat(s) per task: variance across repeats is poorly estimated.
Frontend and performance scores measure what the answer builds (tests, timings, screenshots); they do not judge whether it matches the author's intent.

Headline

Each arm against the baseline on all tasks. A verdict is yes, no or inconclusive; inconclusive means the study cannot tell.

solo-haiku vs solo-opus (47 paired tasks)

Cheaper (per solved task): yes

cost per solved task ratio 0.01 [0.01, 0.02] (below 1 is better): the whole interval is below 1

Faster: yes

seconds per task ratio 0.32 [0.29, 0.35] (below 1 is better): the whole interval is below 1

Not worse: inconclusive

quality difference +0.053 [-0.052, +0.147] over 47 tasks: the interval straddles the -0.03 margin

Better: inconclusive

quality difference +0.053 [-0.052, +0.147] over 47 tasks: the interval includes 0

solo-luna vs solo-opus (47 paired tasks)

Cheaper (per solved task): yes

cost per solved task ratio 0.01 [0.01, 0.02] (below 1 is better): the whole interval is below 1

Faster: yes

seconds per task ratio 0.48 [0.44, 0.52] (below 1 is better): the whole interval is below 1

Not worse: inconclusive

quality difference -0.053 [-0.144, +0.047] over 47 tasks: the interval straddles the -0.03 margin

Better: inconclusive

quality difference -0.053 [-0.144, +0.047] over 47 tasks: the interval includes 0

solo-sol vs solo-opus (47 paired tasks)

Cheaper (per solved task): yes

cost per solved task ratio 0.37 [0.25, 0.53] (below 1 is better): the whole interval is below 1

Faster: inconclusive

seconds per task ratio 0.96 [0.89, 1.03] (below 1 is better): the interval includes 1

Not worse: no

quality difference -0.112 [-0.191, -0.034] over 47 tasks: the upper bound is below -0.03: it loses more than the margin

Better: no

quality difference -0.112 [-0.191, -0.034] over 47 tasks: the interval is at or below 0

fusion-default vs solo-opus (47 paired tasks)

Cheaper (per solved task): yes

cost per solved task ratio 0.11 [0.05, 0.23] (below 1 is better): the whole interval is below 1

Faster: yes

seconds per task ratio 0.69 [0.59, 0.78] (below 1 is better): the whole interval is below 1

Not worse: inconclusive

quality difference +0.022 [-0.072, +0.117] over 47 tasks: the interval straddles the -0.03 margin

Better: inconclusive

quality difference +0.022 [-0.072, +0.117] over 47 tasks: the interval includes 0

fusion-best vs solo-opus (47 paired tasks)

Cheaper (per solved task): yes

cost per solved task ratio 0.06 [0.04, 0.09] (below 1 is better): the whole interval is below 1

Faster: yes

seconds per task ratio 0.50 [0.47, 0.53] (below 1 is better): the whole interval is below 1

Not worse: inconclusive

quality difference +0.009 [-0.079, +0.098] over 47 tasks: the interval straddles the -0.03 margin

Better: inconclusive

quality difference +0.009 [-0.079, +0.098] over 47 tasks: the interval includes 0

Arms

solo-haikusolo-lunasolo-opussolo-solfusion-defaultfusion-best

Quality and cost

Quality against cost per taskOne point per arm. Up and to the left is better. The line joins the arms no other arm beats.0.40.50.60.7$0$0.02$0.04$0.06Cost per task (USD); bars are 95% intervalsMean qualitysolo-haikusolo-lunasolo-opussolo-solfusion-defaultfusion-best

Quality and speed

Quality against median latencyOne point per arm. Up and to the left is better. The line joins the arms no other arm beats.0.40.50.60.70s10s20s30sMedian seconds to complete (bar reaches p90)Mean qualitysolo-haikusolo-lunasolo-opussolo-solfusion-defaultfusion-best

Quality by category

solo-haikusolo-lunasolo-opussolo-solfusion-defaultfusion-best
Mean quality by categoryA bar per arm in each category, from 0 to 1, with a whisker for the 95% interval.architecture0.500.240.370.210.580.26code_review0.730.820.550.420.530.79coding0.931.001.001.001.001.00debugging0.810.750.810.770.870.78frontend0.000.000.460.460.000.99performance1.000.970.980.950.990.96planning0.500.210.450.280.400.30

Where the time goes

One run per arm (the one with the median latency): each bar is a model call. Calls that overlap ran at the same time; a gap is waiting.

Stage:panelsynthesis
Call timeline of a typical run per armEach call of the run with the median latency, on that run's own clock.0s5s10s15s20ssolo-haiku (task cr-ts-05)solo-luna (task plan-18)solo-opus (task arch-05)solo-sol (task plan-13)fusion-default (task real-dacite-optional-union)fusion-best (task plan-22)
Every arm, all tasks. Eval columns are what scoring and judging cost: shown beside, never inside, an arm's cost and time.
ArmTasksQuality (95% CI)Solved$/task$/solvedp50p90Out tok/sTokens/solvedQuality/$Quality/minEval $/itemEval s/itemFrontier
solo-haiku470.643 [0.548, 0.731]57%$0.0009$0.00166.7s8.8s2086,805707.75.834$0.00144.3scost frontier speed frontier
solo-luna470.537 [0.432, 0.648]49%$0.0007$0.00149.7s14.0s1035,052803.53.342$0.00134.2scost frontier
solo-opus470.590 [0.500, 0.679]43%$0.0544$0.127920.7s29.9s11011,32710.81.690$0.00164.7s-
solo-sol470.478 [0.380, 0.580]40%$0.0191$0.047220.7s28.9s817,42025.11.430$0.00154.0s-
fusion-default470.612 [0.519, 0.702]53%$0.0073$0.013713.2s23.6s20717,31083.82.580$0.00133.3s-
fusion-best470.599 [0.492, 0.705]53%$0.0042$0.007811.2s13.7s23717,434144.13.202$0.00133.2s-

Comparisons by category

solo-haiku vs solo-opus (paired by task; hover a verdict for its reason)
ScopeTasksΔ quality (95% CI)W/T/LSign test pEffect d$/solved ratioSeconds ratioCheaperFasterNot worseBetter
all47+0.053 [-0.052, +0.147]23/12/120.090+0.160.01 [0.01, 0.02]0.32 [0.29, 0.35]✓ yes✓ yes? inconclusive? inconclusive
architecture10+0.130 [-0.050, +0.310]7/0/30.344+0.410.000.34 [0.31, 0.36]? inconclusive✓ yes? inconclusive? inconclusive
code_review10+0.179 [+0.060, +0.313]8/1/10.039+0.850.01 [0.00, 0.02]0.34 [0.29, 0.40]✓ yes✓ yes✓ yes✓ yes
coding3-0.074 [-0.222, +0.000]0/2/11.000-0.580.03 [0.02, 0.07]0.38 [0.25, 0.58]? inconclusive? inconclusive? inconclusive? inconclusive
debugging10+0.000 [-0.084, +0.084]2/6/21.000+0.000.02 [0.01, 0.02]0.32 [0.28, 0.38]✓ yes✓ yes? inconclusive? inconclusive
frontend2-0.464 [-0.929, +0.000]0/1/11.000-0.71-0.27 [0.25, 0.29]? inconclusive? inconclusive? inconclusive? inconclusive
performance2+0.016 [+0.004, +0.029]1/1/01.000+0.910.01 [0.01, 0.03]0.26 [0.11, 0.58]? inconclusive? inconclusive? inconclusive? inconclusive
planning10+0.050 [-0.261, +0.353]5/1/41.000+0.100.01 [0.00, 0.03]0.29 [0.26, 0.33]✓ yes✓ yes? inconclusive? inconclusive
solo-luna vs solo-opus (paired by task; hover a verdict for its reason)
ScopeTasksΔ quality (95% CI)W/T/LSign test pEffect d$/solved ratioSeconds ratioCheaperFasterNot worseBetter
all47-0.053 [-0.144, +0.047]12/14/210.163-0.160.01 [0.01, 0.02]0.48 [0.44, 0.52]✓ yes✓ yes? inconclusive? inconclusive
architecture10-0.123 [-0.250, -0.017]1/3/60.125-0.650.010.45 [0.40, 0.52]? inconclusive✓ yes? inconclusive✗ no
code_review10+0.271 [+0.139, +0.404]7/3/00.016+1.190.00 [0.00, 0.01]0.39 [0.35, 0.44]✓ yes✓ yes✓ yes✓ yes
coding3+0.000 [+0.000, +0.000]0/3/0--0.01 [0.01, 0.01]0.40 [0.36, 0.44]? inconclusive? inconclusive? inconclusive? inconclusive
debugging10-0.060 [-0.286, +0.191]2/2/60.289-0.150.01 [0.01, 0.02]0.60 [0.52, 0.71]✓ yes✓ yes? inconclusive? inconclusive
frontend2-0.464 [-0.929, +0.000]0/1/11.000-0.71--? inconclusive? inconclusive? inconclusive? inconclusive
performance2-0.005 [-0.007, -0.004]0/2/0--2.120.01 [0.01, 0.01]0.32 [0.27, 0.45]? inconclusive? inconclusive? inconclusive? inconclusive
planning10-0.240 [-0.393, -0.082]2/0/80.109-0.88-0.50 [0.42, 0.57]? inconclusive✓ yes✗ no✗ no
solo-sol vs solo-opus (paired by task; hover a verdict for its reason)
ScopeTasksΔ quality (95% CI)W/T/LSign test pEffect d$/solved ratioSeconds ratioCheaperFasterNot worseBetter
all47-0.112 [-0.191, -0.034]8/11/280.001-0.410.37 [0.25, 0.53]0.96 [0.89, 1.03]✓ yes? inconclusive✗ no✗ no
architecture10-0.160 [-0.307, +0.053]1/0/90.021-0.520.401.04 [0.93, 1.14]? inconclusive? inconclusive? inconclusive? inconclusive
code_review10-0.136 [-0.263, -0.017]2/1/70.180-0.651.080.93 [0.79, 1.07]? inconclusive? inconclusive? inconclusive✗ no
coding3+0.000 [+0.000, +0.000]0/3/0--0.30 [0.30, 0.32]1.03 [0.88, 1.21]? inconclusive? inconclusive? inconclusive? inconclusive
debugging10-0.046 [-0.171, +0.095]2/3/50.453-0.210.19 [0.11, 0.26]0.66 [0.58, 0.77]✓ yes✓ yes? inconclusive? inconclusive
frontend2-0.005 [-0.010, +0.000]0/2/0--0.710.320.91 [0.79, 1.04]? inconclusive? inconclusive? inconclusive? inconclusive
performance2-0.034 [-0.046, -0.021]0/0/20.500-1.920.45 [0.42, 0.49]1.17 [1.00, 1.53]? inconclusive? inconclusive? inconclusive? inconclusive
planning10-0.178 [-0.433, +0.046]3/2/50.727-0.430.591.08 [0.96, 1.20]? inconclusive? inconclusive? inconclusive? inconclusive
fusion-default vs solo-opus (paired by task; hover a verdict for its reason)
ScopeTasksΔ quality (95% CI)W/T/LSign test pEffect d$/solved ratioSeconds ratioCheaperFasterNot worseBetter
all47+0.022 [-0.072, +0.117]18/18/110.265+0.070.11 [0.05, 0.23]0.69 [0.59, 0.78]✓ yes✓ yes? inconclusive? inconclusive
architecture10+0.217 [+0.057, +0.373]8/1/10.039+0.800.020.80 [0.66, 0.93]? inconclusive✓ yes✓ yes✓ yes
code_review10-0.019 [-0.138, +0.130]3/3/41.000-0.080.110.48 [0.35, 0.67]? inconclusive✓ yes? inconclusive? inconclusive
coding3+0.000 [+0.000, +0.000]0/3/0--0.10 [0.01, 0.14]0.64 [0.27, 0.91]? inconclusive? inconclusive? inconclusive? inconclusive
debugging10+0.056 [-0.042, +0.175]2/7/11.000+0.290.04 [0.02, 0.07]0.42 [0.32, 0.54]✓ yes✓ yes? inconclusive? inconclusive
frontend2-0.464 [-0.929, +0.000]0/1/11.000-0.71-0.96? inconclusive? inconclusive? inconclusive? inconclusive
performance2+0.007 [+0.004, +0.011]0/2/0-+1.410.06 [0.01, 0.08]0.48 [0.23, 0.59]? inconclusive? inconclusive? inconclusive? inconclusive
planning10-0.057 [-0.326, +0.214]5/1/41.000-0.120.08 [0.02, 0.23]0.92 [0.75, 1.13]✓ yes? inconclusive? inconclusive? inconclusive
fusion-best vs solo-opus (paired by task; hover a verdict for its reason)
ScopeTasksΔ quality (95% CI)W/T/LSign test pEffect d$/solved ratioSeconds ratioCheaperFasterNot worseBetter
all47+0.009 [-0.079, +0.098]17/14/161.000+0.030.06 [0.04, 0.09]0.50 [0.47, 0.53]✓ yes✓ yes? inconclusive? inconclusive
architecture10-0.110 [-0.290, +0.077]3/2/50.727-0.350.070.48 [0.44, 0.52]? inconclusive✓ yes? inconclusive? inconclusive
code_review10+0.235 [+0.129, +0.338]8/1/10.039+1.240.02 [0.00, 0.04]0.51 [0.45, 0.60]✓ yes✓ yes✓ yes✓ yes
coding3+0.000 [+0.000, +0.000]0/3/0--0.08 [0.03, 0.12]0.58 [0.52, 0.66]? inconclusive? inconclusive? inconclusive? inconclusive
debugging10-0.030 [-0.138, +0.069]2/5/31.000-0.170.11 [0.09, 0.12]0.55 [0.41, 0.81]✓ yes✓ yes? inconclusive? inconclusive
frontend2+0.524 [+0.062, +0.987]2/0/00.500+0.800.020.54 [0.53, 0.55]? inconclusive? inconclusive? inconclusive? inconclusive
performance2-0.020 [-0.032, -0.007]0/1/11.000-1.110.03 [0.03, 0.05]0.41 [0.33, 0.61]? inconclusive? inconclusive? inconclusive? inconclusive
planning10-0.155 [-0.376, +0.071]2/2/60.289-0.410.280.49 [0.45, 0.54]? inconclusive✓ yes? inconclusive? inconclusive
By difficulty
DifficultyArmTasksQuality (95% CI)$/solved
easysolo-haiku130.709 [0.544, 0.849]$0.0015
easysolo-luna130.558 [0.376, 0.747]$0.0010
easysolo-opus130.637 [0.474, 0.797]$0.0740
easysolo-sol130.502 [0.309, 0.700]$0.0409
easyfusion-default130.755 [0.616, 0.885]$0.0055
easyfusion-best130.709 [0.546, 0.868]$0.0061
hardsolo-haiku150.613 [0.426, 0.782]$0.0016
hardsolo-luna150.561 [0.362, 0.757]$0.0015
hardsolo-opus150.490 [0.321, 0.656]$0.2108
hardsolo-sol150.510 [0.361, 0.645]$0.0421
hardfusion-default150.523 [0.359, 0.685]$0.0137
hardfusion-best150.571 [0.384, 0.753]$0.0086
mediumsolo-haiku190.621 [0.458, 0.773]$0.0016
mediumsolo-luna190.505 [0.335, 0.672]$0.0016
mediumsolo-opus190.637 [0.516, 0.772]$0.1403
mediumsolo-sol190.436 [0.266, 0.610]$0.0567
mediumfusion-default190.585 [0.431, 0.740]$0.0197
mediumfusion-best190.545 [0.369, 0.708]$0.0086
Variance across repeats
ArmMean sd across repeatsTasks whose solved flips
solo-haiku--
solo-luna--
solo-opus--
solo-sol--
fusion-default--
fusion-best--

Judge reliability

Headline verdict allowed. Accuracy floor 80%: mimo-pro 100%, muse-ember 88%.

Most different answers, with evidence

The tasks with evidence where an arm and the baseline differed most in completion.

fe-data-table (frontend): completion differs by +0.99

fusion-best (repeat 1): completion 0.99

  • gate tests-pass: ✓ pass
  • gate works-offline: ✓ pass external_requests = 0, needs <= 0
  • gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E1 tests (measured): all 7 hidden tests pass
  • E2 build (measured): everything compiles
  • E3 static (measured): clean
  • E4 diff_stats (measured): +171 -2 lines in 3 files
  • E5 a11y (measured): no violations (static source rules only)
  • E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)

Judge: All 7 hidden tests pass (E1) and static/build/a11y checks are clean (E2, E3, E5). index.html has lang, viewport meta, a title, a single h1, a <caption>, column headers with scope=col, row headers with scope=row, and a button in each column header with data-sort (lines 14-30). app.js updates aria-sort on the active column (and removes it elsewhere) and toggles ascending/descending per column, using numeric comparison for tickets and localeCompare for text (lines 12-31). The wrapper .table-scroll has tabindex=0 + role=region + aria-label so it is keyboard-reachable, and overflow-x:auto with table min-width:36rem means the table scrolls inside its own container on narrow screens while body max-width keeps the page from breaking (style.css lines 28-47). Design matches the brief: bordered/tinted header row (#e8eef6), zebra rows (nth-child(even)), borderless text-looking header buttons, caption on top. Focus-visible outlines are defined for both the wrapper and buttons (style.css lines 36-40). E3 shows no external requests, viewport meta present, no lint issues; E4 shows a compact 171-line addition across 3 files. Minor nits: numeric detection is hardcoded to the 'tickets' key rather tha

Measuring this cost $0.0050 and 22.2s: not part of the arm's cost or time.

solo-opus (repeat 1): completion 0.00

  • gate tests-pass: ✗ FAIL 6/7 hidden tests pass; failing: HiddenTests.test_exactly_one_column_reports_its_sort
  • gate works-offline: ✓ pass external_requests = 0, needs <= 0
  • gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E1 tests (measured): 6/7 hidden tests pass; failing: HiddenTests.test_exactly_one_column_reports_its_sort
  • E2 build (measured): everything compiles
  • E3 static (measured): clean
  • E4 diff_stats (measured): +178 -2 lines in 3 files
  • E5 a11y (measured): no violations (static source rules only)
  • E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)

Judge: Output A builds a real table with a caption (#people-caption), column headers with sorting buttons, row headers (scope="row" for each name), and a keyboard-reachable scroll wrapper (role=region, tabindex=0, overflow-x:auto with min-width:36rem so it scrolls on a 390px phone) — E3 static PASS, E5 a11y PASS. Sorting works for text and numeric columns via data-type, and the sorted column is toggled to aria-sort with a live status line. However, the hidden suite fails: E1 shows 6/7 with HiddenTests.test_exactly_one_column_reports_its_sort failing consistently across 3 runs, and the tests-pass hard gate is FAIL. This directly undercuts the core sort-reporting requirement even though the JS attempts to reset other headers to "none" (app.js lines 31-40), so correctness and ux_polish are reduced. The tinted header (#eef2f8), bordered wrapper, zebra-capable rows (only border-top given; no alternating row tint is actually applied — a design_fidelity gap), text-like header buttons, caption on top, visible :focus-visible outline, and viewport meta/lang/title all satisfy the design and a11y expectations (E2 build PASS, E4 diff_stats PASS), and everything is local/offline (works-offline pass). C

Measuring this cost $0.0056 and 53.9s: not part of the arm's cost or time.

fe-login-dark (frontend): completion differs by -0.93

fusion-default (repeat 1): completion 0.00

  • gate tests-pass: ✗ FAIL 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
  • gate works-offline: ✓ pass external_requests = 0, needs <= 0
  • gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E1 tests (measured): 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
  • E2 build (measured): everything compiles
  • E3 static (measured): clean
  • E4 diff_stats (measured): +156 -2 lines in 3 files
  • E5 a11y (measured): no violations (static source rules only)
  • E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)

Judge: The output delivers the requested structure: index.html has lang, title, viewport meta, one h1, a labelled Username and Password field, a Show password checkbox with an 8-line app.js toggle (app.js lines 1-8), a full-width Sign in button and a quiet Forgot your password? link (E3, E5). Accessibility is strong: E5 reports zero contrast failures and a 3px yellow :focus-visible outline (style.css line 118-121); autocomplete attributes are present. Design matches the brief: centred dark card (max-width 24rem, .page flex-centred), full-width primary button and underlined secondary link (E3 max_fixed_width_px=0). However, the hidden test suite fails 6/7, specifically HiddenTests.test_fits_small_screens (E1), so the responsiveness criterion cannot be scored highly; the remaining build/static/a11y checks pass (E2, E3, E5) and the file diff is minimal and clean (E4: +156 -2 over 3 files). | E1 shows 6/7 hidden tests pass but test_fits_small_screens fails, so responsiveness is weak and tests hard gate fails despite E2 build PASS, E3 static PASS and E5 a11y PASS indicating correct structure, labels and contrast. E6 screenshot n/a and E7 console n/a limit visual verification, so design and pol

Measuring this cost $0.0046 and 48.9s: not part of the arm's cost or time.

solo-opus (repeat 1): completion 0.93

  • gate tests-pass: ✓ pass
  • gate works-offline: ✓ pass external_requests = 0, needs <= 0
  • gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E1 tests (measured): all 7 hidden tests pass
  • E2 build (measured): everything compiles
  • E3 static (measured): clean
  • E4 diff_stats (measured): +170 -2 lines in 3 files
  • E5 a11y (measured): no violations (static source rules only)
  • E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)

Judge: Output A implements the brief cleanly: index.html (E1/E3) has lang, title, viewport meta, a single h1 'Sign in', a semantic <main>/<form> with properly associated <label for> on both fields and the 'Show password' checkbox, required inputs with autocomplete='username'/'current-password', a role="status" live region, and a 'Forgot your password?' link. style.css centres a 24rem dark card (flexbox + width:min(100%,24rem) and 1rem body padding) so it works from 390px to 1440px, uses #f1f5f9/#cbd5e1/#93c5fd text on #0f172a/#1e293b backgrounds and white on #1d4ed8 — all well above 4.5:1, confirmed by E5 (contrast_failures=0, 0 violations). Focus is a 3px #fbbf24 outline with offset, applied to :focus-visible plus a :focus fallback. app.js is 22 minimal lines implementing the show-password toggle by swapping input.type and a form-submit handler with a polite status message. All assets are local (E2 build PASS, hard gate works-offline PASS), and all 7 hidden tests pass (E1) with the static checks (E3) and diff stats (E4) passing. No images, so alt-text is not applicable. Minor nits: the submit handler always preventDefaults, and the card could use a tiny max-width guard at very narrow wid

Measuring this cost $0.0080 and 35.8s: not part of the arm's cost or time.

fe-login-dark (frontend): completion differs by -0.93

solo-haiku (repeat 1): completion 0.00

  • gate tests-pass: ✗ FAIL 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
  • gate works-offline: ✓ pass external_requests = 0, needs <= 0
  • gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E1 tests (measured): 6/7 hidden tests pass; failing: HiddenTests.test_fits_small_screens
  • E2 build (measured): everything compiles
  • E3 static (measured): clean
  • E4 diff_stats (measured): +44 -2 lines in 1 files
  • E5 a11y (measured): no violations (static source rules only)
  • E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)

Judge: Output A delivers a single self-contained index.html with lang, viewport meta, title, one h1, labelled Username/Password inputs with autocomplete, a working Show password checkbox toggle (inline script lines 42-50), a full-width Sign in button and a quiet 'Forgot your password?' link, all centred in a dark card (E3, E5: 0 contrast failures, 0 lint issues, 0 external requests). However, E1 reports 6/7 hidden tests passing with HiddenTests.test_fits_small_screens failing, so the responsiveness requirement from 390px phone to 1440px desktop is not fully verified despite the CSS using width:100%, max-width:360px and flex centering; this also trips the hard gate tests-pass (FAIL, E1). Everything else builds and passes static/a11y checks (E2, E3, E5). | File shows centred dark card with h1, labelled Username/Password, Show password checkbox toggling type, full-width Sign in button, Forgot link, bright #ffd54a focus-visible ring and high-contrast colours, matching a11y PASS E5 contrast_failures=0. However hard test gate fails: E1 tests FAIL 6/7 with HiddenTests.test_fits_small_screens failing, so responsiveness penalised despite viewport meta and max-width layout; build/static/diff PASS p

Measuring this cost $0.0056 and 66.6s: not part of the arm's cost or time.

solo-opus (repeat 1): completion 0.93

  • gate tests-pass: ✓ pass
  • gate works-offline: ✓ pass external_requests = 0, needs <= 0
  • gate no-console-errors: ? unverified unverified: skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E1 tests (measured): all 7 hidden tests pass
  • E2 build (measured): everything compiles
  • E3 static (measured): clean
  • E4 diff_stats (measured): +170 -2 lines in 3 files
  • E5 a11y (measured): no violations (static source rules only)
  • E6 screenshot (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)
  • E7 console (skipped): skipped: Playwright is not installed (uv sync --extra bench-visual && uv run playwright install chromium)

Judge: Output A implements the brief cleanly: index.html (E1/E3) has lang, title, viewport meta, a single h1 'Sign in', a semantic <main>/<form> with properly associated <label for> on both fields and the 'Show password' checkbox, required inputs with autocomplete='username'/'current-password', a role="status" live region, and a 'Forgot your password?' link. style.css centres a 24rem dark card (flexbox + width:min(100%,24rem) and 1rem body padding) so it works from 390px to 1440px, uses #f1f5f9/#cbd5e1/#93c5fd text on #0f172a/#1e293b backgrounds and white on #1d4ed8 — all well above 4.5:1, confirmed by E5 (contrast_failures=0, 0 violations). Focus is a 3px #fbbf24 outline with offset, applied to :focus-visible plus a :focus fallback. app.js is 22 minimal lines implementing the show-password toggle by swapping input.type and a form-submit handler with a polite status message. All assets are local (E2 build PASS, hard gate works-offline PASS), and all 7 hidden tests pass (E1) with the static checks (E3) and diff stats (E4) passing. No images, so alt-text is not applicable. Minor nits: the submit handler always preventDefaults, and the card could use a tiny max-width guard at very narrow wid

Measuring this cost $0.0080 and 35.8s: not part of the arm's cost or time.

Methodology

Run
v3-headline (completed), created 2026-11-19 10:42:16
Dataset
evals/datasets/v2, split test, 47 tasks, hash 84c35e7998b364bd
Arm solo-opus
strategy solo-opus: solo, models claude-opus, aggregator auto, rounds 1, judge off
Arm solo-sol
strategy solo-sol: solo, models gpt-sol, aggregator auto, rounds 1, judge off
Arm solo-haiku
strategy solo-haiku: solo, models claude-haiku, aggregator auto, rounds 1, judge off
Arm solo-luna
strategy solo-luna: solo, models gpt-luna, aggregator auto, rounds 1, judge off
Arm fusion-default
strategy auto: auto, models claude-haiku, gpt-luna, gemini-flash, gpt-terra, gemini-hydrogen, claude-sonnet, aggregator auto, rounds 1, judge off
Arm fusion-best
strategy panel-nano-select, overrides {'aggregator': 'select'}: panel, models claude-haiku, gpt-luna, gemini-flash, synthesizer claude-haiku, aggregator select, rounds 1, judge off
Model claude-opus
anthropic claude-opus-5-5: $4 in / $20 out per 1M tokens, verified 2026-11-17
Model gpt-sol
openai gpt-6.2-sol: $2 in / $10 out per 1M tokens, verified 2026-11-17
Model claude-haiku
anthropic claude-haiku-5-5: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
Model gpt-luna
openai gpt-6-luna: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
Model gemini-flash
google gemini-3.8-flash: $0.75 in / $3.75 out per 1M tokens, verified 2026-11-17
Model gpt-terra
openai gpt-6.1-terra: $1 in / $5 out per 1M tokens, verified 2026-11-17
Model gemini-hydrogen
google gemini-4-hydrogen: $1.5 in / $7.5 out per 1M tokens, verified 2026-11-17
Model claude-sonnet
anthropic claude-sonnet-5-5: $2 in / $10 out per 1M tokens, verified 2026-11-17
Model mimo-pro
openrouter xiaomi/mimo-v2.6-pro: $0.435 in / $0.87 out per 1M tokens, verified 2026-11-17
Model muse-ember
openrouter meta/muse-ember: $0.1 in / $0.5 out per 1M tokens, verified 2026-11-17
Judge models
mimo-pro, muse-ember
Repeats / seed
1 per task / seed 100
Intervals
95% percentile bootstrap, 2000 resamples of tasks (repeats are not independent); comparisons are paired on the tasks both arms were scored on
Verdict rules
non-inferiority margin 0.03 quality; at least 10 paired tasks; a task is a tie when the arms differ by at most 0.02; baseline solo-opus
Spend
$4.03 on the arms + $0.3940 on scoring and judging (cap $6.00)
Fusion version
0.2.0 (git 5abab47, with uncommitted changes)