Benchmarking¶
Canonical guide for measuring whether a Fusion panel of cheap models is cheaper, faster or better than a single frontier model. Cost methodology is in COSTS.md; config in CONFIGURATION.md.
Methods available today¶
| Method | What it measures | Spends API money |
|---|---|---|
| Benchmark mode | Cost, speed and ground-truth quality of several arms over a dataset, with every call recorded | Yes (live, capped) / No (--mock) |
| Offline dataset evals | Score, latency and cost of Fusion on sample cases | Yes (live) / No (--mock) |
| Claude Code A/B | Claude Code + Opus vs Claude Code + Fusion on your own tasks | Yes |
| Shadow A/B | Real baseline answer and blind pairwise verdict on live calls | Yes |
Benchmark mode is the framework the 0.2.0 study is built on. Its scorers, datasets, statistics and report were built in 0.2.0 and are extended in the 0.3.0 roadmap (Tasks 10 and 13 to 16); this page says what exists today.
Benchmark mode (fusion bench)¶
A study answers one question: for tasks whose right answer is known, is a panel of cheap models cheaper, faster or better than one frontier model? It runs arms over a dataset, measures every call, scores each answer against the task's ground truth, and keeps everything on disk.
Concepts¶
- Task (
BenchTask): one line of a JSONL file (or one YAML document) with anid, acategory(code_review,debugging,architecture,planning,coding,frontend,performance), aprompt, optionalcontextandfiles, adifficulty, andtruth: the category-specific ground truth. Prompts must be unique. A bare dataset name such astoyis looked up insrc/fusion/bench/datasets/, thenevals/datasets/bench/,evals/bench/,evals/datasets/v1/(socodingisv1/coding) andevals/datasets/(sov1is the whole shipped dataset). - Arm (
Arm): a strategy by name, with optionaloverrides(nested mappings merge), so "the cheap panel with three rounds" is one entry. Thedefaultarm set is six:solo-opus,solo-haiku,panel-nano,panel-refine,panel-cascade,panel-cheap-strong-synth. - Job: one (task, arm, repeat), keyed by a hash of the task, the arm's strategy and the model ids behind it, the repeat and the seed. A finished job is never run twice.
- Mode: every job runs in
Mode.BENCHMARK(what that changes): temperature 0 and a seed per repeat (seed + repeat - 1) for models whose catalog entry accepts sampling parameters (none of the packaged ones do since 0.3.0), no prompt trimming, no shadow run, every call streamed so time to first token and decode speed are measured, and no secret redaction (ground-truth tasks may contain secret-looking strings on purpose;--redactturns it back on).
Commands¶
# 1. See what a study would cost and take. No model is called; needs no API keys.
uv run fusion bench plan --dataset toy --arms default --repeats 3 --max-usd 5
# 2. Run it. Free and deterministic with simulated models:
uv run fusion bench run --dataset toy --arms default --mock
# ...or live, within a spending limit you state:
uv run fusion bench run --dataset my.jsonl --arms default --repeats 3 --max-usd 5
uv run fusion bench list # runs, newest first
uv run fusion bench show RUN # one summary row per arm (--json for scripts)
uv run fusion bench show RUN --task ID # one task: gates, criteria, evidence, the judge's reasoning
uv run fusion bench resume RUN --max-usd 8 # continue a stopped or interrupted run
uv run fusion bench resume RUN --retry-halted # ...and run again the jobs that halted (timeouts, no quorum)
uv run fusion bench report RUN # intervals, paired comparisons, $ per solved task, verdicts (md)
uv run fusion bench report RUN --format html # one self-contained file with the charts
uv run fusion bench compare RUN_A RUN_B # before and after a tuning change
uv run fusion bench spend # each budget's live spend and what is left
uv run fusion bench suite list # the packaged studies; `suite show NAME` for one
uv run fusion bench plan --suite ablation # price a suite (also: run --suite NAME)
uv run fusion bench recommend RUN # which arm to make the default, from a dev run
uv run fusion bench results RUN # write docs/BENCHMARK_RESULTS.md from runs
uv run fusion bench calibrate-judge --dataset toy --mock # how well do judges pick the better answer?
uv run fusion bench calibrate-judge --dataset v1 --mock --artifacts # ...the better page or faster code?
uv run fusion bench dataset validate evals/datasets/v1 --release # check a dataset (also: stats, build)
--arms takes strategy names (solo-haiku,panel-nano), name=strategy to rename, or default.
--config study.yaml supplies any BenchConfig field, including arms with overrides; options on
the command line win. Other options: --limit N (use N tasks, spread over categories and chosen by
seed), --seed, --concurrency (jobs at once, default 8), --no-cache, --redact,
--judge-models, --split dev|test|all (see Datasets).
Planning and the cost caps¶
bench plan prices every (task, arm) with the arithmetic of the
hard cost caps, from the catalog's prices and the task's own
size, plus what the scorer will spend. The result is a range, not a promise: answers are assumed
1,500 tokens long, a cascade is assumed to ask its whole panel in 40% of cases, and the low and
high ends scale every token count by 0.3 and 2.5. The time estimate follows the critical path of
each strategy using the catalog's latency tiers. When the expected cost is over the budget, the
plan suggests the largest study that fits, shrinking in a fixed order: fewer repeats, then fewer
tasks, then fewer arms (the dearest first). bench run refuses to start a live study whose plan
does not pass unless --yes is given.
Two caps hold a live study:
--max-usdlimits the run. Before a job starts, its worst case (a cascade escalating, every call at the assumed size, plus a 25% margin because real answers run longer than the assumed size) is reserved against what is left; when it will not fit, the run stops cleanly with statusstoppedand can be resumed with more money. Estimates are not measurements, so a run can pass the cap by at most the excess of the jobs in flight.- The live-spend ledger limits everything run live under one roadmap's budget: $15 for v0.3.0
(the current budget; v0.2.0 had $20). Every live job appends its cost (arm and scorer) to
bench-results/spend.json, a JSON array of{date, task, purpose, usd, budget}that is only ever appended to, and no job starts that would take its budget's total past the cap. An entry withoutbudgetwas written by 0.2.0 and counts asv0.2.0, never againstv0.3.0. A damaged ledger stops live runs rather than being reset. Simulated runs spend nothing and are never recorded.fusion bench spendshows each budget's total and what is left.
What is measured¶
Each item stores the answer, the arm's claims, the full ledger of calls (CallRecord) and a
BenchMetrics block, all measured and none derived later from guesses:
| Field | Meaning |
|---|---|
seconds_to_complete |
wall time from request to final answer |
cost_usd |
every call the arm made; eval_cost_usd is the scorer's, kept apart |
eval_cost_usd, eval_seconds |
what measuring and judging the answer cost in money and wall time (frontend and performance tasks: tests, timings, a browser, a judge); never added to the arm's cost or seconds_to_complete, shown beside them |
input_tokens, output_tokens, reasoning_tokens |
summed over the arm's calls |
output_tokens_per_s |
effective: output tokens divided by wall seconds |
decode_tokens_per_s, ttft_ms |
per model, from streamed calls |
calls, retries |
provider calls and the retries inside them |
quality, solved |
the scorer's output, and whether it reached the category's pass threshold (0.6; for coding, 1.0: every hidden test passes) |
cache_hits, latency_valid |
calls replayed from the cache; false when any were, see below |
Money per solved task is the arm's total cost divided by its solved items; bench show prints it.
A job that halted because a call hit the fan-out's 45-second per-model timeout scores 0, which
understates a slow model (GPT-6.1 Sol timed out on 21 of its 68 final-study jobs, Opus on 3).
resume --retry-halted, with the limits raised for that run (FUSION__FANOUT__PER_MODEL_TIMEOUT_SECONDS=240
FUSION__FANOUT__GLOBAL_TIMEOUT_SECONDS=300), runs those jobs again; every attempt stays in
results.jsonl, the last is scored, and the report says how many jobs were retried.
A job ends completed, halted (the pipeline stopped early, for instance without quorum: the arm
failed the task, and the item is scored and counted) or error (an exception, or a scorer
failure). Errors keep what the run had spent and are tried again by resume.
Resuming, results and the cache¶
Results go to bench-results/<run>/results.jsonl (one line per finished job, flushed as written;
this file is the record of truth), config.json, meta.json (what the run was when it started: the
dataset hash, each arm's models and each model's price and the date it was verified; a resumed run
keeps it), report.html once you ask for one, and the bench_runs and bench_items tables of
bench-results/bench.db (the same schema as the run history, in a separate file so studies never
appear in fusion stats). bench-results/ is git-ignored; FUSION_BENCH_DIR moves it. After a
kill, resume skips the job keys already finished and runs the rest; a torn last line is ignored.
Repeats come first in the job order, so a run that stops early has whole repeats of every arm.
Live runs wrap each provider in a response cache (bench-results/cache/, one file per answer,
keyed by provider and the whole request including model, prompt, schema, temperature and seed).
A replayed call is billed at zero (the provider did not charge for it), flagged cache_hit on its
record, and keeps the latency and speed it had when first answered. Reports still count it at
its price-list cost (list_cost_usd on the record, replayed_cost_usd on the item): two arms
that make the same call, such as a panel's Haiku request and a solo Haiku arm's, would otherwise
look cheaper the second time. The spend ledger counts only what was billed. Only successes are stored. Because repeats use different seeds,
they never share entries; a rerun of the same repeat does. A job with any cache hit has
latency_valid: false: its wall time is not a measurement of the arm, so summaries leave it out of
the percentiles. Use --no-cache for a speed study. The cache is off for --mock.
Simulated models (--mock)¶
--mock runs the real pipeline, with the real strategies, catalog models and prices, on
SimulatedProviders (providers/simulated.py). Each simulated model has a skill (from the
catalog's quality tier), a price, a speed (from its latency tier) and an error rate. It knows each
task's ground truth and finds each true point with a probability set by its skill and the task's
difficulty, sometimes asserting a known-wrong decoy. Whether it finds a point is the sum of three
things: how hard that point is for every model, how hard it is for models of the same provider, and
its own luck, so models make correlated mistakes, which is what decides whether a panel can
beat one model. Every draw is a hash of the seed, the model, the task, the point and the request's
seed: reruns are identical, and a different repeat has different luck.
Simulated runs also use a virtual clock (bench/virtual.py): when every task is asleep the
event loop jumps to the next wake-up, so a study of simulated twelve-second calls takes a second of
real time, parallel calls still overlap, and timings come out identical on every run (to the last
few bits: a job that starts later on the clock subtracts larger numbers). The simulation validates
the harness, the statistics and the cost caps offline; its skill numbers are assumptions, not
measurements, and say nothing about the real models.
Scoring¶
Ground truth first, an LLM judge second. A scorer (bench/scoring/) turns the arm's final answer
into a quality in [0, 1]; the claims behind it are never read, so an aggregator that dropped a point
gets no credit for a model having raised it. One scorer is registered per category, reads the
truth format below, and falls back to PointsScorer for a task whose truth is in points
format. A judge is called only for what the deterministic checks cannot decide, and only when the
study names --judge-models (catalog aliases); without them every scorer is free and offline.
Judge spend is eval_cost_usd, kept apart from the arm's cost and included in plan and in
--max-usd. Every scorer also accepts optional Evidence (measured test, timing or visual
results) so that the evaluators' output can feed a rubric without a new interface
(to_score_evidence converts it).
| Category | Scorer | truth |
Quality |
|---|---|---|---|
code_review |
ReviewScorer |
bugs: [{file, line, category, severity, description, aliases?}], line_tolerance (3); empty bugs = a clean change |
F1 of severity-weighted recall and precision |
debugging |
DebugScorer |
root_cause_tags, root_cause_aliases?, root_cause?, fix_keywords? |
0.7 × cause credit + 0.3 × share of fix keywords |
architecture, planning |
RubricScorer |
required_points, forbidden_points: strings or {id, text, keywords?, gate?, weight?} |
weighted share of required items met, less 0.5 × share of forbidden asserted |
coding |
CodingScorer |
expected_pass, hidden_files, command, timeout_s, solution?, flaws? (see Coding tasks) |
pass@1: the share of expected_pass hidden tests that pass |
frontend, performance |
ArtifactScorer |
the coding keys plus hard_gates, soft_criteria, evaluators?, perf?, site? (see Frontend and performance tasks) |
completion score: 0 if a hard gate fails, else the weighted mean of the soft criteria |
| any | PointsScorer |
points, decoys: {id, keywords, text?, weight?} |
weighted recall less 0.5 × share of decoys asserted |
A frontend or performance task in points format still scores on its points.
Coding. The answer's one patch is run against hidden tests in a sandbox; no model is asked and the scorer costs nothing. How: Coding tasks and the sandbox.
Review. The answer's findings are the places it points at (file:line, file, line N, or a
bare line N when the task has one file). A finding reports a seeded bug when it names the bug's
file within line_tolerance lines: full credit if it also says the bug's category (or an alias),
half if it only points there. Unmatched findings are false positives; one in a file the task does
not contain is a hallucinated file and counts double. Only bugs still unmatched go to the judge
(the first of --judge-models) together with the leftover findings; a finding with no location is
never penalised, since without a location it cannot be called false. A clean change scores
1 / (1 + false-positive weight): one false alarm leaves 0.5, below the pass mark.
Debugging. root_cause_tags are equivalent names for the one root cause. The answer's
hypotheses are its list entries (paragraphs if it has no list), in order; the rank of the first one
naming the cause gives 1.0 (first), 0.7 (second or third), 0.3 (later), 0 (none). If none names it
in so many words the judge is asked whether one of the first five says the same thing.
Rubric. Each item is judged met or not and the judge must quote the answer; a quote that is in
neither the answer nor the evidence voids the verdict. With several judges each item goes by
majority. Items with keywords need no judge. An item with gate: true that is missed caps quality
at 0.4. If no item can be decided (no judge, no keywords) the task errors rather than guesses.
Pairwise judge (PairwiseJudge, used for A/B comparisons between arms). Every judge sees both
orderings; a judge that names a different answer in each (it followed the position) is a tie, and
the rate is reported as position bias. Equal judge votes tie. Cross-family rule: a judge may not
come from a vendor whose models serve either arm (models favour their own family); with none left
the comparison refuses. The vendor is the model's maker, whichever API serves it
(ModelEntry.vendor: Claude through OpenRouter is Anthropic, qwen/qwen3.8-flash is Qwen), so an
open-weight panel excludes only its own makers' judges. Cost is the scorer's, never the arms'.
Neutral judges. neutral_judges: true in a study (the *-v3 suites set it) refuses to plan or
run it unless every judge's vendor is on no arm: not among the members and aggregators of any arm
(including the catalog synthesizer the router picks for an aggregator the strategy does not name,
and an in-run judge), on every rung strategy: auto may climb (fusion.bench.judges). Then every task of every arm is
judged by the same impartial judges, instead of the cross-family rule dropping a judge per arm. A
same-tier spare that stands in for a failed member is not known in advance and is not checked.
Calibrating judges. Before trusting a judge, measure it:
uv run fusion bench calibrate-judge --dataset my.jsonl --judge-models claude-haiku,gpt-luna --max-usd 1
uv run fusion bench calibrate-judge --dataset toy --mock # simulated judges, free
Each task with a known truth yields a good answer (says what the truth says) and a flawed one
(says little of it and asserts what is wrong); --cases file.jsonl supplies {task_id, good,
flawed} pairs instead. Every judge compares every pair in both orderings and the report gives
accuracy, tie rate, position-flip rate, Cohen's κ against the truth (over both orderings, so an
always-"A" judge scores κ ≈ 0), agreement between judges and κ per pair. Reports are kept in
bench-results/calibration/; a live calibration is forecast first, refuses to exceed --max-usd
or the live-spend cap, and records its spend.
The v2 calibration (2026-11-18, live, $0.197 of Task 14's $1). Candidates were OpenRouter models
from vendors on no 0.3.0 panel. Text judges, on the 60 seeded pairs of the v2 dev split, both
orderings: MiMo-V2.6 Flash 97% accuracy (3 failed calls of 120), Muse Ember 93% (6), MiMo-V2.6 Pro 93%
(7), Hy4 Flash 75% (24 failed calls, so ties), no order flips. The agentic judge (--artifacts
--cases-per-set 4, 8 frontend and performance cases): MiMo Pro 100%, Muse Ember 88%, MiMo Flash 62%
(below the 80% floor, so a study that names it gets no headline verdict). The *-v3 suites
therefore use MiMo Pro and Muse Ember (Xiaomi and Meta, on no arm). With two judges a rubric item
split 1-1 counts as not met. The report does not keep why a call failed (an error or no JSON).
Caveats. The seeded pairs are easy by construction (a judge that fails them is unusable; one
that passes may still fail on subtle differences), and an LLM judge validated on them can still
favour verbose or confident answers. Keyword and word-overlap checks reward wording the truth
anticipated: write aliases and keywords generously, and prefer judged items for open-ended
points. Simulated judges (--mock) follow assumed skill and position-bias numbers; their reports
test the harness, not any real model.
Datasets¶
evals/datasets/v2/ is the dataset of the 0.3.0 studies (the *-v3 suites): every v1 task
with its id and split, three more frontend and three more performance tasks, and 34 real-repo
coding tasks, upstream bugs of small open-source projects scored by the maintainers' own tests
(Dataset D): 197 tasks, at least
15 per category. Its split was frozen before any live run (_splits.json, written by fusion
bench dataset freeze; validation fails if a task moves or is not in it, and freeze --add takes in
a new one), its release promise is data (_dataset.yaml, read by validate --release), and
_review.yaml is the owner's review checklist (fusion bench dataset review), described in
evals/datasets/README.md. Files whose names start
with _ are never read as tasks.
uv run fusion bench dataset validate evals/datasets/v2 --release # also the frozen split and the review sheet
uv run fusion bench dataset freeze evals/datasets/v2 [--add] # freeze the split (once), or take in new tasks
uv run fusion bench dataset review evals/datasets/v2 # create or refresh the review sheet
A real-repo task runs pytest over tests/ in the sandbox (the parent commit's test file is the
visible test, the fix commit's is the hidden one), and is solved when every hidden test that passes
with the upstream fix passes (fail-to-pass and pass-to-pass). The rest of this section describes
v1, whose tasks v2 contains unchanged.
evals/datasets/v1/ is the ground-truth dataset the 0.2.0 study runs on: 157 tasks, 25 each of
code_review, debugging, architecture and planning (in three difficulties, with code tasks in
Python, TypeScript and Go, and six clean reviews, 24%, that test false alarms), 33 executable
coding tasks that are scored by running tests (below), and 12
frontend and 12 performance tasks that are scored on what the answer builds
(after that). Truth formats and
scorers are under Scoring. All tasks are synthetic and not yet reviewed by a person;
what that means for validity is in evals/datasets/README.md,
which is also where the licence note and the guidelines for adding tasks live.
Splits. Every task is in dev (15 per text category, 16 coding tasks, 6 each of frontend and
performance) or test (10 per text category, 17 coding tasks, 6 each of frontend and performance). dev is for tuning
(the cascade threshold, prompts, judge choice); test is touched only by the final study. A run uses
--split dev unless told otherwise (BenchConfig.split, default dev), so a tuning run cannot
see held-out tasks by accident; --split test and --split all are for the final study. A dataset
with no split labels (the toy dataset, your own file) is used whole under any split. The validator
rejects a prompt or a file set shared between the splits.
uv run fusion bench dataset validate evals/datasets/v1 --release # schema, ids, lines, secrets, splits, coverage
uv run fusion bench dataset stats evals/datasets/v1 # counts per category, difficulty, split, language
uv run fusion bench dataset build # compile authoring/v1/*.yaml into v1/*.jsonl
uv run fusion bench dataset build --check # fail if the JSONL is out of date
uv run fusion bench dataset build --generate 10 --category debugging --model claude-sonnet --max-usd 1
validate always checks each row's schema, unique ids and prompts, that bug files and line numbers
are inside the files supplied (and on added lines of the diff), that no string looks like a secret,
that a licence note sits next to the dataset (and a LICENSE beside each real-repo task), that no
prompt repeats a line of 32 or more characters from the hidden tests or the reference solution,
and that an answer written from a task's own truth
scores at least 0.9 under its offline scorer and a flawed one at most 0.5, and it runs every
executable task (see below; about 15 seconds for the 33 coding tasks, a minute for the 12
performance tasks, whose timings are taken one at a time). --release adds the published dataset's
promise: at least 100 tasks, 25 per text category, at least 30 coding tasks and 12 each of frontend
and performance, 8 per split (4 for frontend and performance) and 3 per difficulty in each
category, three languages, 20% clean reviews, and 30 to 400 lines per review diff or debugging task;
a dataset's _dataset.yaml replaces these numbers with its own (v2: 120 tasks, 15 per category, 30
real-repo tasks). Tasks are written as YAML (code review diffs mark each seeded bug with «id», and the compiler
computes the line numbers), compiled to JSONL, and a test fails when the two differ. --generate
has a model draft candidates into evals/datasets/authoring/candidates/ for a person to review; it
is forecast, held to --max-usd and the live-spend cap, and recorded in the spend ledger.
Coding tasks and the sandbox¶
The coding category is scored by running code, not by matching words. A task is a directory,
evals/datasets/v1/coding/<task_id>/:
| Path | Holds |
|---|---|
prompt.md |
what to build or fix; the format request for the answer is appended when the task loads |
repo/ |
the files the model sees: the code, and a few visible tests (repo/tests/) |
tests/ |
the hidden tests: copied in after the patch, never shown to a model |
reference/solution/, reference/flaw-N/ |
the files of a correct fix and of plausible wrong fixes, as new content (the loader turns them into diffs) |
task.yaml |
id, difficulty, split, tags, solution_summary, flaws (one summary each) and expected_pass |
The loaded task's truth is CodingTruth: the test command (default
python -m unittest discover -v -s tests -t .), timeout_s, mem_mb, hidden_files,
expected_pass (the ids of the hidden tests a correct patch passes), and the reference solution
and flaws as diffs. The last two are for the validator and for simulated models; a real model
never sees any of truth. How to add a task is in
evals/datasets/README.md.
The answer is one patch. No agent loop, so solo and panel arms are compared fairly: each answer
holds a single patch, either a unified diff or the complete new content of each changed file under
=== path === lines. Panelists put it in the patch field of their claims JSON
(PanelAnswer.patch); the llm aggregator is asked for one patch merged from theirs; vote and a
cascade's early exit return the patch most models gave (identical up to whitespace), as that model
wrote it; best_of returns the answer the others' claims back most. panel-digest returns every
answer for Claude Code to merge, so it holds every panelist's patch and scores 0 whenever they
differ: do not benchmark it on coding. Diffs are applied by matching context, not by trusting hunk numbers,
because models get line numbers wrong; a patch that does not apply, names an unsafe path, or is
missing scores 0, and the item's score.details says which.
Scoring. The scorer applies the patch to the task's files in a fresh sandbox, copies the hidden
files over the result (so a patch cannot weaken a test), and runs the command. Quality is pass@1:
the share of expected_pass that pass, and a task is solved only at 1.0. Flake guard: when the
first run does not pass everything, it is run twice more (fresh sandboxes) and a test counts as
passed if it passed in most runs; tests whose result changed are listed as flaky. A run that times
out is not repeated. Results are memoised by (task, patch), so identical patches from several arms
or repeats run once.
The sandbox (bench/sandbox.py, a reusable Sandbox context manager with copy_in, run and
collect, which the evaluators also use) runs each command in a temporary directory
with a wall-clock limit that kills the whole process group, resource limits (CPU, file size, core
dumps; memory on Linux), output caps, a scrubbed environment (only PATH and locale pass: no API
keys, HOME and TMPDIR inside the sandbox) and, where the platform offers it, no network and no
writes outside the sandbox (sandbox-exec on macOS, unshare --net on Linux). Where it does
not, score.details["isolation"] says none and only the limits apply. FUSION_SANDBOX_ISOLATION
chooses: auto (default), require (refuse to run without isolation), off. It is a best-effort
guard for a dataset you chose, not a security boundary; see SECURITY.md.
It runs only inside benchmark mode, on tasks from the dataset you named, and is never reachable
through MCP.
best-of-n-verified (a strategy, not in the default six): each of the three cheap models
writes a patch and the task's visible tests choose, with no synthesis call: the patch with the
highest visible pass rate wins (a patch that does not apply loses; ties go to the patch more models
gave, then to claim agreement). The visible tests are weaker than the hidden ones on purpose: the
arm measures what cheap verification buys, and on the shipped tasks they catch about two thirds of
the wrong fixes. It runs code, so the pipeline refuses it outside benchmark mode
(BenchmarkOnlyError): Fusion's tools never execute anything and Claude Code stays the only
executor. Verification time counts toward the arm's wall time in live studies; with --mock it
takes no virtual time, so simulated timings leave it out.
verified-synth (a strategy, not in the default six) generalises it: the visible tests are
no longer needed. Each of the three cheap models writes a complete patch (from a patch approach:
minimal fix, root-cause fix) and one to three check scripts that should fail on the code as it
is and pass once it is fixed, plus guardrails. The checks are validated, run on the unpatched copy
(a check that already passes is dropped: it cannot discriminate), then every patch is run against
every check in its own sandbox copy; the task's visible tests, when it has any, join as a trusted
check. A patch scores 1 per check another model (or the task) wrote that it passes and 0.5 per check
its own author wrote, so two models that agree on a wrong fix cannot outvote one that passes
everyone's checks; a tie goes to one select call among the tied patches. The matrix, every check's
fate and the reason for the pick are kept in each item's verification field of results.jsonl
(how it works: ARCHITECTURE.md).
Like best-of-n-verified it runs code, so it is refused outside benchmark mode. Hidden tests are
never used to pick: they grade the result afterwards, and a test checks that no panel prompt
contains them. Simulated models write checks that compare file hashes against their own patch and,
when the model "understood" the task (a draw on its skill), against the reference fix: they are
oracles, so a --mock run of this arm is an upper bound on what real checks achieve.
uv run fusion bench dataset validate evals/datasets/v1/coding # runs every task's solution, flaws and tests
uv run fusion bench plan --dataset coding --arms solo-opus,panel-nano,best-of-n-verified --mock
uv run fusion bench run --dataset coding --arms solo-opus,panel-nano,best-of-n-verified,verified-synth --mock
Caveats. The tasks are small, self-contained Python and standard library only (the sandbox runs
any command, but no other language is shipped), written by an LLM and not yet reviewed by a person
(provenance).
Hidden tests check what the prompt specifies, so a correct patch that reads the prompt differently
fails; the tests were run against a reference fix and two wrong ones, which proves they can tell
them apart and nothing more. Simulated models for a coding task know the reference fix and its
flaws, so --mock results validate the harness, not any model's ability to code.
Frontend and performance tasks: evidence and the agentic judge¶
"Was the task completed better?" cannot be answered by reading text. For a page or a faster function
the judge needs what a human reviewer would collect: run the tests, lint the code, time it, open the
page. Two categories work that way: 12 frontend tasks (landing page, sign-up form, responsive
navigation, dashboard widget, pricing table, FAQ accordion, product grid, dark sign-in card, contact
form, article layout, modal dialog, sortable table) and 12 performance tasks (a slow function, a
benchmark harness and the reference timing: deduplicating, counting words, range sums, sorted
intersection, anagram groups, moving average, maximum subarray, breadth-first search, rendering a
log, counting inversions, a prime sieve, two-sum). They are directories like the coding tasks
(evals/datasets/v1/frontend/<id>/, performance/<id>/, same layout and the same single-patch
answer) with these additions to task.yaml:
| Key | Holds |
|---|---|
category |
frontend or performance |
helpers |
[sitecheck] copies the shared test helper (src/fusion/bench/datasets/helpers/sitecheck.py) into the hidden tests |
hard_gates |
facts that must hold: {id, evidence, metric?, max?, min?}; with no metric the evidence's own verdict must be true |
soft_criteria |
weighted criteria: {id, weight, source, description}, source one of tests, static, perf, a11y, visual, judge |
perf |
{script, sizes, samples, warmup, max_ratio, scaling_slack, max_noise} (performance tasks) |
site |
{entry, timeout_s}: the page to open (frontend tasks) |
evaluators |
the evaluators to run, by name (default: by category) |
The hidden tests of a frontend task are structural (unittest over the parsed HTML and CSS: labels,
landmarks, ARIA wiring, a media query, contrast, no skipped heading level); they cannot run
JavaScript or lay anything out. A performance task's hidden tests check behaviour, and its
benchmark script (tests/bench_workload.py, hidden too) defines setup(size) and run(state).
Slow code is correct code, so a performance task's starter passes the tests and fails the benchmark.
Evaluators (bench/evaluators/) run on a copy of the answer's files and never write to them.
Each returns Evidence (kind, ok, metrics, artifact_path, summary, cost_usd, plus id,
seconds and a status of measured, skipped or unstable); judges cite evidence by id.
| Evaluator | Kind | Measures |
|---|---|---|
tests |
tests |
the hidden tests (rerun once to catch flaky tests, results memoised by task and patch), per test; pass_fraction, visible_pass_fraction |
build |
build |
every Python file compiles; JavaScript passes node --check where node exists |
static |
static |
ruff (default), mypy, eslint, tsc where installed and named in static_tools; cyclomatic complexity of every Python function (read from the AST, so no extra dependency); new third-party imports; secret-looking strings; for pages a viewport meta tag, media queries, the widest fixed width and requests to other hosts |
diff_stats |
diff_stats |
lines and files added, changed, removed |
perf |
perf |
the benchmark at each of sizes, in a fresh process each, after a warm-up and at least five samples: median and p90 wall time, noise (MAD over the median), peak RSS, the scaling exponent (slope of log time on log size, which exposes an O(n²) answer that is fast on a tiny input) and the ratio and scaling excess against the reference solution measured the same way |
visual |
screenshot |
headless Chromium at 390x844 and 1440x900: screenshots (full page and above the fold), horizontal overflow, visible text |
console |
console |
errors the page logged and requests that failed or were blocked |
a11y |
a11y |
violations by impact, contrast failures and a keyboard-focus smoke check |
A perf measurement whose noise is above the task's max_noise is unstable: it carries no verdict
(ok is None) and is measured again next time, never scored. Measurements are taken one at a time.
The browser evaluators need the optional extra: uv sync --extra bench-visual and
uv run playwright install chromium (the extra is fusion-code-orchestrator[bench-visual]);
FUSION_BROWSER_PATH points at a Chromium or Chrome you already have, instead.
Without it they are skipped, not failed: a gate that reads their evidence is unverified,
which does not fail an answer, and a criterion that needs them is left out of the mean. The
accessibility check then falls back to the same rules read from the HTML source and CSS, which
cannot see what a script adds, so it under-reports and never gives full credit. Inside the browser
the rules are assets/a11y_rules.js (a small subset of axe-core's, in axe's result shape); to use
axe-core itself, put its axe.min.js next to it (src/fusion/bench/evaluators/assets/). Nothing is
fetched from a CDN, ever. How the browser is contained: SECURITY.md.
Gates, criteria and the completion score (scoring/completion.py). Every task has hard gates
(the defaults are "the hidden tests pass" and, for performance, "within the time budget" against
the reference) and weighted soft criteria. The completion score is 0 if any gate fails, else the
weighted mean of the scored criteria, so a beautiful page that breaks its tests completes nothing.
A criterion is scored from evidence when it can be (tests: pass fraction; static: 0.1 per lint
finding, 0.03 per complexity point over 10, 0.5 per secret, 0.15 per new dependency, 0.2 per
request to another host; a11y: violations weighted 1 / 0.5 / 0.2 / 0.05 by impact against a
budget of 3; perf: full marks within 1.25x the reference, none at 10x, less for a scaling
exponent more than 0.25 above the reference's; visual: the share of the page's basic checks that
hold) and by a judge otherwise; those weights are judgement calls, kept in one place, and the
benchmark's numbers move with them. Without judges the criteria only a judge could score
(design_fidelity, ux_polish) are left out of the mean, not counted as zero, and bench show
--task ID lists them as unscored.
The agentic judge (scoring/agentic.py; used when the study names --judge-models). An LLM
that inspects the outputs with read-only tools, then rules:
- Tools:
list_files,read_file,grep,view_screenshot(vision),get_evidence(kind),run_evaluator(name)(re-runs a whitelisted evaluator, at most twice) andsubmit_verdict. The providers have no native function calling, so the tools are a JSON protocol that works everywhere: each turn the model answers with one{"tool": ..., "args": ...}object and gets the result as the next message. A hard step cap (default 12, the last step must submit) and a per-run money cap (max_usd, default $0.50 per judge and ordering) end a judge that never decides; its criteria then fall back to the measured scores. Every call and result is logged to the item's trail (bench showcounts them;results.jsonlholds them). - Blind: outputs are labelled
A/Bwith a seeded random assignment that is recorded, and both orderings run, so a judge that names the same position twice is a tie and is reported as position bias. The judge sees paths relative to each output, never a local path, an arm or a model. - Cross-family, several judges: a judge from a vendor that serves the arm is left out (the
arm's members and its named aggregator), at least two must remain, and per-criterion scores are
averaged with disagreement above 0.3 flagged. Only a vision-capable judge (
supports_vision) is offered screenshots; with none, the verdict says so. - Rubric and gates: the harness, not the judge, computes the hard gates from the evidence. The judge scores each soft criterion from 0 to 1 and writes a paragraph that cites evidence ids (a verdict that cites none, or scores outside 0 to 1, is rejected and the judge tries again).
- Untrusted input: everything derived from a model's answer (files, test output, evidence text)
arrives inside
<untrusted>markers, a closing marker inside it is defused, and the judge is told never to follow instructions found there. The tools cannot write, run commands or reach the network.
Judge reliability. calibrate-judge --artifacts runs the judges on the validated pairs the
datasets already contain: each task's reference solution against each of its flaws, which are
deliberately broken pages (a missing label, low contrast, a layout that overflows a phone, a fake
modal) and known-slow or fast-but-wrong code. It reports accuracy, ties, position flips, κ and
agreement as the text calibration does, and accuracy per set (visual, perf).
uv run fusion bench calibrate-judge --dataset v1 --artifacts --judge-models gemini-flash,gpt-sol --max-usd 1
uv run fusion bench calibrate-judge --dataset v1 --artifacts --mock --cases-per-set 4 # simulated, free
A study whose judges scored an artifact gets no headline verdict unless each judge's latest
artifact calibration reaches the accuracy floor (default 80%; FUSION_JUDGE_ACCURACY_FLOOR or
--accuracy-floor on the calibration): bench run and bench show print "Headline verdict
blocked" and why, and show --json carries judge_gate. Calibrations of simulated judges only
count for simulated runs.
What measuring costs. Evaluator and judge money and seconds are stored as eval_cost_usd and
eval_seconds, never added to the arm's own cost or latency, and shown in their own columns of
bench show so the price of measurement is visible. In a --mock study eval_seconds is the one
real-time number: the evaluators really run, so it differs between machines and reruns, unlike every
other figure.
Caveats. The datasets are synthetic (provenance). Hidden frontend tests check structure, not how a page looks; a screenshot judges visuals, not intent, and a judge that sees a pleasant page can still miss a broken interaction no test covers. Reference screenshots are not shipped (a task's brief is text). Static accessibility rules are a floor, not an audit, and the in-page ones are a subset of axe-core's. Timings depend on the machine: the comparison is always against the reference measured in the same process on the same machine, a noisy measurement is left unscored rather than guessed, and a loaded machine produces more of them. The default criterion weights and penalties above are choices, not measurements. Simulated judges read the same evidence a real one would and score it with noise by an assumed skill; their calibration tests the harness, not any model.
Suites and the final study¶
A suite is a study kept as a file (src/fusion/bench/suites/*.yaml): the BenchConfig fields
plus a description and a stage, with arms written as a strategy and overrides. fusion bench plan
--suite NAME prices it, run --suite NAME runs it, and options on the command line still win
(--suite headline --mock --repeats 1). --suite also takes the path of a suite file of your own.
A task quota (tasks per category, a seeded sample of each; categories not named are left out)
gives every suite a stratified subset that includes the executable, frontend and performance
categories in small numbers, and spend_stop_usd stops a suite at $14.50 of the $15 ledger cap,
keeping the reserve. The planner prices a suite's design against what is left below that line.
| Suite | Stage | What it answers | Budget |
|---|---|---|---|
ablation |
tune on dev |
panel size 1/2/3/5, refinement, a strong synthesizer, and the cascade threshold (0.5 and 0.7; more if money remains). The 0.2.0 self-mixture arms (one model sampled at several temperatures) were removed in 0.3.0 | about $4.80 |
latency |
tune on dev |
whether early return, hedging or a cascade shortens the wait; the response cache is off, since a replayed call has no real latency | about $1.20 |
headline |
final, on test |
Opus 5.5, Sol, Haiku and Luna alone against the default panel and the best variant, two repeats | about $11 |
ablation-v3 |
tune on v2 dev |
nano, open-weight, mixed and mid panels; lenses on and off; llm, select, vote and digest aggregation; the cascade at 0.5 and 0.9; verified-synth; auto with the frontier rung off and on; 9 tasks, neutral judges |
$3.70 |
latency-v3 |
tune on v2 dev |
Opus 5.5 alone against auto, panel-duo (the 0.2.0 default) and panel-duo with Luna at effort none, on 30 text tasks, cache off |
$2.35 |
route-v3 |
tune on v2 dev |
each rung of the auto ladder alone (Haiku with a self-check, the nano panel, the mid panel) on 20 text tasks, to calibrate the ladder; neutral judges |
$2.40 |
headline-v3 |
final, on v2 test |
Opus 5.5, Sol, Haiku and Luna alone against auto and the best ablation variant (the nano panel with select, frozen from ablation-v3), 47 tasks (every text task of the split, 3 coding, 2 frontend, 2 performance), neutral judges |
$5.70 |
The *-v3 suites are the 0.3.0 study (roadmap Task 16) on dataset v2; the others are the 0.2.0
studies on v1, kept to reproduce them. Their caps add up to $14.15, but what bounds them is the
ledger's stop at $14.50 of the $15. The executable categories are sampled thinly because a
real-repo coding task shows a whole package (up to about 50,000 tokens) and can cost as much as
several text tasks of the same arm. In the 0.3.0 runs the planner was about 1.6 times pessimistic
(ablation-v3: $3.37 expected, $2.08 spent), so the headline suite was widened to every text task
of the test split.
The procedure that keeps the result honest:
- Plan, then run the tuning suites on
dev(ablation,latency): about $6 together. All tuning (the cascade threshold, panel size, the synthesizer) happens on these tasks and nowhere else. - Choose, with a stated rule.
fusion bench recommend RUNtakes the cheapest arm per solved task among the cost-quality frontier arms within the margin of the best, says when its interval overlaps the best arm's (at this size it usually does: the choice is then a lean, not a finding) and prints thefusion-bestline for the headline suite. If the winner beats the shipped default, change the default strategy instrategies.yamland say so in the changelog, even if that means Fusion's default is not the panel the project started with. - Freeze, then run
headlineonce ontest. Do not change an arm afterwards. About $11. - Publish.
fusion bench results RUN --also ABLATION --also LATENCY --html docs/benchmark-report.html --svg docs/assets/benchmark-cost-quality.svgwritesdocs/BENCHMARK_RESULTS.md: the verdict engine's text unedited, the supporting studies, the spend per run and the limitations. Nothing in it is typed by hand.
A suite's estimate comes from the planner, which assumes 1,500-token answers; the live total can be
higher for models that think at length (Opus 5.5 with adaptive thinking), and the caps hold either
way: a run stops cleanly, resumable, when the next job's worst case would not fit. Stopping early
leaves whole tasks for every arm (repeats first, then tasks), so the arms stay paired. Judges:
the 0.2.0 suites use gpt-luna, whose vendor sits on a panel, for the rubric items the keywords
cannot settle, and their agentic judge of frontend and performance tasks is skipped for panels
(those tasks keep their measured scores); say so whenever quoting one of their scores. The *-v3
suites use the neutral judges MiMo Pro and Muse Ember, from vendors on no arm.
Calibrating the auto ladder¶
strategy: auto starts a request on a rung of its ladder from a difficulty score and climbs when
the answer shows it is not enough (CONFIGURATION.md).
Its thresholds are calibrated on the dev split, never on test:
- Run
route-v3: each rung's strategy as a fixed arm (rung0,rung1,rung2) on the same tasks. A rung-0 item records the model's self-check confidence (self_confidenceinresults.jsonl); anautoitem records its route (route: the difficulty estimate, the start rung, each rung tried and why). - Replay the ladder over those results for a grid of
easy_below,hard_fromandself_confidence_below: each task starts at the rung its score picks (computed from the task as a run computes it, files included) and rung 0 climbs to rung 1 when its confidence is below the threshold. Escalations above rung 1 are not replayed. - Take the cheapest setting whose mean quality is within the 0.03 margin of the best rung alone, and look at the cost-quality frontier around it before freezing it, since one task moves the mean by about 0.03 at this size.
- Run
autoon the same tasks to compare the live ladder with the replay.
What the 0.3.0 calibration found (20 dev text tasks; the numbers are in
BENCHMARK_RESULTS.md): the difficulty signals did not separate the tasks that
needed the mid panel, while rung 0's self-confidence did (answers at 0.85 or more were mostly right,
at 0.6 or less mostly not). The shipped thresholds sat at the knee of the frontier: 95% of the mid
panel's quality at 6% of its cost; the next point bought 0.01 of quality for 2.5 times the cost, so
the thresholds were kept. The live auto run showed that the ambiguity classifier rated 5 of 9
borderline tasks hard and sent them to the mid panel, which cost 16 times the nano panel for about
0.03 more quality on them, so it was switched off (classifier_model: null).
Tuning the prompts (fusion tune)¶
fusion tune improves Fusion's own prompts, the prompt pack (src/fusion/config/prompts/*.md,
described in CONFIGURATION.md), by Harness²'s Propose–Probe–Compose,
on the dev split only: the test split is never loaded for probing, so the headline study stays
untouched by tuning.
fusion tune --mock # free, simulated: shows the procedure
fusion tune --arm auto --limit 24 --max-usd 2 # live: probes are ordinary capped bench runs
fusion tune --mode deep --k 4 --axes inv --max-usd 1
- Slices.
--limitdev tasks (spread over categories) are split, per category and seeded, into tuning tasks and a held-out slice (--holdout, default 25%). - Probe the base. The arm (
--arm, defaultauto) runs on the tuning tasks with the pack in use. A probe is an ordinary bench run (run_benchwith the pack active), so the response cache, the run's cap and the live-spend ledger apply to it. - Propose (wide, the default). For each axis, K proposers (
--k, default 3) each suggest one edit to one prompt of that axis: INV (procedure: output shape and verification steps:rules,claims,select,self_check, the judge and synthesizer roles) or DOM (coding knowledge per task type: the reviewer, debugger, architect and planner roles,task.default). A proposer sees the axis's prompts, the base's mean quality and solved share per task type and the edits proposed before it; never a task. - Validate, then probe each edit alone. An edit must name a prompt of its axis, keep every
{{placeholder}}(the untrusted-input rules among them), keep each prompt under 4,000 characters and the pack's growth under 1,500, and contain no line of grader material (truth points, decoys, hidden tests, reference solutions, rubrics, from every split). An invalid edit is recorded and never probed; one that leaks is not even shown again. - Compare. On the tasks both probes scored: the mean per-task quality difference with a
task-clustered 95% bootstrap interval. The noise band is the larger of half that interval
and
--min-gain(0.02). Verdict:helped(the difference clears the band),hurt(it falls below minus the band),no_effect, orconfounded(it clearly helped one task type and hurt another, or jobs failed so the probes rest on different tasks). Only a task type with at least--min-category-tasks(2) compared tasks counts here: one task moving either way is noise, and the verdict's reason names such task types instead. - Compose. K composers build finals from the measured edits (the behavioural contrasts):
composer 1 merges every edit that helped (appends by gain; the best
replaceof a prompt wins), the others are model calls that may narrow aconfoundededit to the task types it helped. Finals are validated and probed like proposals. - Keep and confirm. The best final is kept only if it beats the base by more than the noise
band; it is then probed once on the held-out slice, with the base, and adopted when it holds up
there (a positive mean effect that is neither
hurtnorconfounded; the slice is too small to ask for the band again).
Deep mode (--mode deep) runs K rounds instead, alternating axes: each round proposes one
edit to the best pack so far and keeps it only if it beats that pack; the result is confirmed on
the held-out slice the same way.
Output (bench-results/tune/<id>/): evidence.md (each edit, its effect with interval and
band, by task type, and its verdict), pack.diff (the adopted pack against the base; empty when
nothing was adopted), pack/ (the adopted pack, for FUSION_PROMPT_PACK) and tune.json. Nothing
changes in use until you point FUSION_PROMPT_PACK at that directory (or copy files into
config/prompts/).
Money. A live run needs --max-usd; it is shared by every probe and every proposer and
composer call, each checked against what is left and against the live-spend ledger (stopping at
$14.50 of the $15 cap). When either would be passed, the run stops cleanly and still writes what
it measured. A run makes at most 1 + axes × K + K + 2 probes (wide) or K + 3 (deep), and prints
the planner's estimate of one probe before it starts. The proposer and composer default to a
judge-role model (--tuner-model names another).
Simulated runs cannot learn anything about wording, since a simulated model ignores it. For
--mock, the simulator plants effects instead: the simulated proposer appends one of a few fixed
sentences (providers/simulated_tune.py) and a panel call whose prompt contains one gets a skill
change on the task types listed with it, so a mock run shows every verdict. These effects are
made up: a mock run tests the procedure, never that an edit helps a real model.
Statistics, reports and how to read them¶
fusion bench report RUN turns a run's items into statistics, verdicts and a report: --format md
(default, for a README or docs page), --format html (one self-contained file, written to the run's
folder or to -o PATH) or --format json (the same data, for scripts). --baseline ARM picks the
arm the others are compared with (default solo-opus, or solo-frontier in a 0.2.0 run, when the run has it, else the first arm);
--margin, --min-tasks and --resamples set the verdict rules below. fusion bench compare
RUN_A RUN_B compares the same arm across two runs on the tasks both have, paired by task, and
says whether the second is cheaper, faster, not worse or better.
Intervals. Every mean has a 95% percentile-bootstrap interval that resamples tasks, not runs: the repeats of one task share a prompt and a truth, so they are not independent, and resampling them would make the interval too narrow. Five repeats do not narrow it; more tasks do. Intervals are seeded, so a report is reproducible. With fewer than two tasks there is an estimate and no interval.
Paired comparisons. An arm is compared with the baseline on the tasks both were scored on, as differences per task. Reported: the mean difference with its interval, wins, ties and losses (a tie is a difference of at most 0.02), the exact two-sided sign test over the non-tied tasks, and an effect size (the mean difference over its spread across tasks, Cohen's d for pairs). Ratios of cost per solved task, cost per task and seconds per task are bootstrapped the same way; below 1 is better.
Money and speed. $ per solved task is the arm's whole cost (every item, including failures
and errors, since the money was spent) divided by the items it solved: the headline efficiency
number. An arm that solved nothing has none. Seconds are the median and 90th percentile over items
whose wall time is a measurement (cache replays are left out). Output tokens/s is effective
(output tokens over wall seconds); decode speed and time to first token are per model and come from
streamed calls. Tokens per solved is input plus output tokens over solved items. Quality per
dollar is mean quality over cost per task; quality per minute is mean quality over mean minutes
per task. Variance across repeats is the mean, over tasks, of the standard deviation of quality
across the repeats, and the share of tasks whose solved/unsolved result changes between them: an
arm whose result depends on the repeat needs more repeats before its mean means much. Scoring and
judging (eval_cost_usd, eval_seconds) are shown beside each arm and never inside its cost or
speed.
Frontier. An arm is on the cost frontier when no other arm is at least as cheap per task and at least as good, and strictly better in one; likewise for the latency frontier (median seconds). The HTML charts join the frontier arms with a line and draw each arm's interval as a bar. The frontier says which arms are worth considering, not that the gaps between them are real: look at the intervals.
Verdicts. For each arm against the baseline, overall and per category, four claims, each yes, no or inconclusive, with the numbers behind it in the report:
| Claim | yes when | no when |
|---|---|---|
| cheaper | the interval of the cost-per-solved ratio is wholly below 1 | wholly at or above 1 |
| faster | the interval of the seconds-per-task ratio is wholly below 1 | wholly at or above 1 |
| not worse | the interval of the quality difference is above −0.03 (the non-inferiority margin, --margin) |
below −0.03 |
| better | the interval of the quality difference is wholly above 0 | wholly at or below 0 |
Otherwise the claim is inconclusive: the study cannot tell, which is a result, not a failure. Below
--min-tasks (default 10) paired tasks nothing is claimed. The intervals are two-sided 95%, so
"not worse" is a slightly stricter test than the usual one-sided non-inferiority check. Verdicts are
about these tasks, this dataset and these judges; they do not transfer to other work without
repeating the study.
What blocks a verdict. When judges scored a frontend or performance task and their latest artifact calibration is below the accuracy floor (or missing), the headline ("all tasks") verdicts and those categories' verdicts read blocked and the report says why; see judge reliability.
Caveats the report prints for itself: a simulated run (the numbers test the harness, not any
model), a run that did not finish (arms may have answered different tasks), costs that are a lower
bound (an unknown price), errors, cache replays left out of the latency, fewer than three repeats,
and a methodology footer read from today's files because the run has no meta.json. The footer
always gives the dataset hash, each arm's strategy and models, each model's id, price and
verified_on date, the judge models, repeats, seed, the bootstrap and verdict rules, and the spend.
The HTML report adds a typical run's call timeline per arm, and, for frontend and performance
tasks, the evidence (gates, measurements, the judge's reasoning and screenshots) of the tasks where
an arm and the baseline differed most.
Offline dataset evals¶
Datasets live in evals/datasets/ as JSONL: code_review, debugging, architecture, planning.
# Live eval on code review cases (costs API credits)
uv run python evals/runners/run_offline_eval.py --dataset code_review
# Save results for analysis
uv run python evals/runners/run_offline_eval.py \
--dataset debugging \
--output evals/results/debugging-$(date +%Y%m%d).jsonl
# Compare all task types quickly, or run without keys
uv run python evals/runners/compare_pipelines.py
uv run python evals/runners/run_offline_eval.py --dataset code_review --mock
Add custom cases to evals/datasets/*.jsonl:
{"id": "my-case-1", "diff": "...", "context": "...", "expected_themes": ["auth", "sql"]}
Evaluating Fusion on your own task¶
- Prepare the input: a diff, an error with logs, an architecture question or a feature description.
- Run it through the CLI and note the
run_id:
uv run fusion review-diff --file my-change.patch > result.json
uv run fusion runs show RUN_ID
- Inspect the scores in the output:
evals.final.overall_score: hybrid quality score (0 to 1)evals.deterministic: safety and completeness flagsdisagreement.disagreement_score: panel disagreementrouting.selected_panel: which models ranconfidence: synthesis confidence
Claude Code A/B¶
fusion bench measures Fusion's answers; fusion bench e2e measures what matters for cost: whole
Claude Code sessions. It runs headless Claude Code on every executable coding task of a dataset,
each in a sandboxed temporary copy of the task's repo, for executor (Opus, Sonnet, Haiku) x Fusion
(off, on) arms, and records whether the hidden tests pass, wall time, host cost (Claude Code's own
token counts at catalog prices) plus Fusion's cost from its ledger, turns, tool calls and Fusion calls. Results go to the
same store and reports as any study (fusion bench report adds a "Claude Code end to end" table).
uv run fusion bench e2e --mock --limit 4 # simulated: free, no keys
uv run fusion bench e2e --limit 6 --max-usd 3 # live: needs claude and ANTHROPIC_API_KEY
Arms, isolation, every metric and the manual comparison on your own tasks are in CLAUDE_CODE_AB.md.
Where Fusion is expected to help: security-sensitive review, high-stakes architecture decisions, debugging with ambiguous symptoms, and answers you want scored before merging. A single frontier model is expected to win on small localized edits, thin context, and latency-sensitive loops where panel fan-out is too slow. These are hypotheses to test, not results.
Shadow A/B (live measurement)¶
To measure quality against the real baseline instead of an estimate, enable shadow mode. Fusion then also sends the same sanitized task to the baseline model; a blind pairwise judge scores both answers in randomized order, and the verdict plus actual baseline cost and latency are stored in SQLite.
# .env: off (default) | sampled | always
FUSION_SHADOW_MODE=sampled
FUSION_SHADOW_SAMPLE_RATE=0.2
Per-call override on any orchestration tool: shadow_baseline: true (or false). Shadow calls cost
real API money and are never counted as Fusion cost. Any shadow failure degrades to a warning; the
main run always succeeds. Internals: ARCHITECTURE.md.
View cumulative results:
uv run fusion stats # spend, savings, shadow win-rate
uv run fusion stats --json # machine-readable
Inside Claude Code, call fusion_stats. Every run's display_markdown also ends with a lifetime
footer (see COSTS.md).
History¶
An earlier isolated agent harness (an Opus agent versus a Fusion-planned agent in workspace copies) was removed in 0.2.0. Its only real run, in June 2026, was inconclusive because both arms hit the step cap without writing files; the record is kept in evals/archive/2026-06-30-agent-harness-benchmark.md.