Claude Code A/B: with and without Fusion¶
Canonical for the Claude Code A/B procedure: the automated end-to-end harness (fusion bench e2e)
and the manual comparison on your own tasks. Benchmark methodology in general (datasets, scoring,
statistics, reports) is in BENCHMARKING.md.
The question is the one that matters for cost: inside a real Claude Code session, with the host model's own tokens counted, does a cheaper executor that consults Fusion solve as many tasks as a frontier executor, for less? Fusion never edits files: Claude Code stays the executor in every arm.
Automated: fusion bench e2e¶
Each run is headless Claude Code (claude -p) on one executable coding task of a dataset (default
v2: every coding task with hidden tests, synthetic and real-repo), in a sandboxed temporary copy
of the task's repo. The hidden tests decide whether it solved the task.
| Arm | Executor | Fusion |
|---|---|---|
opus, sonnet, haiku |
Claude Code on Opus 5.5, Sonnet 5.5 or Haiku 5.5 (the catalog's model ids) | off: no MCP server |
opus+fusion, sonnet+fusion, haiku+fusion |
the same | on: Fusion's MCP server and plugin |
The haiku+fusion and sonnet+fusion arms (a cheap executor with a cheap advisor) are the cost
play; opus is the baseline.
Commands¶
# Free and deterministic: a simulated `claude` replays seeded runs (no API keys, no network).
uv run fusion bench e2e --mock --limit 4
# Live, within a cap you state (costs money: host tokens plus Fusion's panels).
uv run fusion bench e2e --limit 6 --executors opus,haiku --max-usd 3
# Continue a stopped run (finished runs are skipped, errors are tried again).
uv run fusion bench e2e --run-id bench-20261119-101500-ab12 --max-usd 4
# Read the results: the usual reports, with a "Claude Code end to end" section.
uv run fusion bench show RUN_ID
uv run fusion bench report RUN_ID --format html
| Option | Default | Meaning |
|---|---|---|
--dataset |
v2 |
Any dataset; only its executable coding tasks are used |
--split, --limit, --seed |
dev, all, 0 |
Which tasks (a seeded sample spread evenly) |
--executors |
opus,sonnet,haiku |
Executors to run |
--fusion |
both |
both, off or on |
--repeats |
1 |
Runs of each (task, arm) |
--max-usd |
required live | The run's cap |
--run-budget-usd |
1.0 |
Claude Code's own --max-budget-usd for each run |
--fusion-allowance-usd |
0.25 |
Reserved for Fusion's calls in each Fusion run |
--spend-stop-usd |
the ledger's cap | Stop before cumulative live spend would pass this |
--timeout |
900 |
Seconds before a run is killed |
--concurrency |
1 |
Runs at once |
--plugin-dir |
this checkout's plugin/ |
The plugin the Fusion arms load |
--claude |
claude |
The executable |
Before a live run¶
- Claude Code on
PATH(claude --version; the version is stored with the run), new enough to know every executor's model id. A build that predates a model printsunrecognized_model, prices it at a default rate and so stops early on its own--max-budget-usd; the item then carries a warning. Claude Code 2.1.287 knows Opus 5.5 and Sonnet 5.5 but not Haiku 5.5; 2.1.327 knows all three (claude update). ANTHROPIC_API_KEY: runs use--bare, which reads no login, keychain, hooks,CLAUDE.mdor installed plugins, so every arm starts from the same clean setup and the host cost is billed to that key.- Provider keys for the Fusion server (
.env, see CONFIGURATION.md); they are handed to the server only, never to Claude Code's environment. - A share of the live budget:
fusion bench spendshows what is left. Each run reserves its worst case (--run-budget-usd, plus--fusion-allowance-usdfor a Fusion arm) against--max-usdand the spend ledger before it starts, and the run stops cleanly, resumable, when the next one would not fit. Its measured cost (host plus Fusion) is appended to the ledger afterwards.
What one run does¶
- Copies the task's files into a fresh temporary directory (
bench/sandbox.py) and commits them to a throwaway git repository there, sogit diffand Fusion'sdiff_refwork. The task's source is only read. - Starts
claude -p "<task>" --output-format stream-json --model <id> --max-budget-usd <cap> --bare --permission-mode acceptEdits --permission-prompts none --settings <file> --strict-mcp-config --mcp-config <file>in that copy, withHOMEandCLAUDE_CONFIG_DIRinside it and a scrubbed environment. A Fusion arm also gets--plugin-dir(a copy of the plugin without its.mcp.json),--allowedTools mcp__fusion, and one appended system-prompt paragraph telling it which Fusion tools to use when. The task prompt itself is the same in every arm: the task's text, asking for edits in place. - In a Fusion arm, the MCP config starts this checkout's server (
python -m fusion.cli.main mcp) with its project root on the copy and its run database inside the sandbox. - Reads the result: Claude Code's final
resultevent (cost, usage, turns), everytool_usein the transcript, and the Fusion server's ledger (runs.db: one row per Fusion run, with its cost). - Diffs the edited copy against the task and runs the hidden tests on that diff in a second, network-less sandbox, exactly as a patch from any bench arm is judged (Coding tasks and the sandbox). Then deletes the copy.
Isolation¶
| Control | How |
|---|---|
| Task source | Copied, never written; a test hashes the dataset before and after a run |
| Bash commands | Claude Code's own sandbox (sandbox.enabled, failIfUnavailable, no unsandboxed retry), with an empty strict network allow-list: the task's code gets no network |
| Web tools | WebFetch and WebSearch denied |
| File edits | acceptEdits covers the working directory only; anything else needs a permission prompt, and --permission-prompts none answers every prompt "no" |
| Secrets | The run's MCP config (with the provider keys) sits beside the copy, in a directory Claude Code's file tools and sandbox are denied |
| Limits | A wall-clock timeout that kills the process group, Claude Code's --max-budget-usd |
The claude process itself is not under Fusion's sandbox-exec: Claude Code needs the network for
its own API calls, and on macOS its Bash sandbox cannot start inside another Seatbelt profile. The
guarantees above therefore rest on Claude Code's sandbox and permission settings. See
SECURITY.md.
What is measured¶
Stored with every item (BenchItem.e2e; the bench's own metrics hold the same in its terms):
| Field | Source |
|---|---|
tests_passed / tests_expected, solved |
The task's hidden tests on the edited copy |
wall_seconds |
Wall time of the claude process (claude_seconds: Claude Code's own figure) |
host_cost_usd |
Claude Code's per-model token counts (modelUsage in its JSON) at the catalog's prices; a model the catalog lacks counts at Claude Code's own costUSD (the item says which). Unknown when neither applies, and the report marks such totals as a lower bound |
host_cost_reported_usd, host_usage |
Claude Code's own total_cost_usd and modelUsage, kept for comparison: for a model it does not know, its cost uses a default price |
fusion_cost_usd, fusion_calls |
The Fusion server's ledger in the sandbox |
total_cost_usd |
Host plus Fusion |
turns, tool_calls, fusion_tool_calls |
num_turns; tool_use blocks in the transcript |
input_tokens, output_tokens, cache_read_tokens |
Claude Code's usage (input counts cache reads and writes) |
files_changed, patch |
The diff of the edited copy |
Claude Code's raw output of each run is kept beside the results, in
bench-results/<run>/evidence/e2e/<job key>.jsonl (and .stderr.txt).
An item is completed when Claude Code finished, halted when it stopped on its own limit, its
budget or the timeout (the tests still judge what it left), and error when it produced no result
or a Fusion arm's server did not connect (such a run is not a Fusion run; --run-id tries errors
again).
Simulated runs (--mock)¶
--mock replaces claude with python -m fusion.bench.e2e.fake_claude, which replays a seeded
recording per (task, arm): the executor's invented success rate decides whether it writes the task's
reference fix, one of its known flaws, or nothing; tokens are priced from the catalog; Fusion arms
record Fusion runs in the sandboxed ledger. Everything else (the copy, the hidden tests, the store,
the reports) is the real code path. The numbers test the harness and say nothing about real models.
Manual: your own tasks¶
To compare on a task from your own repository instead of a dataset:
- Start from the same clean repository state for both arms, with the same
task.md(the exact prompt and acceptance criteria). - Arm A: Claude Code on Opus, told not to call Fusion. Arm B: Claude Code on a cheaper
model with Fusion installed (INTEGRATIONS.md), told to call
fusion_askorfusion_planbefore editing,fusion_debug_errorafter a failing test, andfusion_review_diffbefore finishing. Let both edit, run tests and iterate normally;/fusion:abdoes this inside one session. - Record for each arm: the final answer, the tests run and their results, the whole wall-clock
time, and the cost (Claude Code's usage; Fusion's from
fusion runs show RUN_ID). Leave a cost unknown rather than guess it. - Compare with the MCP tool
fusion_compare_claude_runs, or from files:
uv run fusion compare-claude-runs \
--task-file task.md --opus-file opus-output.md --fusion-file fusion-output.md \
--context-file verification.md \
--opus-cost 0.42 --fusion-cost 0.07 --opus-latency-ms 90000 --fusion-latency-ms 45000
It returns a judged quality winner and the cheaper and faster arm (unknown where a value is
missing). Prefer passing tests over a judge's opinion, and do not count a cheaper arm as better
when its tests fail. One task proves little.