Skip to content

Architecture

Fusion Code Orchestrator is a Python MCP server that gives Claude Code model-like multi-model workflows for coding tasks. It calls providers directly through adapters.

Canonical for how Fusion works inside: the pipeline stages and data flow, strategies and modes, the cost ledger, concurrency and the module layout. Settings are in CONFIGURATION.md and the security model in ../SECURITY.md.

Pipeline flow

A run is a list of stages that read and write one RunState (src/fusion/orchestration/):

Claude Code -> MCP tool -> specialized pipeline -> BasePipeline.run(ctx)
   RedactStage       redact secrets, open the run record
   DifficultyStage   strategy auto only: estimate difficulty, pick the start rung, run its strategy
   RouteStage        apply the strategy: panel/aggregator/judge models, fall back by catalog role
   ContextEvalStage  score the context            -- halts: "insufficient context"
   BudgetStage       forecast the cost against max_cost_usd; shift the strategy down if it is over
                                                  -- halts: cap below the cheapest option
   ShadowStartStage  start the shadow baseline call now, if a shadow run is wanted
   PanelStage        concurrent fan-out           -- halts: quorum not met
                     (a cascade asks its cheapest members first and may stop there)
   RefineStage       the strategy's extra peer-review rounds (rounds - 1; none for rounds: 1),
                     skipped when the panel already agrees or a cascade already stopped
   ClaimsStage       read answers as claims, cluster them across models, measure agreement
   EscalateStage     strategy auto only: check the rung's answer, aggregate it, and climb to the
                     next rung when a trigger fires (also runs after a lost quorum)
   ConcurrentStages  at the same time (skipped when EscalateStage already aggregated):
     JudgeStage        safety checks and claim-derived scores per answer; LLM judge if on
     AggregateStage    final answer (solo: the answer itself; llm: one synthesizer call;
                       digest, vote, best_of, union: built from the claims, no call;
                       union+check: union plus one checker call; select, select+patch: one
                       small call that picks the best answer; auto is settled per task first)
   FinalEvalStage    final eval, structured output, budget warnings
   ShadowStage       collect the baseline answer and judge it blind against Fusion's
   PersistStage      build the result from the ledger, store the run (always runs)

The stage list is the same for every strategy; each stage reads the strategy from RunState and skips what it does not call for (a solo run has no refinement round and no synthesis call). A halted run skips straight to PersistStage, which still stores a diagnostic result. Stages hold no state of their own, so each can be unit-tested with a prepared RunState. The modules are context (inputs, shared dependencies, RunState), strategy (strategies and run modes), stages, ledger, result, output (display text, usage and persisted views), pipeline (the runner), specialized (the six task pipelines that map results to tool outputs) and factory (build_pipelines(Settings, providers)). pipelines.py only re-exports; create_pipeline(s) are deprecated aliases.

Cost ledger

Every LLM call goes through one CallGateway (ledger.py), which times it, prices it from the catalog and appends a CallRecord to the run's RunLedger: stage (panel, refine, judge, synthesis, shadow_baseline, shadow_judge, ...), model, tokens (input, output, cached, reasoning), cost and whether it is known, latency, start offset, retries and speed metrics. Cost, token, latency and per-stage totals are all derived from the ledger, so judge and eval calls are part of the reported Fusion cost; shadow stages are excluded because they are measurement overhead. A call that fails or times out may still have been billed, so its cost is recorded as unknown and the run's total is flagged unknown. RunLedger.task_metrics() gives the per-task numbers the benchmark mode will store. The full ledger is saved in each run's output.

The gateway also keeps prompts inside the model's context window: if a prompt exceeds the catalog context_window it is trimmed in the middle and a warning naming the model is added to the run. Models with no declared window are never trimmed.

Strategies and run modes

A strategy (strategy.py, packaged in config/strategies.yaml) is data: kind (solo, panel or cascade), members (catalog aliases, each with an optional role, temperature and reasoning effort), rounds, aggregator, judge and optional cost/latency caps. RunState.start resolves it once per run, from PipelineContext.strategy or else the legacy budget through budget_strategies, so an unknown name fails before a run is recorded. The router turns the strategy into the models to call, and the stages do the rest. Benchmark baselines are solo strategies, so a baseline and Fusion share one code path and one ledger. The fields are documented in CONFIGURATION.md.

A run also has a mode (Mode.REAL or Mode.BENCHMARK), passed as Pipeline.run(ctx, mode=...) and kept on RunState and PipelineResult. MODE_SETTINGS holds what a mode changes: whether the lifetime footer and shadow A/B are allowed, whether prompts are trimmed to the context window, whether secrets are redacted, whether calls stream, and a fixed temperature and seed. CallGateway applies the sampling and streaming defaults for every call of the run (a request or member that sets its own temperature wins) and, unless the run turned redaction off, removes secrets from every outbound request, so the prompts that quote model output are covered as well as the task (SECURITY.md). run() also takes seed, redact and ledger for one run, and a halted run's PipelineResult.halt_reason says why it stopped. The table is in CONFIGURATION.md.

Offline and live

Offline (mock) and live mode differ in configuration, not in code paths. factory.build_deps picks the mode once (Settings.use_mock, or FUSION_DEFAULT_PROVIDER=mock): live mode builds a registry of real models and the packaged strategies; offline mode builds a registry of mock models and strategies of the same shapes running on them (mock_strategy_book). Router, stages and fallbacks never ask whether they are under test; role fallbacks (panel, judge, synthesizer) come from the catalog roles of whichever registry is in use. A panel model's persona prompt comes from its catalog persona field; models without one answer with the task's own system prompt.

src/fusion/orchestration/prompts.py builds every prompt; the parts fusion tune may edit (the rules, the claim format, the select criteria, the self-check and the role prompts) come from the prompt pack (config/prompt_pack.py, files in config/prompts/; see CONFIGURATION.md), read through active_pack(), which a tuning probe switches with use_prompt_pack for the length of one bench run.

Components

MCP server

src/fusion/mcp_server/server.py registers the public tools:

  • fusion_review_diff
  • fusion_ask
  • fusion_debug_error
  • fusion_plan (kind: feature | decision; the 0.2.0 names fusion_plan_feature and fusion_decide_architecture stay callable but unlisted, see below)
  • fusion_eval_answer
  • fusion_compare_claude_runs
  • fusion_stats
  • fusion_digest
  • fusion_propose_patch and fusion_select_patch
  • fusion_feedback

src/fusion/mcp_server/tools.py converts MCP input schemas into orchestration pipeline inputs and returns the pipelines' output models as dictionaries (FusionTools, also used directly by tests and the CLI). response.py shapes those into what the server returns: compact text plus a typed record by default, everything with detail: full. schemas.py holds the inputs (one shared base, with the old context names folded into context) and the typed outputs the tools declare as outputSchema. Before a pipeline runs, gather_workspace in tools.py reads what the call names in paths, globs and diff_ref through fusion.workspace.WorkspaceReader (reader.py for files and globs, git.py for read-only git), under the root the server took from the client's roots/list, FUSION_READ_ROOT or its cwd. The text joins file_snippets (or becomes the review's diff), and the reader's WorkspaceReport (read, skipped with reasons, bytes, host_tokens_saved_estimate) travels on PipelineContext.workspace into the stored run input and the response. fusion_digest is the ask pipeline with a short-answer instruction and a tighter compact cap. server.py adds tool annotations and descriptions, three resources (fusion://runs/{id}, fusion://stats, fusion://strategies), three prompts, and the stdio and HTTP transports. The tool contract is in INTEGRATIONS.md.

Progress, cancellation and the soft limit. orchestration/progress.py holds a per-request sink in a ContextVar; the server installs one that calls Context.report_progress, and the pipeline calls progress.report as each stage starts (a stage's label) and as each panel member finishes. Without a sink it does nothing. A cancelled request raises CancelledError through the stages: CallGateway.call records the interrupted call on the ledger and re-raises, and the fan-out and refinement cancel the tasks they started before re-raising, so no provider call outlives the request. BasePipeline.soft_timeout_s (real mode only; the server sets it from FUSION_TOOL_SOFT_TIMEOUT_S) wraps the panel, refinement and judge/synthesis stages in one deadline; when it passes, SoftTimeoutRecovery finishes the run without another model call from what the panel produced (claims, deterministic scores, the digest) or halts with timeout when there were no answers.

The command line. src/fusion/cli/ is split by job: main.py (the root app, --set, --verbose, the error contract), run_cmds.py (ask, review-diff, debug, decide, plan, digest, patch, eval-answer), runs_cmds.py, config_cmds.py, doctor.py, models_cmd.py, with common.py (exit codes, shared flags, input reading) and live.py (the live view). app.py only re-exports app. The live view listens on the progress channel above plus a second one, progress.call_observer: the call gateway tells it when each model call starts and when its record is on the ledger, so a display never reads the ledger itself and a failing display never breaks a run.

Installers and the plugin

src/fusion/install/ holds fusion install <client>. common.py has the server command (uvx --from git+<repo> fusion mcp, or uv run --directory PATH for a clone), an atomic merge into an mcpServers JSON file that never touches other servers or keys and refuses malformed input, and the launch check (start the server on the mock provider, list its tools). claude_code.py registers the server with claude mcp add (user scope), merges .mcp.json (project scope) or installs the plugin through claude plugin; it only reads Claude Code's own config. Every install is idempotent and has a dry run. The plugin lives in plugin/ (manifest in .claude-plugin/, .mcp.json, commands, skills, the fusion-advisor subagent), and .claude-plugin/marketplace.json at the repo root lists it; tests keep the two server definitions, the tool names the components mention and Claude Code's own claude plugin validate in agreement. Usage is in INTEGRATIONS.md.

Providers

Providers implement ModelProvider in src/fusion/providers/base.py. The three cloud adapters and the two local ones share HttpProvider (providers/http_utils.py), which owns:

  • One pooled httpx.AsyncClient per provider, created lazily (HTTP/2 for cloud providers) and re-created if the event loop changes. aclose() releases it; FusionTools.aclose() and the MCP server lifespan close every provider on shutdown, and the CLI closes them after each command.
  • Retries: only transient failures (429, 408, 5xx, timeouts, connection errors) are retried, with full-jitter exponential backoff. Retry-After is honoured, and one longer than 60 s is returned as an error instead of waited for. 4xx errors are never retried.
  • A typed error taxonomy surfaced as ModelResponse.error_type: RateLimit, Auth, Timeout, BadRequest, Server, Connection. Error text is scrubbed of API keys; keys travel in headers, never in URLs.
  • Rate limiting through ProviderLimiter (providers/limits.py): a semaphore for in-flight requests plus a sliding one-minute request window, shared by all calls to a provider. Limits are configured in the catalog (CONFIGURATION.md).

Per-model behaviour comes from the catalog entry (supports_sampling_params, supports_json_schema, supports_reasoning_effort, supports_vision), not from model-name regexes. supports_sampling_params defaults to false, so temperature and seed reach a model only when its entry says so (none of the packaged ones); a model missing from the catalog gets none. The Anthropic adapter sends anthropic-version: 2026-10-21, which removed temperature, top_p and top_k, and never sends them. ModelRequest carries the optional features; an adapter maps each to its provider's native form:

ModelRequest field Anthropic OpenAI / LM Studio Gemini
response_schema output_config.format (JSON Schema) response_format: json_schema responseSchema (refs inlined)
reasoning_effort output_config.effort reasoning_effort thinkingConfig.thinkingLevel
cache_prefix cache_control on the system prompt and every message before the last automatic automatic
images base64 image blocks image_url data URIs inlineData parts
stream SSE messages stream SSE with include_usage streamGenerateContent?alt=sse

When a model does not support native structured output, the schema is appended to the system prompt and the JSON is scraped from the reply. Forced tool choice is not used for structured output because Claude 5.x rejects it. thinking_budget_tokens is only valid for models without adaptive thinking and is refused locally otherwise.

Adapters normalize usage so costs are computed from one definition: input_tokens is the total including cached reads and cache writes, output_tokens includes reasoning tokens. Anthropic reports cache reads and writes outside input_tokens and Gemini reports thinking tokens outside candidatesTokenCount, so the adapters add them back; OpenAI already includes both. Cache writes are priced at the catalog's cache_write_per_1m.

ModelResponse also carries speed metrics measured at the source: total_tokens_per_s (output tokens over wall latency), and for streamed calls ttft_ms (first non-empty text delta) and decode_tokens_per_s (output tokens over latency after the first token). Token counts always come from provider usage, never from character counts. Local providers do not stream.

Cloud providers are Anthropic, OpenAI, and Google. Ollama and LM Studio support local-only routes. MockProvider supports deterministic offline tests and development.

Router

src/fusion/routing/policy.py classifies task type, complexity, and risk, then turns the run's strategy into:

  • panel models (the strategy's enabled members; if none is enabled, the catalog's panel-role models with a warning);
  • the judge model (the task policy's judge_model, else a JSON-capable model with the judge role);
  • the aggregator model (llm, union+check, select and select+patch only; empty for solo, digest, vote, best_of and union);
  • the resolved strategy name, an estimated cost tier and routing warnings.

RoutingDecision.strategy reports the strategy that ran (for strategy: auto, the strategy of the rung that answered; RoutingDecision.escalation says which rung and why). Model metadata and prices live in the catalog, src/fusion/config/catalog.yaml; per-task judge and context thresholds and fan-out settings in src/fusion/config/routing_policies.yaml; strategies in src/fusion/config/strategies.yaml.

Concurrency and latency

A run's wall time is its critical path, not the sum of its calls. What overlaps:

t0 ─┬─ panel member A ──────────┐
    ├─ panel member B ────┐     │  (every member starts at once; stragglers may be cut off or hedged)
    ├─ panel member C ───────┐  │
    └─ shadow baseline ─────────────────────────┐   (started at t0, collected at the end)
                             └──┴─ refine A/B/C ─┐ (all start once every round-1 answer exists;
                                                 │  skipped if the panel already agrees)
                                                 ├─ judge x N ──────┐  (concurrent with each other
                                                 └─ synthesis ──────┴─ final eval ─ shadow judge
                                                                       and with synthesis)
  • Panel (fanout.py): every call starts at once. max_concurrency caps in-flight calls per provider, so a provider that is slow or rate limited cannot hold up the others; the provider's own limiter (provider_limits) enforces its quota inside the provider. The per-model timeout starts once the call has a fan-out slot. A cancelled call is recorded in the ledger with its cost unknown, because the provider may still bill it. The latency path (all on by default):
  • Early return stops waiting once quorum answers are in (never fewer than min_successful_responses) and cancels the stragglers after a grace. With grace_ms: auto the grace comes from the catalog's measured speeds (observed_ttft_ms, observed_decode_tps): a straggler is expected after its time to first token plus, at its decode rate, as many tokens as the members who answered wrote; the run waits for the next expected straggler only if it is due within max_grace_ms. The grace used is FanoutResult.grace_ms.
  • Hedging: a member still silent after hedge_after_ms is also asked of a spare model, and whichever answers first stands in for it.
  • Same-tier fallback: a member whose call fails with NotFound (404/410, the model is unknown or retired), Server (5xx after the adapter's retries), RateLimit or Connection is asked of a spare at once, so one failing model does not cost the quorum (FanoutResult.fallbacks, and a warning). Other failures (bad request, auth, timeout) are not retried elsewhere.
  • Spares (catalog.same_tier_spares, for hedges and fallbacks): enabled panel-role models of the failed or slow model's price tier (local and mock models only stand in for each other), never the same model under another alias, never another member of the strategy (the avoid list, so a cascade's second wave is not asked twice); another vendor first, then the fastest by measured speed, then catalog order. A stand-in uses its own catalog settings.
  • One prompt prefix (prompts.compose_user_prompt): every member gets the task type's system prompt and the same user prompt, which puts the file snippets first and the request, context and output rules after them; a member's role is appended at the very end. The length of the file part travels as ModelRequest.shared_prefix_chars, so a later call about the same files starts with the same text. OpenAI and Google cache such prefixes by themselves; for Anthropic the fan-out sets cache_prefix (fanout.prompt_cache: auto, only when files were given) and the adapter puts a cache_control breakpoint after the file part; the OpenRouter adapter does the same for anthropic/ models, which cache only at breakpoints there too. Calls that run at the same time cannot share a cache entry (it exists once the first response starts), so the reuse is across calls, not across members of one fan-out. Cache reads are CallRecord.cached_tokens, priced at the catalog's cached rate, and summed per stage and in TaskMetrics.
  • Refinement: each member's revision needs every peer's round-1 answer, so a round starts when the slowest member is in, and then all revisions run together. A round is skipped when the panel's agreement already reaches refinement.skip_above_agreement (never on a lensed run, see Lenses and the union).
  • Judge and synthesis: per-answer judge calls run together, and the whole judge stage runs alongside synthesis (ConcurrentStages) unless the strategy sets judge_feeds_synthesis, in which case synthesis waits and receives the judge's scores. If one stage fails, the other is cancelled and the original exception is raised. The synthesis call is streamed (its time to first token and decode rate are recorded; the MCP response still arrives whole), and a strategy with skip_aggregator makes no synthesis call at all when the panel already agrees (see Cascade, aggregators and caps).
  • Speed-ranked choices: a strategy with speed: fast ranks its members by measured speed for a typical answer (cascade.fastest_first; never-measured models last), so a cascade's first wave and a panel cut by max_models are its fastest members; balanced keeps the configured order and a cascade's cheapest-first wave.
  • Shadow baseline: it needs only the task prompt, so ShadowStartStage starts it right after the context check and ShadowStage collects it after Fusion's answer is final. Fusion's wall time is stamped before the wait and shadow calls are excluded from Fusion's cost and critical path. A run that halts or fails cancels the baseline call.
  • Timeline: every CallRecord has started_at_ms on the run's clock, and RunLedger.timeline() returns each call as a start and end span, so overlap can be read (and tested) from the ledger.

Config keys and failure semantics are in CONFIGURATION.md. A panel call that is still pending when the global timeout hits is attributed to its own model. The MCP request is still a normal blocking request from Claude Code's perspective.

Lenses and the union

Panel diversity comes from lenses (lenses.py, config in lenses.yaml), not from sampling temperature, which current models reject. The idea is Harness²'s orthogonal dimensions: on a task whose answer is a list of findings, each member covers one area and leaves the rest to the others, so the answers are pieces of one result.

  • Seats (RouteStage._seats): a model listed more than once becomes alias, alias#2, ... (lenses.seated); ModelRegistry.add_seat registers each copy as a roleless clone of the catalog entry, so pricing (by model ID), spares (never the same model under another alias), the ledger and fan-out need no change. A model gets at most as many seats as the task type has lenses.
  • Assignment (PanelStage._assign, after BudgetStage may have shifted the panel down): assign_lenses(task_type, members) returns {member: Lens}; a member holding several lenses gets one merged Lens (a+b), and a seat that would repeat its model's lens is dropped from the run. A cascade and a solo run get no lenses (rejected at load).
  • Fan-out (fanout.py): each member's prompt is the shared prompt (unchanged, so the provider prompt cache still works) plus its role and prompts.lens_instruction: its area, what it ignores, the other members' areas, and that none_found is a complete answer. A lens replaces the catalog persona. A hedge or same-tier fallback asked for a member gets the member's lens; FanoutResult.answered_for maps each answering model to its member, so RunState.answer_lenses knows every answer's lens. A lensed fan-out has early return switched off (PanelStage. _fanout_config): members are not interchangeable.
  • Agreement (claims.agreement_score(..., lenses=)): a single-model cluster whose model is the only holder of its lens is lens_exclusive: left out of score and of outliers. Refinement prompts keep a lensed member in its area, and a round is never skipped for agreement on a lensed run (agreement there covers only the overlap). aggregate.aggregator_for(..., lensed=True) and AggregateStage._skip turn a vote or best_of into union; an llm synthesizer is told the responses are pieces (build_synthesis_prompt(lensed=True)).
  • Union (union.py, no model call): merge_findings reuses the run's claim clusters as the duplicate merge, then check_claim judges each member claim against Material (the redacted input, indexed: normalised text, and per file the line numbers shown by diff hunks, snippets and stack-trace frames). A cluster is backed if any member's quote is in the input or its file:line is shown, unbacked (dropped) if every member's quote or file is absent, else unchecked. Recommendations and tests are kept apart as advice. union+check sends the unchecked findings to one cheap call (check_unchecked, stage synthesis, 400 output tokens) whose confirmations must carry a quote that passes the same match. FinalEvalStage builds the structured fields from UnionResult.kept_clusters(), so dropped points do not come back, and the run's disagreement holds lenses, lens_exclusive, none_found and the union's counts.
  • Forecasts add LENS_PROMPT_TOKENS to each lensed panel and refinement call.

Approaches and the select aggregator

Lenses suit answers that are lists of findings that add up. Where every member gives a complete answer to the whole task (an implementation plan, an architecture decision, a fusion_ask answer, a patch) the answers are rivals, and one is picked instead of a synthesizer rewriting them (approaches.py, select.py, config in approaches.yaml).

  • Approaches. An Approach is a stance a member answers the whole task from (plans: smallest change, cleanest design, safest rollout; decisions: each option argued by its strongest advocate; asks: direct, grounded, thorough; patches: minimal fix, root-cause fix). assign_approaches works like assign_lenses (a member that names one keeps it, the rest take the others in order, a model listed twice runs as seats and a seat that would repeat its approach is dropped), and a task that expects a patch uses the patch approaches whatever its type. The member's prompt gets prompts.approach_instruction after the shared prefix (it replaces the catalog persona, as a lens does; a hedge or fallback keeps the member's approach); RunState.answer_approaches and the run's disagreement.approaches record who answered from what. Lenses and select never combine: a lens makes an answer cover one area, so no whole answer would be left to pick (rejected at load).
  • Selection (AggregateStage._select). select.anonymise labels the readable answers c1..cN in an order shuffled by a hash of the run id (stable for one run, no position tell) and scrubs the panel's aliases and provider model ids from the text. prompts.build_select_prompt ports Harness²'s select(): delivery, conformance, fidelity against the supplied material (reject invented specifics; where the material is silent prefer the candidate that says so), internal consistency, operational fitness, finish; length and polish are not merit. The call (stage synthesis, role selector, SELECT_OUTPUT_TOKENS = 400, a JSON schema) returns why (at most 6 lines), choice and runner_up; the selector never sees a model, a vendor or an approach. parse_selection also reads prose ending in a fenced choice block. The chosen answer is returned as written, and FinalEvalStage builds the structured fields from that answer's clusters. select+patch also takes up to three one-sentence points (each must name another candidate) that the chosen answer lacks and appends them after it, with no rewrite.
  • Fallback. A failed call, a choice that is not an id, a cost or latency cap with no room, or no selector model falls back, with a warning, to a vote (the patch vote for a patch task) when a majority backed something and to best_of otherwise; disagreement.selection.fallback says which. A single surviving answer is returned without a call.
  • aggregator: auto (the schema default; the packaged strategies name theirs) is settled in RunState.start by aggregate.resolve_aggregator: default_aggregator(task_type, patch=) gives union (with lenses) for review and debugging and select+patch (with approaches) for plans, decisions, asks and patches; answer_eval keeps llm. Every later stage sees a concrete strategy; the forecast of an unresolved one is its dearest form (a synthesis).
  • Cost. budget_guard.select_calls plans the call: the task and every answer in, at most 400 tokens out, about a tenth of a synthesis. Both kinds of call are in the ledger under synthesis with their roles, so the 0.3.0 study can compare them. Forecasts add APPROACH_PROMPT_TOKENS to each member of a select panel.

Verified generation: checks and the cross-check

aggregator: verified-synth (strategy verified-synth) picks a patch by running code instead of asking a model (src/fusion/verify/). It runs in benchmark mode and under fusion_propose_patch, the only callers that hand the run a cross-checker (PipelineContext.crosscheck); anywhere else the run is refused (strategy.may_run_code). Given a cross-checker, aggregator: auto settles to verified-synth for a panel whose answer is a patch (select+patch otherwise).

  • What members write (verify/synthesis.py). It is a select aggregator, so members get the patch approaches (minimal fix, root-cause fix) and each answers with a complete patch. The fan-out adds checks_instruction and asks for panel_answer_schema(checks=True): one to three CheckScripts (python or shell; repro and property fail on the code as it is and pass once fixed, invariant passes before and after) and up to five prose guardrails. PanelAnswer.checks and .guardrails hold them; other runs' schemas leave both fields out. A strategy with rounds > 1 is refused, because refinement would rewrite answers without checks.
  • Validation before anything runs (verify/validate.py). validate_check drops a check that is too big (8 KB, 200 lines), has an unsafe name, does not compile (Python compiled in memory; shell through bash -n), names the network (network modules, __import__/importlib/exec, curl/wget/nc, /dev/tcp) or writes to an absolute, ~ or .. path. These are heuristics; the sandbox is the boundary.
  • The cross-check (verify/crosscheck.py, crosscheck(candidates, CheckEnv)). The source is read once into memory and every run gets a fresh SandboxCopy (a bench.sandbox.Sandbox holding the copy, the patch applied, deleted on exit); check scripts go beside the copy (Sandbox.write_aux), not in it. Each valid check first runs on the unpatched copy: a repro or property check that passes is discarded (not_discriminating), an invariant that does not pass is discarded (broken). Candidates whose patches produce the same files form one group and run once; each group's patch runs against every kept check plus the task's own checks (CheckEnv.task_checks: the visible tests, trusted, never discarded). A cell is pass, fail, error (patch did not apply, could not start, killed by a signal) or timeout.
  • The pick (pick(Matrix)). A candidate scores 1 for each check it passes that another group or the task wrote, and OWN_WEIGHT = 0.5 for each check its own group wrote (CodeT's dual agreement, with authorship kept apart): two members agreeing on a wrong patch cannot outvote a patch that passes everyone's checks. A candidate whose patch did not apply cannot win. The best score wins (AggregateStage._verified_synth returns its answer as written, with no model call); a tie between groups goes to the select call among the tied candidates only.
  • The record. CrossCheckReport.record() (winner or tie, reason, scores, shares, groups, every check with its status, reasons and baseline, the matrix, guardrails, isolation) is stored as disagreement.crosscheck and as BenchItem.verification; disagreement.verified holds each candidate's share of kept checks passed, which the escalation ladder's patches_fail trigger reads. Orchestration never imports the sandbox: the bench passes PipelineContext.crosscheck (bench.scoring.coding.coding_crosschecker), as it passes verifier to verified, and fusion_propose_patch passes one of its own (below). A run with a cross-checker is never cached.
  • fusion_propose_patch (verify/propose.py, mcp_server/patch.py). The tool runs the generic patch pipeline (no soft time limit: the checks run after the panel) on the task plus PATCH_REQUEST, with expects_patch and a cross-checker chosen by verify. sandbox: WorkspaceReader.copy_project reads the project once into a ProjectCopy (the reader's refusals: nothing ignored, secret, under .git or a symlink; raw bytes, never sent to a model) and sandbox_crosschecker runs crosscheck on it with isolation require, the caller's tests_cmd joining as a trusted task check. host: host_crosschecker runs nothing (host_report validates checks, normalises and groups patches, and keeps a handoff of the candidates and check scripts in the record); fusion_select_patch later scores the host's results with the same pick (select_from_results, no model call). none: no cross-checker, so panel-nano-select selects. Host and none read only the files a patch touches (ProjectFiles). Whatever the mode, the chosen patch is returned through normalize_patch: a unified diff against the project as it is, which git apply takes. choose_mode falls back from sandbox to host when verify.sandbox_enabled is false, no isolation works, or the project is over the copy limits.
  • Evidence boundary. Hidden tests, the reference solution and its flaws, and rubrics never enter a prompt that proposes an answer: synthesis.grader_material lists their telling lines and assert_no_grader_material fails on a prompt that contains one (a test runs a simulated bench and checks every panel prompt). Check results go to no model except, on a tie, the selector, which sees only the tied candidates' answers.

Experience library

src/fusion/library/ keeps per-project lessons learned from the host's feedback on runs, ported from Harness²'s evidence-backed library. It is off unless library.enabled is true (CONFIGURATION.md; what is stored: SECURITY.md).

  • Store (store.py). A Lesson is {id, project_key, task_type, cue, lesson, evidence, verdict, confidence}: when a situation shaped like cue comes up, do lesson. Lessons, run_feedback and lesson_uses (which lessons each run's prompt carried) are tables of the run database (migration 6). project_key is the first 16 hex digits of the SHA-256 of the resolved project root; the path is not stored. FusionTools.library turns the library off for a call whose run database lies inside the project.
  • Distil (distil.py). fusion_feedback(run_id, outcome, note) records the feedback and makes one cheap call (library.distil_model, else the task type's judge) that sees the run's redacted request, its answer, the redacted note and this project's lessons for the task type, and returns at most one lesson or NO_ENTRY. The prompt carries Harness²'s rules: the cue is the shape of the situation, not this run's facts; the lesson names what to do; refine an existing lesson rather than copy it. An ignored outcome without a note teaches nothing and calls no model. The code then applies the evidence rules (apply_distillation): each run adds one evidence line (verdict: run X, answer outcome: note); a lesson's verdict is the one its non-confounded lines share, confounded when they conflict, and a lesson resting only on confounded evidence is not created, or is deleted when a refinement leaves it so. Confidence is capped at medium with one run behind it, high needs two that agree, conflicting evidence makes it low, and any confounded line lowers it a step. The call's cost is added to the run's record (library_distil step).
  • Retrieve (retrieve.py). Before a panel tool runs (ask, review, debug, decide, plan, digest, propose_patch), retrieve takes up to max_lessons lessons of the project and task type that share words with the request (cue words count double, times a confidence weight; no model call), and render_lessons renders them within max_prompt_tokens under a header that calls them hypotheses to re-check. The block reaches the panel prompt through PipelineContext.lessons, after the context and before the request, outside the files' shared prefix, so prompt caching is unchanged; it is part of the response-cache key and of the budget forecast. The shadow baseline, judge and aggregators do not see it. The run stores the lesson ids (lesson_ids in its input, lesson_uses), and the response lists them in lessons_used, so later feedback on that run is evidence about those lessons.

Difficulty routing and the escalation ladder

strategy: auto (the default until the 0.3.0 study chose panel-nano-pick) is a kind: auto strategy with no members of its own. It starts a request on the cheapest rung that its difficulty calls for and spends more only when the cheaper answer shows it is not enough (routing/difficulty.py, orchestration/escalate.py; settings in routing_policies.yaml under routing:).

  • Difficulty (DifficultyStage, after redaction). estimate_difficulty(request, facts) scales nine deterministic signals to 0..1: the task type's base difficulty, input size, files and diff hunks touched, languages (file extensions), stack-trace depth, the kind of error (a plain KeyError is easy, a race or a leak is hard), risky areas (concurrency, security, migrations, money, auth) and the question's shape (yes/no 0, lookup 0.25, design 1). Their weighted mean is the score; easy_below and hard_from cut it into easy, normal and hard. A score within ambiguity_margin of a threshold is ambiguous (confidence < 0.5); when classifier_model is set (it is off by default since the 0.3.0 calibration), one call to it (a nano model, at most 150 output tokens, ledger stage route, role difficulty) decides; a failed or unreadable call keeps the signals' level with a warning.
  • The ladder (EscalationPolicy): rung 0 solo-haiku-check (Haiku 5.5 alone, with self_check: true, which appends prompts.self_check_instruction), rung 1 panel-nano-auto (lenses and union for review and debugging, approaches and select+patch otherwise), rung 2 panel-mid, rung 3 solo-opus, a frontier: true rung used only with allow_frontier (strategy: auto-frontier, or the config). Easy starts at 0, normal at 1, hard at 2. The start moves down past a frontier rung that is not allowed, a rung that cannot run in this mode (a verified aggregator outside a benchmark), and a rung forecast over the run's max_cost_usd. A rung must be a solo or panel strategy (checked when RoutingPolicy is built).
  • Triggers (EscalateStage, after the claims). Before aggregation (early_verdict): a lost quorum; a solo answer whose self-reported confidence is below self_confidence_below (or that reported none); a panel without lenses or approaches whose agreement is below agreement_below (tau). A rung that fails these is never merged, so no synthesis or selection is paid for an answer that will not be returned. After aggregation (late_verdict): the selector set acceptable: false (no candidate good enough; it still names the least bad); union kept no finding the input backs where findings were expected (always for debugging, otherwise when a lens reported one); every candidate patch failed its checks (verified). Agreement is not a trigger on lensed or approach runs: those answers differ on purpose.
  • Climbing. The next rung's strategy is routed on a fresh copy of the run state (_fresh_rung). Models that already answered (by base alias, so seats count) are not asked again; their answers are handed over in RunState.carried, and PanelStage asks only the new members and merges the two fan-outs (cascade.merge_fanouts), carried answers first. A rung that needs at least one new answer and gets none loses its quorum. A rung with nobody new to ask is recorded as skipped and the next one is tried. A solo rung never takes carried answers (its answer is its own). Before a rung runs: the soft time limit (the rung's steps, each as long as the slowest call so far or a step of the rung below by the wall clock) and the cost cap (BudgetGuard.check on the rung's planned calls, money held until the rung has run) are checked, and max_rungs caps the climb. When the next rung cannot run, the latest rung that produced answers is aggregated and returned with a warning ("best answer so far").
  • The record. PipelineResult.escalation (EscalationOutcome, stored under escalation in the run's output) holds the estimate and its signals, the classifier's cost, the start rung and why, and per rung tried: who was asked, whose answers were reused, its own cost and wall time, the outcome (answered, escalated, stopped, skipped, interrupted) and the trigger with its reason. RoutingDecision.escalation carries one line ("answered at rung 1 (nano-panel): panel agreed (0.86)") that the compact response shows under Route and the tool result returns as escalation; the routing reasons start with one line per rung.
  • Outside a run. RoutingPolicy.ladder(strategy, task_type, content) lists the rung strategies a request may use (from its signals-only start); RoutingPolicy.route routes auto to the start rung. fusion bench forecasts an auto arm as its start rung plus the higher rungs at the escalation share (all of them for the worst case), keys its jobs by the ladder and the difficulty settings, and lists every rung's models in the run metadata.

Cascade, aggregators and caps

Three mechanisms cut what a run spends without a model call of their own. All of them read the claim clusters and the ledger, never an answer's wording.

  • Cascade (cascade.py, run by PanelStage). The members are sorted cheapest first by catalog list price and the first cascade.first form wave one. measure_claims (the measurement ClaimsStage also uses) and decide judge that wave: it may stop only if two or more models answered, agreement_score reaches the threshold, no cluster is contradicted and the router's risk is not high. Stopping sets RunState.cascade.exited_early; the panel is trimmed to wave one (so coverage is measured over the models actually asked), RefineStage does nothing and AggregateStage uses early_aggregator. Otherwise wave two runs the rest, merge_fanouts combines the waves (wall times add, the quorum is judged again over every model asked) and the run continues as a panel. A wave-one call that fails counts as "fewer than two answered", so a flaky cheap model escalates rather than answering alone. CascadeOutcome (reason, agreement, threshold, risk, who ran) is stored with the run.
  • Aggregators (aggregate.py). aggregator_for(strategy, cascade_exited_early, lensed) picks one; digest, vote and best_of are pure functions of the clusters and answers (union is in Lenses and the union, select in Approaches and the select aggregator). vote keeps the consensus clusters, best_of scores each model by the mean share of the other models backing its clusters (halved when disputed) and returns that model's answer as written, and digest lists consensus, disputed and single-model clusters ahead of every answer for Claude Code to merge. llm stays in synthesize.py. A strategy's skip_aggregator turns llm into its use aggregator (vote or best_of) for a run whose answers pass the cascade's own test (cascade.agreement_decision: two or more answers, agreement at least the threshold, nothing disputed, not high risk); the routing reasons say which way it went ("Synthesis skipped: ..." or "Synthesis ran: ..."). The default panel-duo uses it.
  • Caps (budget_guard.py, routing/budget.py). BudgetStage runs preflight: plan_calls lists the calls a strategy makes with assumed token counts (prompt size from the redacted text; fixed overheads and output sizes as constants in routing/budget.py), forecast_calls prices them from the catalog, and down_shifts yields cheaper strategies until one fits or none is left (a halt with reason budget). The result is stored as RunState.preflight. During the run a BudgetGuard on RunState is asked before each optional stage (refinement, judge, synthesis, a cascade's escalation): it adds the money held by stages already approved and not yet in the ledger (release frees it), compares with max_cost_usd, and compares the slowest call so far with the time left to max_latency_s. A refused stage degrades instead of failing: no refinement, deterministic judging, the digest instead of a synthesis, the first wave instead of escalation. BudgetReport (caps, forecasts, shifts, skipped stages) is stored with the run. Settings: CONFIGURATION.md.

An optional response cache (cache.py, BasePipeline.run) returns an earlier answer for an identical request in real mode; see CONFIGURATION.md. Its key is a hash of the redacted task, the task type, the whole strategy definition and max_models, so changing any of them is a miss.

Refinement (mixture-of-agents)

src/fusion/orchestration/refine.py runs after fanout, once for each round the strategy asks for beyond the first (rounds - 1; the packaged panel-refine has rounds: 2). Each successful panel model receives the peer answers anonymized as "Response A/B/C" plus its own answer and returns a revised answer. Failures keep the previous answer. Refined answers feed the judge, disagreement analysis, and synthesis; each call is traced as refine:{model} with full usage and cost.

Shadow baseline A/B

src/fusion/benchmark/shadow.py optionally calls the configured baseline model (Opus 5.5 by default) on the same sanitized task after synthesis, then asks the judge model for a blind pairwise verdict (answers presented in randomized order, unlabeled). Trigger via FUSION_SHADOW_MODE (off | sampled | always, with FUSION_SHADOW_SAMPLE_RATE) or a per-call shadow_baseline flag which overrides the env in both directions.

Results are stored in the shadow_comparisons table (winner, blind scores, actual baseline cost and latency). When a shadow ran, the run's cost_comparison reports the actual baseline cost instead of the token-volume estimate. Shadow costs are measurement overhead and are never counted as Fusion cost. All shadow failures degrade to warnings.

Stats

RunStore.get_stats() aggregates cumulative totals: run counts, Fusion spend, baseline estimates, per-task-type breakdown, shadow win/tie/loss counts, and the host tokens saved by runs that read the project themselves (summed from the stored workspace input). Rendering lives in src/fusion/telemetry/stats_format.py, surfaced through the fusion stats CLI command, the fusion_stats MCP tool, and a one-line lifetime footer on every real-mode run's display_markdown at detail: full (benchmark mode omits it). Both also show the latest fusion bench e2e study, read from the benchmark store by bench/e2e/summary.py (a simulated one is labelled), so the host-token savings and the Claude Code A/B sit next to the spend.

Claims and agreement

Panelists do not write free text. Every panel and refinement call carries the JSON Schema of PanelAnswer (orchestration/claims.py) as its response_schema, so providers that enforce structured output return a summary, a confidence and a list of atomic Claims; providers that cannot are given the schema in the prompt. A claim has a kind (finding, hypothesis, recommendation, risk, test), an optional severity (low, med, high, critical), file, line and evidence. A reply that is not valid claims JSON is read from its list items, flagged in the run's warnings, and counts against coverage.

ClaimsStage then works without any model call:

  1. cluster_claims groups claims across models. Two claims are the same point when they have the same kind and either cite the same file within 3 lines, or their normalized wording (lowercased, stop words dropped, light stemming) has Jaccard similarity of at least 0.5 (0.34 when they cite the same file). A cluster holds at most one claim per model, and models are visited in name order, so the result does not depend on arrival order.
  2. agreement_score sorts the clusters into consensus (backed by at least two models and half the panel), unique (one model), contradicted (backed by several, with severities two or more levels apart) and partial. score is the mean over clusters of (support - 1) / (n - 1), halved for contradicted ones, and 0 for fewer than two models. A model that shares nothing while the others share something is reported as an outlier (three or more models).
  3. calibrated_confidence is 0.5 * agreement + 0.3 * evidence + 0.2 * coverage, minus 0.15 * (contradicted clusters / clusters), clamped to [0, 0.95]. Evidence is the share of claims with a quoted evidence or a file and line; coverage is models that answered divided by models asked, times the share of answers that were valid claims JSON. With fewer than two answers there is nothing to agree with: the report is marked low_information, confidence is capped at 0.50, and the run carries a warning saying so. A model's own confidence is reported but never used.

The synthesizer prompt receives the clusters (with status and who backs each) and the answers rendered as Markdown, not raw JSON. For a solo run or a digest aggregator, and whenever the synthesis is not JSON, the task fields (critical_findings, ranked_hypotheses, ...) are filled from the clusters by kind, most severe and most agreed first (output_parser.py). The calibrated confidence always replaces whatever confidence the synthesizer wrote.

Evals

src/fusion/evals/engine.py coordinates the checks and scores:

  • safety checks always run on every answer: secret leakage, dangerous shell commands, cited files that are not among the provided ones, and a missing test claim in a coding answer;
  • the LLM judge runs only when the strategy's judge is light or full and a judge model is available (the default, off, makes no judge call);
  • without a judge (or when its call fails) per-answer scores are measured from the claims (evals/structural.py): the share of claims that cite a file, carry evidence, propose an action or are rated for severity. Dimensions that need a judge or a reference stay at a neutral 0.5, and an answer that was not valid claims JSON scores neutral throughout;
  • the final evaluation (evals/final_eval.py) takes confidence from the agreement report and derives usefulness (share of recommendation and test clusters), readiness (share with file and line), test plan (a test cluster exists, for coding tasks) and residual risk (the highest severity found) from the clusters. With no claims, as in a halted run, they are all 0.

No score depends on the wording of an answer. The aggregate score in evals is a weighted blend of these measures (context, agreement, answer quality, final quality, provider success rate) less penalties for unsupported claims and residual risk.

Synthesis

For an llm aggregator (the other aggregators are described under Cascade, aggregators and caps) the synthesizer prompt receives:

  • the original task;
  • the claim clusters and a short agreement summary;
  • each panel answer rendered as Markdown;
  • the requested JSON schema for the task's fields.

Raw panel outputs are not included in MCP responses unless include_raw_outputs=true.

Telemetry

src/fusion/telemetry/cost.py owns usage and baseline comparison schemas:

  • ModelUsage
  • UsageSummary
  • CostComparison
  • PricingRegistry

Prices come from the model catalog (src/fusion/config/catalog.yaml, date-aware price schedules with provenance); baseline comparison is loaded from src/fusion/config/baseline.yaml. See COSTS.md.

Storage

src/fusion/storage/run_store.py stores each run in SQLite (location and upgrade notes: CONFIGURATION.md). A RunStore owns one WAL-mode connection guarded by a lock; async code uses its a* methods, which run the query in a worker thread. Schema migrations apply atomically (BEGIN IMMEDIATE) so concurrent first opens are safe. Each run stores:

  • run ID and timestamp;
  • original and sanitized inputs;
  • routing;
  • trace;
  • panel, synthesis, and final outputs;
  • evals;
  • usage summary;
  • cost comparison;
  • warnings and errors.

The same schema also holds the benchmark tables (bench_runs, bench_items), used only by bench-results/bench.db, and the experience library's tables (lessons, run_feedback, lesson_uses; see Experience library).

CLI inspection commands:

uv run fusion runs list
uv run fusion runs show RUN_ID
uv run fusion runs costs
uv run fusion runs compare-baseline RUN_ID
uv run fusion runs export --format jsonl

Benchmark framework

src/fusion/bench/ runs studies; methodology, commands and the measurement block are in BENCHMARKING.md. The modules:

Module Role
spec.py, arms.py BenchTask, Arm, BenchConfig, the truth formats, dataset loading and splits; arms resolved to strategies, with overrides
datasets/ build (authoring YAML to JSONL, and spend-capped draft generation), validate (checks and statistics; it runs every coding task), generate, coding (reads a coding, frontend or performance task directory into a BenchTask, synthetic or real-repo), manifest (a dataset's _dataset.yaml release rules, the frozen _splits.json and the _review.yaml sheet), helpers/sitecheck.py (the test helper copied into frontend tasks); sources in evals/datasets/authoring/
runner.py run_bench: jobs keyed by hash, run concurrently under the per-provider limiters, resumable; a governor reserves each job's worst case against max_usd and the spend ledger
plan.py the cost and time estimator, built on budget_guard.plan_calls, and the fit-to-budget suggestion
store.py results.jsonl per run plus the bench_runs and bench_items tables (migration 5) in bench-results/bench.db
cache.py, spend.py the disk response cache (a provider wrapper that flags replays with ModelResponse.cache_hit) and the live-spend ledger
scoring/ the Scorer protocol (base) and the scorers: points, review, debug, rubric, coding (hidden tests in the sandbox), artifact (frontend and performance: evaluators, gates and criteria, completion); pairwise (the A/B judge), agentic (the tool-using judge) and calibration and artifact_calibration (judge accuracy, κ and the accuracy floor); see BENCHMARKING.md
evaluators/ what an answer's files measure: Evidence and the evaluators tests, build, static, diff_stats, perf (with its stdlib-only harness _perf_harness.py), visual, console and a11y (Playwright optional, in-page rules in assets/); EvaluatorSet holds a study's instances and caches; see BENCHMARKING.md
suite.py, suites/ the packaged suites (YAML BenchConfig fields with arms and overrides), loaded for --suite
judges.py neutral judges: every model an arm may call (every rung of auto) and the check that no judge shares a vendor with an arm (neutral_judges)
recommend.py the default-arm rule used on a dev study: cost-quality frontier, within the margin, cheapest per solved task
stats.py the statistics: task-clustered bootstrap intervals, paired comparisons (sign test, effect size), cost per solved task, the Pareto frontiers and the verdict engine; pure Python, see BENCHMARKING.md
meta.py, report/ meta.json (the facts of a run's start, for the methodology footer) and the reports: data assembles a Report (stats, caveats, evidence examples, call timelines), markdown and html (inline SVG from charts, no external asset) render it, compare diffs two runs, results writes docs/BENCHMARK_RESULTS.md
calibrate.py, metrics.py, summary.py running a judge calibration with the spend cap; BenchMetrics built from a run's ledger; per-arm summary rows
sandbox.py, patch.py the Sandbox context manager (limits, scrubbed environment, isolation) and patch extraction and application; used only by benchmark scorers, the verifier and the cross-check (fusion.verify, see Verified generation), never by MCP, see SECURITY.md
virtual.py an event loop with a virtual clock, used for simulated runs, and offload for blocking work
cli.py the fusion bench commands
fusion.tune fusion tune (BENCHMARKING.md): propose (edits, their validation and the proposer and composer calls), probe (a pack run on a slice through run_bench, the paired comparison, noise band and verdict), compose (the merge composer and the contrasts composers read), run (wide and deep modes, the held-out confirmation, the outputs), cli

providers/simulated.py holds the simulated models, providers/simulated_judge.py the way they play a judge (including the agentic tool protocol) and providers/simulated_coding.py the patches (and, for verified-synth, the checks) they write for coding tasks. providers/simulated_tune.py holds the simulated proposer and composer and the prompt effects planted for fusion tune --mock. The runner builds an ordinary BasePipeline over a strategy book extended with the study's arms and calls run(..., mode=Mode.BENCHMARK) with a RunLedger of its own, so a failed run's spending is still known; nothing in the pipeline knows it is being benchmarked beyond the mode.

Dashboard

src/fusion/dashboard/ is fusion dashboard: a Starlette app (already a dependency of FastMCP) that serves one static page (static/index.html, app.js, style.css; no build step, no CDN, no inline script) and a small JSON API. api.py holds the queries: the run database is opened through ReadOnlyStore, a RunStore whose connection is mode=ro, and a missing database is an empty dashboard rather than a created file. The overview reads get_stats() plus per-day sums (and the host tokens saved and the latest end-to-end study, bench/e2e/summary.py); a run's detail reads its stored output (the ledger.calls records, with their started_at_ms, are the latency timeline) and returns the sanitized input; the stored raw input is reached only when SecurityPolicy.log_raw_prompts is true. Benchmarks reuse fusion.bench: build_report and render_html for a run's report (shown in a frame under its own CSP) and compare_runs. app.py adds the two defences of a login-less local service: it answers only loopback Host headers, and it sends a Content-Security-Policy that allows nothing from any other origin. Only GET and HEAD are routed. Usage is in CONFIGURATION.md.

Safety boundaries

The orchestration MCP tools are side-effect free inside MCP: they do not execute shell commands in user repositories or edit files themselves. They read only the files and git diff a call names, read-only and under the project root (SECURITY.md). fusion_propose_patch is the one tool that runs code, and only in sandboxed temporary copies of the project; its patch is returned as text (SECURITY.md). That boundary should not block Claude Code. Claude Code can use Fusion's answer like a cheaper model response, then edit, run tests, and continue the normal coding workflow.