Skip to content

Changelog

All notable changes to RAGBench will be documented in this file.

The format follows Keep a Changelog, and this project uses semantic versioning once it reaches public releases.

[0.4.0] - 2026-10-27

Added

  • scripts/capture_report.py and refreshed report screenshots. docs/assets/report-*.png and docs/benchmarks/multihop-rag-run/report-*.png now show the published MultiHop-RAG run's report instead of the 0.3.0 demo run, and none is cut off: headless Chrome will not make a window narrower than about 500 px, so the earlier phone shots were laid out wider and cropped on the right. The script renders the report in an iframe of the exact width (390 px for the phone, 1280 px for the desktop shots, light and dark) and stops each image where the recommendation card ends. The README GIF's last scene shows that whole card. The published runs and the demo-dataset example runs were made on the unreleased 0.4.0 code, whose version string still said 0.3.0, so their manifests, report data, reports and the pages that quote the version now say 0.4.0.

  • Positioning. The README opens with a 20-second GIF (docs/assets/auto-to-report.gif, a real ragbench auto --mock run followed by the published run's report; python scripts/make_demo_gif.py remakes it, needs Pillow), the three-pillar story (trust the numbers, benchmark anything, improve the winner), "benchmark your own RAG, agent or model in 3 commands", and a headline sentence generated into the results block from the published run (scripts/benchmark_templates/multihop-rag.readme.md.j2, so check_results.py keeps it true). The roadmap no longer lists GraphRAG or MCP tools. New docs/comparison.md says how RAGBench differs from RAGAS, ARES, TruLens, DeepEval, promptfoo and Harness², what it does not do, and how to use them together; it is in the docs-site nav. docs/system-wiki.md gains a diagram of the optimize loop. CONTRIBUTING.md gains a good-first-issue list, the issue-label scheme (.github/labels.yml, synced to the repository by python scripts/sync_labels.py --apply, a dry run without the flag) and a short-subject commit convention.

  • The published benchmark: every RAG solution on MultiHop-RAG. docs/benchmarks/multihop-rag.md is a live run of configs/all_solutions.yaml (32 entries on 300 questions, gpt-6-luna generating, Haiku 5.5 judging), the three agent harnesses on 150 of them next to three of those systems, and six open-weight generators plus two references in three systems on 100: the full leaderboard with confidence intervals, significance, per-category winners, the Pareto front, the harnesses, the frameworks, the model comparison, the Task 22 optimize verdict and the limitations, every number rendered from the committed run folders under docs/benchmarks/multihop-rag-*. The README's example result is now this run (the demo-dataset run stays in docs/example-run.md), and the docs site lists the page. Pulled datasets (data/multihop-rag*/) are gitignored. docs/finding-results.md (Where the results are, linked from the README, the docs-site navigation and the top of the site's Leaderboard) says which benchmark is current, which page and folder holds what, and how to read a leaderboard.

  • The published benchmark configs. configs/all_solutions.yaml holds every RAG solution RAGBench can run, on the MultiHop-RAG 300-question subset: the systems of configs/all.yaml (with full_context at a 50k-token budget, full_context_50k), graph_rag local and global, the LangChain, LlamaIndex and Haystack reference pipelines, and the Claude Code, Codex and OpenCode harnesses; a test fails when a registered system type is missing from it (only external_http is exempt: there is no public endpoint to benchmark). configs/all_solutions_models.yaml runs three of its systems, configured identically, with the six open-weight generators of docs/models.md plus gpt-6-luna and claude-haiku-5-5, on a 100-question subset.
  • evaluation.judge.scorers: all | first. With first, only the first panel judge allowed to score an answer (by independent) scores it, so every answer has one judge and none is its own family's: panel: [anthropic:claude-haiku-5-5, gpt-6-luna] has Haiku judge everything except the Claude systems. ragbench estimate charges one judge per answer; such a run reports no judge_agreement and records judge.scorers in run_summary.json (methodology). The default all is unchanged.
  • ragbench run --run-dir DIR writes the run into DIR instead of a new timestamped folder; running into it again resumes, keeping the systems that finished (checkpoints) and running the rest, as ragbench auto --resume already did.
  • A budget stop keeps the answers of unfinished systems. A system the spending cap stopped partway, or that finished with failed questions, now leaves a partial checkpoint (checkpoints/<system>.partial.json, the answers it gave without an error, under the same context hash); ragbench run --run-dir DIR or ragbench auto --resume DIR gives them back and asks only the skipped and failed questions, instead of re-running the system from its first question (Claude Code at 138/150 and OpenCode at 105/150 re-ran all 150 in the published run, and Run B model combinations lost 14–76 answered questions each time the cap was raised). What the kept answers were charged counts towards max_cost_usd, as restored systems already did, so resuming under the same cap starts nothing new. Finished-system checkpoints and their hash are unchanged (configuration).

  • scripts/check_results.py and scripts/publish_benchmark.py. Every docs/benchmarks/<name>.md and the README's results block are rendered from scripts/benchmark_templates/ and the result files committed under docs/benchmarks/<name>-run/ (--write regenerates them); CI fails when a page or block differs from its rendering, or when a page under docs/benchmarks/ has no template, so a published number cannot drift from the CSVs. publish_benchmark.py NAME --run DIR [--models DIR] [--optimize DIR] copies a finished run's small files there (never the per-question rows) and rebuilds its reports from them. It also writes patterns.json (per-question counts and means, no question or answer text), from which the page's Patterns across the runs section is rendered: which question categories are hard, how far retrieval and cost explain the ranking, the failure types, and how a generator's rank changes with the system; the README block quotes three of them. A run folder may also hold a hand-recorded corrections.json (kept on re-publish, like the screenshots): the MultiHop-RAG page uses it to correct OpenCode's cost, which OpenCode estimated at OpenRouter's cheapest catalog price; OpenRouter probably billed 2.74–4.25× that.

  • Docs site, cross-run page and versioned result schemas. python scripts/build_site.py builds an mkdocs-material site (pip install 'ragbench[docs]') from the repository's Markdown (README as the home page, links rewritten to follow the files or to GitHub, --strict so a broken link or anchor fails), and the site / deploy-site CI jobs build it on every change and publish it to GitHub Pages from main. Its Leaderboard page and a filterable twin are generated from the result files committed under docs/benchmarks/*-run/ (the two committed runs moved there from docs/example-run/ and docs/decision-gate-run/), never typed by hand. ragbench runs site FOLDER [--out DIR] [--title T] turns many run directories into one static, filterable, sortable cross-run page (one HTML file, no network requests, each run's report.html copied and linked; mock runs hidden when a live run exists; a missing number stays —, not zero). run_summary.json now carries schema_version: 1, and ragbench runs schema [run_summary|metrics_summary] prints the JSON Schemas of run_summary.json and of one row of metrics_summary.csv; a test validates fresh and committed runs against them (results). New extra docs; jsonschema joins the dev extra.

  • Trace export to Phoenix, Langfuse and any OpenTelemetry backend. ragbench export-traces RUN [--format jsonl|openinference|otlp] [--endpoint URL] [--header NAME=VALUE] [--project P] [--system S] [--question Q] [--limit N] [--out FILE] turns a finished run's traces into spans: one trace per answered question (and trial), a root ragbench.answer span with the question, answer, system, judge score and cost, and one child per step (retrieve:bm25_search, generate:answer, ...) with previews, tokens, cost and retrieved document ids. jsonl (default) writes traces.jsonl offline; openinference and otlp send over OTLP/HTTP (new extra otel: pip install 'ragbench[otel]'), with OpenInference or OpenTelemetry GenAI attributes. Ids derive from run, system, question and trial, so a second export never duplicates; header values may use ${ENV_VAR} and are never printed; reference answers and the judge's reasoning are never exported. Nothing is re-run or paid for. A new otel CI job runs the sending tests against the real exporter and a local collector (observability).

  • Experience library. ragbench optimize now keeps what it learned: when a harness is adopted (the test split confirmed the gain) an entry is appended to ~/.ragbench/library/entries.jsonl (or RAGBENCH_LIBRARY, or library.path in the config; library.enabled: false and --no-library switch it off) holding the harness, the verified gain with its confidence interval and one lesson, keyed by a corpus fingerprint from the dataset profile. Later runs read the nearest entries (similarity of at least 0.75, only those older than the run's own start): optimize shows their lessons to the proposer in the evidence tree's library/ folder, and ragbench auto without --preset chooses the preset that includes the system the nearest entry was optimized for (never smaller than standard; the matches go to library.json). Nothing held out is written (features come from the train questions; every entry passes the leakage guard), and mock and live entries never mix. ragbench library list|show|export|clear (ragbench.optimize.library; docs in optimization.md).

  • ragbench optimize: Propose → Probe → Compose over a system's harness, proven on held-out data. ragbench optimize --config C --system NAME [--k 2] [--mode parallel|sequential] [--feedback contrast|labels|none] [--proposer REF] [--finalists 2] [--select auto|score|arena] [--test-trials 1] [--max-cost-usd X] [--out DIR] [--mock] improves one system's prompts and knobs (its harness:) on the train split. Parallel mode probes k base rollouts, k INV and k DOM candidates and k compositions; sequential mode runs k rounds carrying the best harness forward. Both make exactly 4k train probes and 3k proposer requests, which ledger.json asserts. The candidates that beat the base on train go to dev (answer score, or pairwise arena points on a label-free dataset); the pick and the base are then compared on test (paired bootstrap, Holm), and only a significant gain is adopted: anything else is "no reliable gain" and the base is kept. Writes optimized.yaml (your config with the adopted harness, else unchanged), optimization.md (lineage, diffs, train and dev results, candidate diversity, verdict) and ledger.json, plus every probe's run folder, evidence tree and saved proposal. A spending cap is required for a live run and honoured between and inside probes; re-running into the same folder resumes by harness fingerprint, without asking the proposer again. --mock runs the whole loop offline with a scripted proposer. ragbench estimate --optimize NAME --k K prices the loop (probes, dev, test, proposer requests). The proposer gained a COMPOSE request, and build_evidence a trial filter. New guide docs/optimization.md; the decision rule is in methodology.md.

  • Evidence tree and harness proposer (groundwork for ragbench optimize). ragbench.optimize.evidence.build_evidence(run_dir, split="train", system=..., feedback=labels|contrast|none) writes what a proposer may read about one system of a finished run: MISSION.md, task.md (from the dataset profile), harness/ (effective harnesses and the fields that may change), rollouts/ (one trace per train question), contrasts/ (the contrast sets against the run's other systems), failures.md, library/ and manifest.json, by default under <run>/evidence/<system>-<feedback>. Only the train questions of the run's splits are read (dev and test are held out, and a changed splits file is refused); every passage from the run goes through the leakage guard and is withheld when it quotes held-out text, and the finished tree is checked as a whole and deleted if anything held out remains. contrast mode (the default) shows behaviour only, with no grade, reference answer, failure type or winner, and a tripwire enforces it. ragbench.optimize.proposer.Proposer (any model ref, or a scripted mock) turns a tree into one INV (general) or DOM (collection-specific) harness patch; an unusable patch is retried once with the reason, then the current harness is kept. ragbench optimize drives both. See methodology.md.

  • Contrast sets: where systems disagree. ragbench contrast RUN A B [--source score|verdicts] [--threshold T] and evaluation.contrast: {enabled, pairs: vs_baseline|all, source, threshold} (off by default) list the questions where exactly one of two systems is right (its mean answer score reaches the threshold, 4.0 like cost per correct answer, or it won the pairwise arena's verdict), with both traces aligned step by step and the first step at which they part (different_stage, different_documents, different_output, different_answer), clustered by the loser's failure type, question category and divergence stage. Written to contrasts.jsonl and contrast_summary.json, with a "Where systems disagree" section in leaderboard.md and report.html (rebuilt by ragbench report); computed from the run's files, so it costs nothing, and build_contrasts(run_dir, question_ids=...) is the entry point for the optimizer's evidence. Runs without it are unchanged. See methodology.md.

  • Benchmark a RAG you did not build: external_http and python_callable systems. external_http sends each question to your endpoint (url, method: POST|GET, headers with ${ENV_VAR} secrets filled in per request and never logged or saved, a Jinja request template) and maps the JSON reply with a JSONPath subset (response.answer, contexts, and per passage text, doc_id, score; optional prompt_tokens, completion_tokens, cost_usd), with timeout_s, retries (timeouts, connection errors, 429, 5xx), limits and post_documents_to (posts the corpus once before the questions). python_callable calls target: pkg.module:function or path/file.py:function as function(question, top_k), returning a dict or an AnswerResult, with an optional ingest_target. Each answer is one external step (a new trace stage), unless a python_callable records its own steps on the active tracer (current_tracer()) or returns them (an AnswerResult's steps, or steps dicts in the reply), which then make up its trace and, without a reported cost, its cost; this is what framework adapters use to split retrieval from generation. Passage ids are matched to corpus documents by id, path, file name or stem; a system whose passages are missing or do not map gets blank retrieval and context metrics (never zeros) and a notice; unknown cost is $0 with a notice; both are listed under external in run_summary.json. retrieval.model declares the model the system uses, for pricing reported tokens and judge independence (RAGBench calls no model for an external system; undeclared, its family is unknown). --mock does not stub external systems, ragbench estimate never calls them, and their calls are not cached. New owner doc docs/benchmarking-external-systems.md ("your deployed RAG in 3 steps"), configs/external_example.yaml, and two runnable examples: examples/fake_rag_server.py (a stdlib stand-in service, also the tests' fixture) and examples/simple_rag.py.

  • Framework adapters: LangChain, LlamaIndex and Haystack pipelines as python_callable targets. ragbench.adapters.langchain (from_chain, from_retriever), ragbench.adapters.llamaindex (from_query_engine) and ragbench.adapters.haystack (from_pipeline) wrap a pipeline you built into the (question, top_k) function python_callable calls, and split each answer into a retrieve step and a generate step per model call, instead of one external step. Steps come from the framework's own hooks (a LangChain callback handler, LlamaIndex instrumentation events, wrapped Haystack components) and are filed per question, so concurrent questions never mix; the tokens a model call reports are priced at the model the framework names, or at retrieval.model when it names none or one without a price. Passages are the pipeline's own retrieved documents, mapped to corpus files through their metadata (source, file_path, ...). New optional extras langchain, llamaindex and haystack; a missing one fails with the one-line pip install 'ragbench[...]'. Query-embedding calls inside a framework are not seen and not charged.

  • The three frameworks out of the box, for the leaderboard: examples/langchain_default.py, examples/llamaindex_default.py and examples/haystack_default.py build the pipeline each framework's own tutorial recommends (default splitter, in-memory vector store, plain question-answering prompt) with the built-in systems' generator and embedding model, and configs/frameworks_example.yaml runs them next to bm25. They call OpenAI only with RAGBENCH_REFERENCE_LIVE=1 and a key (--mock does not stub external systems, so spending is its own decision); otherwise they run on RAGBench's mock models, with no key and no cost. A frameworks CI job installs the three extras and runs tests/test_framework_adapters.py against the real frameworks. See benchmarking-external-systems.md.

  • MCP tools for agents. tools: [{type: mcp, server: {command: [...]} | {url: ...}, allow: [tool names]}] offers the listed tools of a Model Context Protocol server to agent_search (and adaptive's routes) with the server's own descriptions and argument schemas; optional prefix, cost_usd (a price per call, for every tool or per tool, since MCP reports none) and timeout_s, and ${VAR} in env / headers filled at start-up so secrets never reach a config or a run directory. One connection per server per run, shared by all questions; calls are traced tool steps, so tool_usage.csv reports each MCP tool's calls, errors, latency and cost, and a result that lists doc_ids feeds the trajectory metrics. The config check never starts a server; --mock never does either (each allowed tool answers a fixed placeholder at its configured price). A command server counts as a filesystem side effect, a url server needs tools.allow_network: true; allow on the entry replaces a tools.allow listing. New extra mcp (mcp>=2.3) and CI job; example server examples/mcp_server.py and configs/mcp_example.yaml; guide in docs/tools.md.

  • Agent harnesses as systems: cli_agent. Presets claude_code (claude -p --output-format stream-json --restricted --strict-mcp-config --disable-slash-commands --tools Read,Grep,Glob), codex (codex exec --json --sandbox read-only --ignore-user-config), opencode (opencode run --format json --pure --dir <workspace> with an opencode.json that denies edits, shell and webfetch) and custom (any command, with {prompt}, {model}, {workspace}, {task_file}). Per question: a fresh temporary workspace with a read-only copy of the corpus under docs/, the task in TASK.md, optional harness files (system_prompt, instructions as AGENTS.md, files), never a reference or label; timeout_s, max_steps (tool calls; the harness is stopped and answers with its last message), answer_file, env, limits, network: off|on (a custom harness's network cannot be switched off: a warning). The event stream is parsed into steps (model turns, tool calls with their input, output, error and the documents they searched or read, the answer); documents read are the first passages, then files whose matching lines a search showed; cost is the harness's reported cost (not Claude Code's guess for a model it does not know), else its tokens at model's price; agent.max_prompt_tokens records the largest prompt; harnesses run with PWD set to the workspace, and a file tool pointed outside it is flagged (outside_workspace, agent.outside_workspace_calls, a warning). Parsers are pinned on recorded real runs of Claude Code 2.1.287, Codex 0.162.1 and OpenCode 1.19.4 (tests/fixtures/cli_agent/*_recorded.jsonl). --mock never starts a harness: a bundled offline harness (ragbench.rag_systems.cli_agent_mock) prints the preset's format. Guide: Agent harnesses; example: configs/cli_agent_example.yaml.

  • A warning when a harness's global instructions file has content (~/.claude/CLAUDE.md, $CODEX_HOME/AGENTS.md, ~/.config/opencode/AGENTS.md): it may reach every prompt whatever the workspace holds.
  • A run that includes a cli_agent is live unless --mock is given, even when it calls no RAGBench model (judge off), so the harness is never silently replaced by the offline one.

  • Agent trajectory metrics for every agentic system: redundant_search_rate and avg_steps_to_evidence in metrics_summary.csv and the report's agent table, next to tool_error_rate (methodology). Retrieval and tool steps now record the documents they surfaced (metadata.doc_ids).

  • raptor (recursive summary tree, Sarthi et al. 2024). Chunks are embedded and clustered (scikit-learn Gaussian mixture on PCA-reduced embeddings, the number of clusters by BIC), an LLM summarises every cluster, and the summaries are clustered and summarised again until max_levels or fewer than min_cluster_nodes nodes; a cluster longer than summary_input_chars is clustered again within itself, so each summary reads its members whole. mode: collapsed (default) searches chunks and summaries as one index; tree descends from the top level keeping branch_k nodes per level. Summaries are cached and built ingest_workers at a time; a retrieved summary counts as its lead document for the retrieval metrics (all its documents in metadata.doc_ids).

  • compression (contextual compression). Retrieves candidate_k (default 20) hybrid candidates, cuts each to the sentences that bear on the question and packs them to context_token_budget: compressor: llm_extract (one call quotes the sentences; a quote not found word for word in its chunk is dropped and counted), embedding_filter (cosine to the question, no LLM) or decision (the decision-model interface). The uncompressed ranking is the one scored for retrieval; the judge and the context metrics read the compressed text. New summary columns avg_context_tokens_before and avg_context_tokens_after, and a compression block per question.
  • late_interaction (ColBERT MaxSim). Per-token embeddings scored by MaxSim: mode: rerank (default) reorders hybrid or BM25 candidates, full scores every chunk exactly from an fp16 index built at ingestion, refused above max_index_vectors. A local model (default answerdotai/answerai-colbert-small-v1), $0 API cost, loaded once per run and shared; new extra colbert (transformers, torch, safetensors) with RAGBench's own checkpoint loader for the Stanford ColBERT and PyLate layouts (pylate 1.6.0 pins sentence-transformers 5.3.0). Mock runs use deterministic hashing token vectors and download nothing.
  • Editable harness: harness: per system. Every answer prompt is now built from fields a config can patch: system_prompt, grounding, refusal_policy, answer_instructions, citation_style (one rule each; "" drops a rule), query_rewrite_prompt (the system message of the query-writing call of hyde, llm_heavy, rag_fusion, decompose, corrective, iterative), tool_guidance (tool-using agents) and few_shot ({question, answer} examples, train split only: a run stops before it starts when an example holds a dev or test question, or, without a split, any question of the dataset), plus bounded knobs that set retrieval: / chunker: options (top_k 1–50, chunk_size 64–4096, reranker, vector_weight 0–4, max_hops, max_rounds, max_steps). Omitted fields are inherited; unknown fields get a did-you-mean, and a field the system does not use is rejected with the ones it has. With no harness: every prompt is byte-for-byte what it was (checked over every model call of configs/all.yaml, graph_rag.yaml and decision_gate.yaml in mock) and checkpoint hashes are unchanged; a patched system adds its harness fingerprint to its checkpoint hash, so editing one system's prompt reruns only that system. The LLM cache key already holds the full prompt, so no harness hash is added there (an unchanged call, such as ingestion, still hits). ragbench harness show SYSTEM [--config C] prints a harness as YAML (patched fields marked) and ragbench harness diff A.yaml B.yaml compares two configs field by field. Ingestion and control prompts (summaries, graph extraction, grading, routing, reranking) are not part of the harness; external systems take none. See configuration.md; a custom system declares its prompts with default_harness (extending.md).
  • configs/all.yaml now also runs raptor_collapsed, compression_llm_extract and late_interaction_rerank; the existing systems' mock results are unchanged, and the golden snapshot is untouched (it ignores systems added after it).
  • On macOS a live run that would load torch (late_interaction, cross_encoder, local: embeddings) and faiss into one process, adaptive routes included, is refused before anything is built, naming the systems that bring each: the two ship separate OpenMP runtimes and the process would abort ("OMP: Error #15") or, with KMP_DUPLICATE_LIB_OK set (scikit-learn sets it on import), crash. Mock runs and other platforms are unaffected.
  • A system can tell the evaluator what text the generator actually read for each context chunk (metadata["context_texts"]); the judge, the context metrics, claim checks and ragbench estimate use it.

  • graph_rag (GraphRAG-lite). At ingestion an LLM extracts each chunk's entities and relations (one temperature-0 JSON call per chunk, ingest_workers at a time, disk-cached), the replies merge into one graph, Louvain splits it into communities, and each community gets an LLM report. mode: local matches the question to entities, follows their relations and returns the chunks they came from, with the entities, relations and main community report as notes for the generator; mode: global ranks the community reports and answers by map-reduce over the ones the generator reads. Options entities_k, neighbours_per_entity, max_relations, notes_max_chars, max_entities_per_chunk, min_community_size, community_max_entities. Ingestion is reported as ingestion cost and priced by ragbench estimate; mock runs build a real graph from capitalised names. Example: configs/graph_rag.yaml; how it works: system-wiki.

  • A bundled hard suite that separates systems. ragbench demo --suite hard writes data/hard (162 fictional documents, ~52k tokens, 156 questions with graded qrels and evidence quotes), and configs/hard.yaml runs eight representative systems on it (no_retrieval, full_context at a 16k budget, bm25, vector, hybrid_rerank, decompose, iterative, agent_search with tools). Categories: conflicting document versions, dated (temporal) questions, 2- and 3-hop chains whose bridge is never named, aggregation over quarterly reports (calculator), near-name distractors, facts buried in long annual reports, and 25% unanswerable near misses. It is generated deterministically from one seed by scripts/generate_hard_suite.py (--check and a test keep data/hard equal to its output) and ships in the wheel and the Docker image next to the demo. See dataset-format.md.

  • Dataset hub: public benchmarks on demand. ragbench datasets list and ragbench datasets pull NAME [--output DIR] [--subset N --seed S] [--force] [--yes] download a benchmark after telling you the size and asking (no terminal and no --yes: it refuses), verify every file against a checksum and commit pinned in RAGBench (SHA-256, or the git blob id for small non-LFS files; a mismatch deletes the file and fails), convert it, and write docs/, questions.jsonl, qrels.jsonl, SOURCE.md (license, citation, caveats) and hub.json. Files are cached in ~/.cache/ragbench/datasets (RAGBENCH_DATASETS_DIR). Nothing is bundled. Entries: multihop-rag (609 news articles, 2,556 questions, null queries become unanswerable; ODC-By), hotpotqa (distractor validation, 7,405 questions with sentence-level evidence; CC-BY-SA), and the twelve test subsets of Galileo's RAGBench as galileo-<subset> (CC-BY-4.0, each subset also carries its source's terms). CRAG (CC-BY-NC, multi-gigabyte web pages) and FRAMES (live Wikipedia, which cannot be pinned) are left out on purpose; BEIR stays ragbench import --format beir.
  • --subset N is a stratified sample that keeps every question category and the unanswerable share (at least one of each stratum, the rest in proportion, deterministic per seed); a shared corpus is kept whole, per-question passages are kept for the kept questions only. Every hub dataset ships a configs/<name>.yaml (three systems and the usual judge) generated by scripts/generate_docs.py, which also keeps the dataset table in docs/dataset-format.md current.
  • New optional extra hub (pyarrow) for the Parquet-based datasets; CI installs it so their reader tests run.
  • Train / dev / test splits. ragbench split DATASET --train 0.5 --dev 0.1 --seed 42 writes splits.json (question ids per part, stratified by category and answerable / unanswerable, deterministic for a seed) and refuses to overwrite it. dataset.split: train|dev|test|all (default all) and dataset.splits_path run on one part; max_questions and ragbench estimate count that part's questions. The split name and the hash of the assignment are recorded in run_summary.json (dataset.split), run_manifest.json (split), the HTML report and leaderboard.md, and join the checkpoint context hash, so resume never reuses answers from another split or another splits file; ragbench compare-runs notes runs of different splits. A run with split: all records nothing new and its outputs and hashes are unchanged.
  • LeakageGuard (ragbench.datasets.leakage): rejects any text, file or folder that contains a held-out (test by default) question's id, text or reference answer, naming the file and question but not the content. It is the tripwire the evidence builder, proposer and experience library will pass their artifacts through.
  • Claim-level faithfulness. evaluation.claims: {enabled, model, max_claims} splits every answer into atomic claims and checks each against the context the generator was given, one yes/no request per claim through the decision-model interface. New summary columns claim_support_rate and unsupported_claims, a claims block per question, and the unsupported claims listed in the report's drill-down. An answer that declines to answer has no claims and no rate, a claim the model could not answer is left out of the rate, and spend is charged as judge cost (counted by max_cost_usd, priced by ragbench estimate). Mock runs split into sentences and check word overlap. See methodology.md.
  • Model-based refusal detection. evaluation.refusal_detector: heuristic|model (default heuristic, which mock runs always use): model asks the judge model, through the decision interface, whether an answer declines to answer, so refusals phrased in unusual ways count in the abstention metrics, refused and the failure types; an unusable reply falls back to the wording heuristic.
  • Chunk-level evidence. A question (or a qrels.jsonl row) can list evidence: [{doc_id, quote}]; retrieval is then also scored per chunk as evidence_recall@k (the share of quotes found in the top k chunks of their own document), averaged over the questions that have evidence. The dataset warnings flag a quote that is not in its document. Datasets without evidence are unchanged. See dataset-format.md.
  • Pairwise arena. evaluation.pairwise: {enabled, pairs: all|vs_baseline|adaptive, max_pairs_per_question} has the judge compare two anonymous answers to the same question after the main pass, in both orders (the first order is a seeded coin flip), with ties allowed; a verdict that follows the position cancels into a draw. The verdicts become a Bradley–Terry arena score on the Elo scale with a bootstrap interval over questions, a win-rate matrix and the judge's position-bias rate (flip_rate, first_position_rate), in arena.csv, pairwise.jsonl, new metrics_summary.csv columns (arena_score with interval, arena_win_rate; an "Arena" leaderboard column), the run summary, leaderboard.md and a "Pairwise arena" report section. It needs no reference answers, so label-free datasets get a ranking that is not only faithfulness to the context. With a panel and independent: true each pair goes to a judge of neither side's family (or to every judge, so their leanings offset). Comparisons are charged to the run, count toward max_cost_usd, and ragbench estimate lists them. Off by default: runs without it are unchanged. See methodology.md.
  • ragbench.evaluation.pairwise.select_best(candidates, question, judge) picks the best of several answers without labels (anonymous, seeded shuffled order, every pair in both orders; ties and failures keep the first candidate), for the optimizer's label-free mode.
  • Judge panels. evaluation.judge.panel: [model, ...] has two or more judges score every answer independently; the verdict is their mean (or aggregate: median) per axis, a judge whose reply was unusable is left out of that answer, and ragbench estimate charges every judge. With independent: true (the default) a judge sits out the systems whose generator is from its own model family (families are read from the model name or OpenRouter vendor), so a mixed-family run can be judged without self-preference. Each row records judges, sat_out, member_fallbacks and every judge's panel_scores; run_summary.json gets a judge block with judge_agreement, Krippendorff's alpha between the judges per axis, shown in the report banner, with a warning below 0.667. A one-judge config writes exactly the same rows as before. See methodology.md.
  • ragbench judge-audit RUN. --sample 50 writes a blind, stratified sheet (answers drawn evenly per system and judge-score band; no system names or judge scores on it) for a person to score; --labels filled.csv writes judge_audit.md / .json (judge-human alpha, mean absolute difference and bias per axis, verdict agreement, bias per system, and each panel judge against the human) and links it from the report. See methodology.md.
  • Repeated trials and variance-aware statistics. evaluation.trials: N answers and judges every question N times; each trial is a fresh answer (the trial index joins the disk-cache key of every LLM and decision call from trial 2 on, so trial 1 still hits older caches). A question's score is the mean over its trials; intervals, significance.csv, the recommendation's tie test and ragbench compare-runs use a cluster bootstrap (questions resampled with all their trials), which widens the interval of a system whose answers vary without counting trial noise twice. New summary columns when N > 1: answer_score_trial_sd, pass_at_k, pass_hat_k (all k trials correct) and rank_stability; per-answer rows carry trial; n_ok, n_error and max_error_rate count answers; resume checkpoints include the trial count. ragbench estimate multiplies question cost and time by N. With trials: 1 every output is unchanged. See methodology.md.
  • Latency intervals. Every run reports an exact, distribution-free 95% interval on the p95 (latency_ms_p95_ci_lo / _hi, shown as p95 [lo, hi] in the leaderboards); it has no upper bound until there are 72 measurements, so the default 5-call probe gives a lower bound only. evaluation.latency_probe_questions: auto probes until the p95 is within ±15% (72 to 120 calls per system). The golden snapshot is unchanged (the new columns are latency, which it excludes).
  • The latency probe now counts toward evaluation.max_cost_usd (it stops at the cap) and is included in ragbench estimate's total; tools run in the calling answer's context, so a custom tool that calls a model gets a fresh answer per trial.
  • Release plumbing. .github/workflows/release.yml publishes to PyPI with trusted publishing (OIDC, no stored token) and creates the GitHub release from the matching CHANGELOG.md section when a tag vX.Y.Z is pushed; scripts/release_notes.py first checks that the tag, pyproject.toml, CITATION.cff and the changelog agree. The package checks (sdist and wheel, twine check --strict, demo data in the wheel, clean-environment install) moved to the reusable .github/workflows/package.yml, which CI and the release both call, so the release publishes the very files that passed them. The distribution name is ragbench (free on PyPI); the README says how it differs from Galileo's RAGBench dataset.
  • A Dockerfile (slim, multi-stage, non-root, ENTRYPOINT ["ragbench"]; the mock demo works with no key) and a CI job that builds it and runs the demo in it. The README quickstart is now pip install ragbench / uvx ragbench demo; docs/release-checklist.md has the tag-and-publish steps, including the one-time PyPI trusted-publisher setup.
  • openrouter: provider for any model on OpenRouter, frontier or open-weight (openrouter:deepseek/deepseek-v4.1-flash, openrouter:z-ai/glm-5.3, ...), with OPENROUTER_API_KEY. A call is charged what OpenRouter billed (usage.cost), also on a cache hit; requests go out with reasoning off, or at the lowest effort a model that must reason accepts; a new openrouter: config section sets reasoning and provider routing (order, allow_fallbacks, data_collection) for all models or per model, and the settings are recorded in run_manifest.json (model_settings) and keyed into the cache. Tool and JSON support come from the model's supported_parameters: a model without function calling runs the tool agents in react_json mode.
  • ragbench models list|sync. list prints every price RAGBench knows (built-in table and OpenRouter catalog, with long-prompt tiers, open weights, tools and default reasoning); sync saves OpenRouter's public model list as a dated catalog file that prices openrouter: refs offline. A snapshot of the shortlisted models ships with the package.
  • docs/models.md: which generator, judge, embedding and proposer to use, the shortlist of frontier and open-weight models with prices (checked 2026-10-24, and again on 2026-10-27 for the release: openai 3.39, ruff 0.17.3, langchain-openai 1.7.3, haystack-ai 3.4, transformers 5.21, torch 2.14.1 and httpx 0.28.1 raised, pre-commit's ruff hook with it; a fresh environment with every extra shows nothing outdated but three packages other packages pin), the self-preference rule, and how OpenRouter models are called.
  • configs/models_compare.yaml: three systems swept over six open-weight generators, judged by Claude Haiku 5.5.
  • .env.example lists OPENROUTER_API_KEY; the release checklist asks for the model-and-price audit before every paid run.
  • decision_gate, a decision-gated RAG system. It retrieves exactly like hybrid_rerank, then asks a typed decision model two yes/no questions before generating: does each retrieved chunk help answer the question (chunks below answerability_threshold are dropped), and is what is left enough to answer it (below sufficiency_threshold the system abstains: no generator call, the usual "could not find" answer). The ranking handed back and scored is the ungated one, so retrieval metrics stay comparable; the context metrics and the judge see the kept chunks. Every request is a traced grade step, the probability of every judged chunk is saved per question in a new gate block of per_question_results.jsonl, and new summary columns report gate_abstain_rate, avg_chunks_kept and avg_decision_calls (correct and wrong refusals are scored by the existing abstention metrics). The measured comparison with hybrid_rerank, corrective and the no_retrieval floor, with real API runs, is in docs/decision-gate-evaluation.md; configs/decision_gate.yaml reproduces it, and configs/all.yaml now includes the default and strict gate, so the example run (regenerated from a joint run on 7 Oct 2026) is the one leaderboard of every system.
  • Pluggable decision models (ragbench.models.decisions, DECISION_MODELS registry, ragbench.decision_models entry-point group): openai_decisions (OpenAI's Decisions API, POST /v1/decisions, model gpt-6-luna), llm_json (the same questions put to a chat model that answers with JSON, the like-for-like control), mock (deterministic probabilities, used by every --mock run) and jev (reserved: it raises a clear error until a client exists). Requests go through the run's disk cache under the same fairness rules as chat calls.
  • A decision_cost / decision_input_tokens pair in the cost breakdown. The Decisions API bills input tokens only, at the model's input price; ragbench estimate prices the measured decision tokens at the decision model's rate instead of the generator's.
  • scripts/generate_docs.py expands a nested option block (decision_model:) into dotted rows in docs/systems.md.

Changed

  • For extension authors: prompts now come from the system's harness. ToolAgent(llm, box, budget, harness, mode=...) takes the resolved harness instead of instructions=, a tool agent's _instructions() hook is gone (declare tool_guidance in default_harness), ANSWER_SYSTEM_PROMPT, NO_RETRIEVAL_SYSTEM_PROMPT and REDUCE_SYSTEM_PROMPT became the default fields ANSWER_PROMPTS, NO_RETRIEVAL_PROMPTS and REDUCE_PROMPTS in ragbench.harness.prompts (render one with PromptHarness.resolve(...).answer_system()), and decompose's DECOMPOSE_SYSTEM_PROMPT is now DECOMPOSE_NOTE, appended to the harness's answer prompt. The mock model now recognises a tool agent's turn by its tools or ReAct block rather than the agent's opening line, which a harness may change.
  • Shipped configs judge with another model family. Every configs/*.yaml with OpenAI generators is now judged by anthropic:claude-haiku-5-5 (same price as gpt-6-luna), so a live run of them also needs ANTHROPIC_API_KEY and pip install 'ragbench[anthropic]'; mock results and the golden snapshot are unchanged (mock runs use the heuristic judge). The default judge_model stays gpt-6-luna, so a config without one needs only an OpenAI key.
  • The self-preference warning is about model families, not one model: it names every system that only judges of its generator's family scored (it used to fire only when the judge and generator were the very same model, so gpt-6.1-sol judged by gpt-6-luna went unflagged). The notice's build_notices argument is judge_shares_family_with (was judge_shares_model_with).
  • The evaluator is split into focused modules (ragbench/evaluation/): evaluator.py keeps run orchestration (checkpoints, spending cap, scheduling), next to plan.py, question_runner.py, rows.py, summary.py, outputs.py and warm_up.py; no module is much over 400 lines (evaluator.py was 1,025). New seam: EvaluationPlan (which systems × which questions × how many trials) is what SystemRunner.run_all(plan, documents) runs; it supports one trial for now. No behaviour change: every output file of configs/all.yaml --mock is byte-identical under a fixed clock, and the golden snapshot is unchanged.
  • Faster, stricter dev loop. pytest-xdist is a dev dependency; the eight slowest tests (full parallel runs, the live path, the retrieval-depth sweep) are marked slow and --strict-markers is on. pytest -m "not slow" -n auto runs 1405 tests in about 40 s (the whole suite is 1413 tests: about 4 min serial, 85 s with -n auto). CI runs the fast tests on every Python version and the whole suite, with coverage, on 3.13 only. mypy runs with check_untyped_defs = true (nothing needed fixing).
  • Dependency floors are the latest releases (checked 2026-10-24), for the core, dev and every extra, including the majors anthropic 2 and twine 7, plus ruff 0.17, mypy 2.4, openai 3, pandas 3.1, numpy 2.5 (2.4.6 on Python 3.11, which numpy 2.5 no longer supports), pydantic 2.14 and sentence-transformers 6.2. The build requires setuptools>=84.2.1.
  • CI: actions/upload-artifact@v7 (replaces the open Dependabot branch for it), Python 3.14 is a normal matrix cell now that every dependency ships wheels for it, and Python 3.15 is added as a best-effort cell (pyyaml has no 3.15 wheel yet). The note that FAISS has no macOS wheel for 3.14 is gone: 1.15 ships abi3 wheels.
  • pre-commit hooks updated (ruff-pre-commit v0.17.2, mirrors-mypy v2.4.0); Dependabot groups the weekly pip and github-actions updates into one pull request each.
  • Price table refreshed (2026-10-24, from OpenAI's and Anthropic's pricing pages). claude-haiku-5-5 ($0.10 / $0.50 per 1M tokens, $0.50 / $2.50 for a prompt over 100,000 tokens) replaces claude-haiku-4-5 ($1 / $5), whose row is removed; it is also the independent judge in the docs' examples. gpt-6-luna, gpt-6.1-sol, gpt-6-astra and text-embedding-3-small are still current. Haiku 5.5 is called at low effort like the other Claude models that think by default.
  • Prices can depend on prompt length. A price row (and a pricing: override) may have tiers: [{max_prompt_tokens, input, output}, ...]; each request is priced on its own at the tier its prompt falls in. The GPT-6 rows now carry their long-context rates (above 272K prompt tokens). ragbench estimate, dataset synthesis and pooled grading pick the tier per call rather than from summed tokens. Requests over a threshold are rare in RAG runs, so mock results and the golden snapshot are unchanged.

Removed

  • The jev decision backend, a stub that only raised. The DECISION_MODELS registry and the ragbench.decision_models entry-point group stay the extension slot (see docs/extending.md); decision_model.provider: jev is now an unknown-provider error.

Fixed

  • OpenCode on OpenRouter is charged what OpenRouter billed. OpenCode prices its own tokens, reasoning included, at its price table's list price, and for an OpenRouter model that is the cheapest upstream's price ($0.07 / $1.40 for z-ai/glm-5.3). OpenRouter bills whichever of the model's upstreams it routes the request to, and most of GLM 5.3's charge $0.50–2.80 / $2–8.80. The cost OpenCode reported was therefore a floor: on the Task 26 benchmark it was an estimated quarter to a third of the bill. With an openrouter: model, the opencode preset now points OpenCode's OpenRouter base URL (in the workspace opencode.json) at a local relay (ragbench.rag_systems.openrouter_relay, 127.0.0.1 only). The relay forwards each request with usage: {include: true} and the openrouter: section's provider routing and reasoning effort, the same settings as RAGBench's own OpenRouter calls, so data_collection: deny now holds for the harness too. It records each response's billed usage.cost. That bill replaces OpenCode's cost, spread over the turns by tokens, and it also covers OpenCode's requests outside the turns, such as the session title. A stream cut off at the step cap is still read to its end for its cost. A response without a cost is logged, and the cost is then a lower bound. The published opencode_glm cost ($0.01023 a question) predates this fix and is kept as published (benchmarking external systems).
  • The LangChain and Haystack reference pipelines run live. Both sent temperature: 0, which GPT-6 models reject (only their default is accepted), so every live question failed; they now leave the temperature at the model's default, as RAGBench's own client ends up doing. The Haystack pipeline used OpenAIGenerator and PromptBuilder, which Haystack 3 removed; it is now the tutorial's ChatPromptBuilder + OpenAIChatGenerator. Their live paths had only been tested on fakes; the Task 26 probe found both.
  • ragbench estimate shows swept systems by their full name. The [generator=...] part of a sweep's name was read as console markup and dropped, so every swept row printed the same base name.
  • Prices re-audited for the release (2026-10-27), from OpenAI's and Anthropic's pricing pages and OpenRouter's catalog. gpt-6.1-terra ($1 / $5, $2 / $7.50 above 272K prompt tokens) is new in the price table and docs/models.md; the Anthropic rows were unchanged. On OpenRouter deepseek/deepseek-v4.1-flash is now 0.14 / 0.42 (was 0.048 / 0.19) and moonshotai/kimi-k3 input 0.189 (was 0.179); the bundled snapshot and PRICING_AS_OF are dated 2026-10-27. The published benchmark's costs are what its providers billed or those tokens at the prices of 2026-10-24, and stay as they were.

  • OpenRouter prices re-audited (2026-10-26). The bundled snapshot and docs/models.md now carry the current prices of deepseek/deepseek-v4.1-flash, z-ai/glm-5.3 (0.07 / 1.40) and moonshotai/kimi-k3 (output twice the old price).

  • A question that fails partway still charges what it spent. A question that raised after paid calls (an agent whose last call failed, a harness that timed out, a judge that failed after the answer) used to become a run_error row costing $0, so evaluation.max_cost_usd did not count that spend and the run's charged cost was too low. Each question now meters its paid calls (cache-wrapped LLM, embedding and decision models, and the turns a cli_agent harness ran before failing); the error row carries that cost, the spending cap charges it, and the run's charged cost includes it. Failed questions stay out of every mean.

  • A judge outage no longer fails the question. A judge call that still fails after the provider's own retries (rate limit, overload, timeout) leaves that sample without a verdict, like an unusable reply: in a panel the other judges score the answer; with no verdict at all the heuristic scores it with fallback: True and the error in call_error. A permanent error (credentials, no credit, a bad request) still fails the question.
  • Out of credit fails fast. OpenAI's 429 insufficient_quota was retried as a rate limit, six attempts with backoff on every call; it is now a permanent error that says to top up and resume.
  • The reference pipelines cannot slip mock answers into a live run. Without RAGBENCH_REFERENCE_LIVE=1 the LangChain, LlamaIndex and Haystack reference pipelines answer with RAGBench's offline models; in a live run those answers were scored, and priced at gpt-6-luna, as the real pipeline's. A python_callable that uses the offline models in a live run now fails with a message naming the variable (under --mock they are what it runs on).
  • The LangChain and Haystack reference pipelines embed through RAGBench's client, as the LlamaIndex one already did: the same text-embedding-3-small vectors, now under the run's limits.tokens_per_minute, retries and embedding cache. LangChain's own client sent 1000-chunk batches with two retries, enough to trip OpenAI's embedding rate limit on a corpus the size of MultiHop-RAG and fail the whole system at ingestion. Their chat models are still the frameworks' own.
  • Resuming notices changed openrouter: settings. Reasoning effort and provider pinning change the answers of openrouter: models, so a non-default openrouter: section is now part of each checkpoint's context hash (the default section leaves existing hashes unchanged).
  • graph_rag (global): a map reply scoring a point Infinity or 1e999 raised OverflowError and failed the question; scores are clamped as numbers first (NaN counts as missing). The LLM reranker books the call behind an unusable reply as rerank_cost, as it does a usable one (it was llm_cost).
  • ragbench report on a published run folder keeps the stage tables. A folder without per_question_results.jsonl (what docs/benchmarks/*-run/ keeps) rebuilt leaderboard.md without its "Cost by stage" and "Latency by stage" tables. Every run now writes them to stages.json as well, and a rebuild reads that file when the per-question rows are absent (or says the tables were left out).
  • An unknown cost never wins on cost. An external system that reported no cost is recorded at $0; the recommendation now treats that cost as unknown, and an unknown cost counts as the worst of a tied group when choosing among systems of equal quality, instead of as the cheapest.
  • winner.yaml keeps the run's openrouter: section (reasoning and provider pinning), so the winner runs as it was measured. Markdown tables escape a | inside a cell (a question containing one split its row). lookup_table matches select columns ignoring case, as it already did for where.
  • The LlamaIndex adapter works on shared Python installs. LlamaIndex keeps the NLTK data its sentence splitter needs inside site-packages; where that tree is group-writable (a CI runner with umask 002), NLTK refuses to read or download there and every question failed with a LookupError (the frameworks CI job's llamaindex_default at 100% errors). ragbench.adapters.llamaindex now points NLTK_DATA at a private cache directory (<user cache>/ragbench/nltk_data, mode 0700) when the default location is not private; an NLTK_DATA you set is always kept.
  • The llamaindex extra installs again. llama-index-llms-openai requires openai<3 in every release (latest 0.8.3, 2026-10-12), which no RAGBench install can satisfy since the openai>=3.37 floor; the CI frameworks job could not install. The extra is now llama-index-core alone, and the LlamaIndex reference pipeline (examples/llamaindex_default.py) reaches gpt-6-luna and text-embedding-3-small through RAGBench's own client, wrapped in LlamaIndex's CustomLLM (a chat model) and BaseEmbedding; its splitter, index and prompts are LlamaIndex's as before.
  • Repeated trials (evaluation.trials): run_summary.json records trials (only when above 1, so one-trial runs are unchanged); the report's columns read pass@3 / pass^3 instead of pass@k; tool_usage.csv questions_using and the routes table's questions count questions, not answers; ragbench label pools every trial's documents at their best rank (it kept only the last trial's) and lists a flagged system once.

[0.3.0] - 2026-10-01

Breaking

  • Python 3.11+ is now required (3.10 reaches end-of-life in October 2026 and blocks current numpy/pandas). CI covers 3.11-3.13 and runs 3.14 as best-effort.
  • chunker.type: token now counts real tokens (tiktoken o200k_base), not whitespace words. The old behaviour is the new word chunker (tokenish stays an alias of it). All shipped configs/*.yaml moved to type: word, so their results are unchanged and comparable with 0.2.0; a config of your own that says token (or omits chunker:, where token is the default) now produces different chunks (roughly 25% fewer words per chunk at the same chunk_size), so its results are not comparable with 0.2.0. markdown also measures its size in tokens now and keeps headings in the chunk. Without network access tiktoken cannot load its vocabulary, so token-based chunkers fall back to a deterministic approximate tokenizer (with a warning) instead of failing; parent_doc keeps word-based chunking.
  • Retrieval metrics are now measured on a deeper ranking than the LLM context. Systems used to return only top_k (5) chunks, so every @10 metric (Recall@10, MRR@10, nDCG@10) silently equalled its @5 value. Systems now return max(k_values) ranked chunks, and the generator still receives only its configured top_k (or evaluation.context_k). Numbers labelled @10 are not comparable with 0.2.0 results; @5 values are unchanged. parent_doc searches enough child chunks to fill the deeper parent ranking.
  • The default vector backend is exact NumPy, and chromadb is no longer a core dependency (install ragbench[chroma]). Asking for a backend whose library is missing now raises an error naming the extra instead of silently running NumPy search (before, a "chroma" result could be NumPy: any Chroma failure, including its 5461-row insert limit, was swallowed). The fallback_reason metadata key is gone. vector_store: in_memory still loads as a deprecated alias of numpy, and persist_directory with a backend that cannot persist is now a config error. All shipped configs/*.yaml use numpy, so results for Chroma-based configs can differ from 0.2.0 (exact instead of approximate search; mock results for the vector systems are unchanged).

Added

  • pricing: config section to add or override per-model prices; dated model snapshots now resolve to their base model's price.
  • Retry with exponential backoff, jitter, Retry-After support and a 120 s timeout for OpenAI chat and embedding calls, with an error taxonomy (RateLimitError, TransientModelError, PermanentModelError, ModelInitError).
  • run_summary.json now records mode (mock/live), models_used, unknown_priced_models and pricing_as_of; reports show a banner for mock runs and for models with no registered price.
  • evaluation.context_k, retrieval_depth, primary_k and max_error_rate options; run_manifest.json (versions, git commit, config and dataset hashes, models, mode, timings); n_ok / n_error columns in metrics_summary.csv; an in_context flag on every retrieved chunk in per_question_results.jsonl.
  • A question that raises no longer aborts the run: it is recorded as a run_error row and excluded from all means. A system whose ingestion fails is marked failed while the other systems finish. The run exits non-zero (after writing all results) when a system exceeds max_error_rate.
  • Component registries (ragbench.registry): systems, chunkers and rerankers register by name, unknown names fail with a did-you-mean suggestion, and third-party packages can add components through the ragbench.systems / ragbench.chunkers / ragbench.rerankers entry-point groups.
  • Typed system configs: every retrieval:, chunker: and llm_features: section is validated against a Pydantic option model, so a typo such as candidate_top fails at load time with a suggestion instead of being ignored. Systems also declare a SystemSpec (title, summary, best-for, cost/latency profile).
  • docs/systems.md, docs/cli.md and the README system table are generated from the code by scripts/generate_docs.py; CI and a test fail when they are stale.
  • A golden mock-run snapshot (tests/golden/mock_metrics_v3.json, python scripts/update_golden.py) that guards refactors.
  • Step tracing: every answer carries an ordered trace (AnswerResult.steps, RetrievalResult.steps) of what the system did: retrieve / rerank / LLM / generate steps with latency, tokens, cost, and short input/output previews. Steps are persisted as steps in per_question_results.jsonl, each question's stage costs appear as stage_<kind>_cost columns in cost_breakdown.csv (plus a new tool_cost field), and leaderboard.md gains "Cost by stage" and "Latency by stage" tables. Step costs always add up to the answer's cost; anything a system fails to trace shows up as an explicit untracked step.
  • Persistent disk cache (cache: config section, --no-cache, ragbench cache stats|clear): live runs cache temperature-0 LLM responses (generation, rewriting, reranking, judging) and corpus embeddings in .ragbench_cache/cache.sqlite3 (SQLite, WAL), so re-running an unchanged benchmark or resuming after a crash costs almost nothing. Cache hits are still charged to each system at the current standalone price, cached LLM calls replay their original latency, and run_summary.json gains a cache block with hits, avoided spend (saved_cost_usd) and real_spend_usd. Mock runs never use the disk cache. A corrupted cache file is ignored with a warning.
  • Parallel execution engine (ragbench.runtime): evaluation.system_workers runs systems concurrently (each still ingests once), evaluation.ingest_workers parallelizes llm_heavy's per-chunk LLM enrichment and embedding batches, and --system-workers sets the former from the CLI. Output rows, scores and their order are identical to a sequential run. On a simulated I/O-bound workload (8 systems, 24 questions, ~50 ms per call) wall time drops from 13.9 s to 0.74 s with system_workers=8, max_workers=4; mock runs are CPU-bound and gain little.
  • Rate limiting: a limits: section (max_concurrent_requests, requests_per_minute, tokens_per_minute) throttles OpenAI requests across all threads; every retry attempt counts, cache hits do not.
  • Latency probe: live runs re-ask evaluation.latency_probe_questions (default 5) questions per system one at a time with the cache off, after the parallel pass. The Latency column shows the clean mean, a p95 column is added, and metrics_summary.csv gains latency_ms_p50, latency_ms_p95, latency_source and avg_latency_concurrent_ms. Probe spend is reported as probe_cost_usd (in run_summary.json's new execution block) and included in real_spend_usd. Mock runs skip it.
  • Progress callbacks are serialized, so listeners never need their own locking.
  • Multi-provider models. Every model setting is now a provider:model ref (a bare name is still an OpenAI model): anthropic:claude-… (native Messages API; pip install 'ragbench[anthropic]'), openai_compatible:<endpoint>/<model> for any OpenAI-compatible server (Ollama, vLLM, LM Studio, OpenRouter, Together, …) defined under a new top-level providers: section with its own base_url, api_key_env and limits, and local:<hf-model> embeddings through sentence-transformers (pip install 'ragbench[local]'; chosen over fastembed because it runs any Hugging Face model and has wheels for Python 3.11-3.14, which fastembed's onnxruntime lacks on Apple Silicon 3.14). Refs are validated when the config loads. Providers are pluggable through the new LLM_PROVIDERS / EMBEDDERS registries and the ragbench.llm_providers / ragbench.embedders entry-point groups.
  • Tool-calling interface: LLM.generate(messages, *, temperature, max_tokens, json_mode, tools) takes OpenAI function schemas and returns LLMResult.tool_calls / finish_reason; LLMResult.assistant_message() feeds a turn back. Anthropic messages, tool_use / tool_result blocks and tool schemas are translated both ways; the disk cache stores tool calls and keys on provider and tools. MockLLM accepts tools= (it never calls one yet).
  • A live run now refuses to start (before spending anything) when a model it uses has no credentials while another does, instead of mixing real and mock models. A config is live when any model it calls is reachable; a config using only providers: endpoints or local: embeddings needs no API key. ragbench reports a missing optional extra as a one-line error naming the pip install to run.
  • Cross-encoder reranker (reranker: cross_encoder, optional pip install 'ragbench[rerank]'): scores every (question, chunk) pair with a Hugging Face cross-encoder (retrieval.reranker_model, default BAAI/bge-reranker-base; CUDA, Apple MPS or CPU is picked automatically). Available in rerank, hybrid_rerank and llm_heavy. The model loads on first use and is shared by every system in the run, it costs $0, and a missing extra fails when the system is built with the pip install to run. Mock runs substitute the TF-IDF reranker so --mock stays offline. No hosted reranking API is built in (see docs/extending.md).
  • rag_fusion and decompose systems (LLM query strategies). rag_fusion has the LLM write num_queries alternative queries, searches each (plus the original) with retriever: hybrid|vector, and merges the rankings with RRF; unlike hybrid's local multi_query the variants come from the model. decompose splits a complex question into at most max_subquestions sub-questions, searches each (each is a traced retrieve step), optionally in sequence with earlier findings fed forward (sequential: true), and answers from a prompt that lists every sub-question with the passages it ranked highest and their doc IDs. An unusable (non-JSON) planner reply falls back to the original question and flags the step fallback: true. MockLLM answers both planner prompts deterministically, so --mock runs stay offline; BaseRAGSystem gained a _generate_from_retrieval hook for prompts that need retrieval metadata.
  • Agent loop and the first agentic systems. ragbench.agents.AgentLoop is a bounded controller (AgentBudget: max_steps, max_cost_usd, max_tokens) that runs a policy until it finishes, aborts or the budget says stop; a spent budget ends the loop with termination: budget and a best-effort answer, never an error, and a policy that misbehaves cannot loop past the step cap. corrective (CRAG-style) retrieves, has the LLM grade the chunks, and when too little is relevant rewrites the query and widens the search (expand_to: hybrid|full_doc|none, max_rounds), with an optional answer groundedness self-check that regenerates once. iterative (IRCoT-style) searches, asks the LLM what is known and missing, and searches for the missing part until it says DONE or max_hops is reached. Every answer carries an agent block (steps, hops, termination, budget_exhausted, llm_calls, sufficient), persisted per question in per_question_results.jsonl; metrics_summary.csv gains avg_steps, avg_llm_calls and budget_exhausted_rate for agentic systems only. The loop's stop rules and the new metrics are described in docs/methodology.md.
  • Tools framework (ragbench.tools, docs/tools.md). A tool is a typed, traced, budgeted function an agent can call; every call becomes a tool step (arguments, output, latency, tool_cost) and never raises or hangs. Built in: calculator (whitelist AST evaluator, no eval; 9**9**9 and __import__('os') are error results), date_calc (differences, offsets and weekdays against the run's frozen evaluation.tools_now), list_documents, read_document, corpus_grep (keyword or regex search; catastrophic regex shapes such as (a+)+$ are refused up front because Python's re cannot be interrupted), lookup_table (rows of CSV/JSON tables) and search (the system's own retriever). Your own functions become tools from their type hints and docstring: tools: [{name: my_tool, path: "pkg.module:function"}]. New config: a per-system tools: list (checked when the config loads; unknown names get a did-you-mean hint, and a system that does not use tools rejects it), a top-level tools: section (allow, allow_network, timeout_s, max_output_chars) and evaluation.tools_now; tools with side effects must be listed in tools.allow, and no network tool is built in. requires_tools on questions is now validated against the tool registry. AgentBudget gains max_tool_calls. The run reports avg_tool_calls, tool_error_rate and required_tool_used_rate in metrics_summary.csv for systems that have tools, plus a per-tool tool_usage.csv. SystemSpec gains supports_tools; the tool table in docs/tools.md is generated from the registry.
  • Tool-using agents: agent_search and grep_agent. agent_search is a function-calling agent with a search tool (hybrid or vector, the same retrieval as the plain systems) plus its tools: list; grep_agent is index-free (no chunking, no embeddings, no vector store) and only lists, greps and reads the raw documents. Both run inside AgentLoop, so max_steps, max_tool_calls, max_cost_usd and max_tokens apply; when a limit ends the loop the agent is asked to answer from what it found (best_effort in the agent block) instead of failing. An unknown tool or a failing tool is an error observation the model can recover from. agent_mode: react_json drives models without function calling through JSON actions (fenced or garbled JSON is parsed leniently). Retrieval is scored on the passages the tool calls returned. MockLLM has a deterministic tool policy (calculator for arithmetic questions, date_calc for date questions, otherwise search or grep), so --mock runs exercise the whole loop. configs/all.yaml runs agent_search with tools: [] and with [calculator, date_calc, corpus_grep] so the tool axis is benchmarked by default. AgentLoop no longer records a finish marker when Finish carries the answer, and a step can be relabelled (a model turn that turns out to be the answer is the generate step).
  • adaptive router system. Each question is sent to the best of several pipelines defined inline under retrieval.routes (every route is a full system config, inheriting the system's models). router: heuristic is free and rule-based: exact identifiers (HS-4127, quoted phrases, UUIDs, versions) go to lexical, arithmetic, percentages and date calculations to computation, comparisons and questions with several linked parts to multi_hop, everything else to default, and a role that is not configured falls through to the next match and finally default. router: llm asks the generator model for {route, reason} (an unreadable or unknown choice falls back to default and is flagged) and allows route names of your own with route_descriptions. Routes, types, options, tools and model refs are validated when the config loads. The router is a route step, the chosen pipeline's steps are copied into the trace tagged with the route, and costs add up exactly. New outputs: route and routing_hint per question in per_question_results.jsonl, routes.csv (per route: questions, share, mean answer score and cost, and precision/recall when questions carry the new optional routing_hint) and a route_accuracy summary column for systems that route.
  • Pluggable vector backends (retrieval.vector_store, registry VECTOR_BACKENDS, entry-point group ragbench.vector_backends): exact numpy (default), faiss (exact) and faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma with ragbench[chroma], and qdrant (embedded local mode, exact) with ragbench[qdrant]. RetrievalResult.metadata["vector_backend"] always names the backend that actually ran. Chroma inserts are now sent in batches no larger than the client's get_max_batch_size().
  • Chunking suite: new chunkers recursive (paragraph, line, sentence, word boundaries), sentence, semantic (breaks where the meaning moves between sentences; uses the system's embedding model and charges that cost to ingestion) and a rewritten markdown (heading path in metadata["heading_path"], prefix_heading, small sections merged up to min_chunk_size, headings inside code fences ignored), plus the common fields min_chunk_size and prefix_title. Every chunker shares one base class that guarantees no empty chunks, honest start_char/end_char and deterministic ids. ragbench chunk-preview --docs PATH --chunker '{type: markdown}' shows chunk counts, token-size statistics and the first chunks (--json, --doc, --limit, --mock) before you spend anything on a benchmark.
  • Document loaders rebuilt around a LOADERS registry (one loader per file extension; entry-point group ragbench.loaders): .docx (pip install 'ragbench[docx]'; headings become Markdown headings, tables become text), .csv / .tsv / .json / .jsonl (one document per row or record, with column names as context; dataset.tabular.text_columns / id_column; ids are <file id>#<row>), HTML parsed properly with the standard library (title, nested scripts/styles, comments, entities, paragraph breaks preserved), and PDF page_spans so every chunk carries its page (and page_end). Files are decoded with BOM detection and a Windows-1252 fallback instead of crashing on non-UTF-8 text. New dataset.include / exclude globs, a .ragbenchignore file (gitignore-style), dataset.max_file_mb (default 25) and dataset.on_error: raise | skip; skips and encoding fallbacks are reported by inspect-dataset (new --include, --exclude, --on-error) and as document_warnings in run_summary.json.
  • Per-provider rate limits: the top-level limits: applies to OpenAI and Anthropic separately, and each providers: endpoint takes its own limits:.
  • Demo dataset v2: 60 documents and 163 questions over 12 categories (paraphrase, exact identifiers, multi-hop, comparison, aggregation, numeric reasoning, date arithmetic, temporal conflicts, near-duplicate distractors, long-context, unanswerable), with graded qrels and versioned, near-duplicate, table, code and long-handbook documents. In mock mode Recall@5 now spans 0.59 to 0.76 across the eight systems instead of clustering in a narrow band. Described in docs/dataset-format.md.
  • Question.requires_tools (optional, metadata only) names the built-in tools a question needs; the demo marks its numeric and date questions.
  • Three new systems (11 in total): no_retrieval (the floor baseline: the model answers from its own knowledge, so its hallucination rate shows what retrieval buys), full_context (the ceiling baseline: whole documents, most relevant first by BM25, up to context_token_budget; truncation is deterministic and recorded as truncated: true, and you pay for every token) and sentence_window (search single sentences, return each match with window sentences either side; overlapping windows merge).
  • MMR diversity for vector and hybrid (diversity: mmr, mmr_lambda, mmr_candidates): maximal marginal relevance over the candidate pool, recorded as a rerank step. mmr_lambda: 1.0 reproduces the plain ranking. configs/all.yaml gained the three systems and a vector_mmr variant.
  • SystemSpec.retrieves (default True) and chunker=None. A system with retrieves=False gets blank retrieval metrics (— in the console, Markdown and HTML reports, empty cells in the CSVs), is never classified as a retrieval miss, and stays out of the qrels audit.
  • Two ingestion-time LLM systems (13 in total): contextual (Contextual Retrieval: an LLM writes a one or two sentence context for every chunk from its document; the context is prepended before embedding and BM25 indexing, hybrid RRF retrieval runs on top, and answers and judging use the original chunk text) and hierarchical (an LLM summary per document; the question is matched against title+summary cards first, then chunks are scored only inside the top docs_k documents, chunks_per_doc from each). Their ingestion LLM spend shows up on the system's ingestion row in cost_breakdown.csv; calls are temperature 0, so the disk cache makes re-runs free. One failed call degrades that chunk or document (counted in the ingestion metadata); if every call fails, ingestion raises.
  • MockLLM.register_responder(predicate, respond) / unregister_responder(name): the mock's prompt matching is now a table instead of an if-chain, so new mock-safe LLM steps no longer edit one function. Existing behaviour is unchanged.
  • Metrics v2 and docs/methodology.md, the one place for metric definitions, judge design, cost accounting, fairness rules and limitations (the README's Metrics and Cost warning detail moved there). New metrics_summary.csv columns: deterministic answer metrics exact_match, token_f1 (against reference_answer) and keyword_recall (against expected_keywords); context metrics context_recall and context_precision, scored on the chunks the generator was actually given rather than on the whole ranking, so a system that ranks the evidence 7th and reads 5 chunks no longer looks as good as one that shows it to its LLM; abstention metrics abstain_precision, abstain_recall and false_refusal_rate; and judge_fallback_rate. The same values are per question in answer_metrics.csv and per_question_results.jsonl (answer_metrics, context_metrics, answerable, refused). A metric a question cannot have (no reference, no keywords, no retrieval) is missing, not 0. The leaderboard gains F1 and Ctx recall columns.
  • Judge hardening (evaluation.judge: section with model, samples, temperature, independent; judge_model keeps working as a shorthand). samples: N averages N independent judgments and stores the spread (answer_score_variance); the report warns about self-preference when the judge is also a system's generator (silence it with independent: false). Judge prompt v2 has written 0 to 5 rubric anchors for each axis, tells the judge to ground verdicts only in the reference and the supplied context, and stores prompt_version with every judgment. A reply is accepted only if all five scores are present and numeric (before, a missing axis silently counted as 0); an unusable reply is retried once with a format reminder, and if that fails the heuristic scores that one question, flagged fallback: true. Fallbacks are counted per system (judge_fallback_rate) and shown as a warning in the reports, and the judge records judge, parse_error and retries per question. Retries and extra samples are charged to the judge's cost.
  • Statistics and efficiency metrics (docs/methodology.md#statistics). The leaderboards show 95% percentile-bootstrap confidence intervals (mean [lo, hi]) for recall, MRR, nDCG, answer score and faithfulness, and metrics_summary.csv gains <metric>_ci_lo / <metric>_ci_hi. New outputs: significance.csv (every system against a baseline on the same questions, for answer score and recall: mean difference, interval, p-value, Holm-Bonferroni-adjusted p-value, a better / worse / no clear difference verdict and wins, ties and losses), stats.json and pareto.json. cost_per_correct (all spend on a system's questions divided by the answers scoring at least 4; blank when none did) and pareto_optimal (no other system is at least as good on answer score, cost and latency and better on one) are summary columns; leaderboard.md adds $/correct and Pareto columns and a significance table, report.html shows the intervals and marks Pareto-optimal systems. Configure with evaluation.stats (n_boot, seed, baseline, default the cheapest system; an unknown baseline is a config error with a suggestion). Everything is seeded and uses numpy only, so identical results give identical intervals. Latency percentiles now come from ragbench.evaluation.stats.latency_percentiles.
  • Recommendation engine (ragbench.selection, docs/methodology.md). Every run now ends with recommendation.md, recommendation.json and, when a system qualifies, a runnable winner.yaml (the winner's system block with the providers, tools and limits it needs; replace its dataset: with your documents). Systems that break a constraint are excluded and the report says which one; of the rest, every system whose answer quality is not significantly worse than the best (paired bootstrap, Holm-adjusted) is tied with it, and the tie is broken on cost and latency by the profile's weights, then on the fewest model calls, so a lead inside the noise is never bought. Four profiles (balanced, max_quality, cheapest_acceptable, lowest_latency), per-category winners, the Pareto front among the feasible systems, and a rationale generated from the data ("a vs b: +0.12 answer score (95% CI +0.04 to +0.21, significant), 1.9× cost"); a mock run or fewer than 30 questions carries a caveat, and when nothing qualifies there is no winner and the closest miss is named. Configure with the new selection: section (profile, constraints, weights; constraints: max_cost_per_question, max_latency_ms_p95, min_faithfulness, min_answer_score, max_ingestion_cost, require_local_models, require_no_network). New command ragbench recommend --run <dir> repeats the decision on any finished run with other constraints (--profile, --max-cost, --max-latency, --min-faithfulness, --min-answer-score, --max-ingestion-cost, --local-models, --no-network, --export winner.yaml; exit status 1 when nothing qualifies), and compare / run print a Recommendation panel after the leaderboard. A recommendation that cannot be written never fails a finished run.
  • Config sweeps, presets, estimate and a budget guard (docs/configuration.md). A system with a sweep: (axes such as chunker.chunk_size, retrieval.reranker, models.generator and tools, each a list of values) expands into one system per combination, named hr[chunk_size=300,reranker=cross_encoder] in a fixed order; an invalid axis lists the valid ones, a sweep may not expand beyond 100 systems, and the run directory's config.yaml records the expanded systems (a config without sweeps is still copied verbatim). --preset quick|standard|thorough|agentic replaces a config's systems with a ready-made list (the dataset from --docs / --questions / --qrels or the config), and --systems a,b runs a subset (a sweep's base name selects its variants); both work with run, compare, evaluate and estimate, and --config is optional when a preset is given. New ragbench estimate: it runs each system on the mock models (all of the corpus, a sample of the questions), measures prompts, call counts and embedding volume, and prices them with the configured models' rates, printing ingestion, query and judge cost per system, the total and a rough time, with the price-table date and warnings for prices older than 90 days, unpriced models and agentic systems (which loop longer on real models); against a mock run priced as live it lands within a few percent. A live run / compare estimates first and, above evaluation.cost_confirm_threshold_usd (default $1), asks for confirmation, or refuses when no one can answer or CI is set, unless --yes is given. evaluation.max_cost_usd caps what a run may be charged: once it is reached no new question or system starts, the results of the systems that finished are written (the others' answers go to per_question_partial.jsonl and stay out of the leaderboard and recommendation), run_summary.json gains a budget block, and the command exits with status 1 and says what finished. The mock models count their calls and tokens for this (ragbench.models.usage).
  • Dataset onboarding (docs/dataset-format.md). ragbench init [DIR] --docs PATH [--questions FILE] [--preset ...] writes a runnable ragbench.yaml (the preset's systems, the judge on, max_cost_usd: 5.0 as a first-run safety net) and, when you have no questions yet, a questions.jsonl of placeholder questions built from your document titles; it refuses to overwrite a config without --force and never overwrites questions. ragbench import --format csv|beir|qa-md converts a CSV of questions, a BEIR dataset (documents become one Markdown file each, loader-safe ids with an id_map.json, graded qrels) or Markdown Q: / A: pairs. inspect-dataset is now a profile: document size distribution (tokens p50 / p95 / max) and a language guess, question balance and answerable share, label coverage, suggested chunk sizes (from where the corpus's text sits, token-weighted), the projected cost of the standard preset (--no-estimate skips it), and warnings for near-duplicate documents, questions that appear verbatim in a document (leakage) and corpora small enough that full_context will likely win.
  • Label-free mode. Questions with no relevant_doc_ids and no qrels run: retrieval and context metrics are blank for them (never 0) while answer quality, cost, latency and the recommendation work as usual. Question gains an optional answerable: bool (absent keeps the old rule: answerable exactly when relevant_doc_ids is non-empty); a dataset with no labels at all treats questions that do not set it as answerable. Validation warns instead of failing, and run_summary.json records dataset: {questions, labeled_questions, label_free}.
  • Synthetic questions and pooled labels (docs/dataset-format.md). ragbench generate-questions --docs PATH --out questions.jsonl writes --n questions in a configurable --mix of single-hop, multi-hop (from BM25-neighbouring document pairs), paraphrased (avoiding the document's vocabulary), numeric and unanswerable ones (rejected when their key entity occurs anywhere in the corpus), with each category within one question of its share, documents sampled stratified by length, near-duplicates dropped, a reproducible --seed, and every row flagged metadata: {synthetic, needs_review, generator}. It prints a cost estimate and asks before a live run above cost_confirm_threshold_usd (--yes, --max-cost), caches replies on disk, and never replaces --out without --force. --mock (or a missing key) writes deterministic template questions flagged mock. ragbench label --run RUN_DIR pools every system's top --top-k documents per question, has an LLM grade each (0-3, cached, confirmed before spending), and writes qrels.proposed.jsonl (graded pairs only) and qrels_review.md (disputed labels, possible missing labels, labeled documents nobody retrieved, the run's possible_qrels_gap cases, a pooling-bias note); --apply also writes qrels.merged.jsonl in which your labels always win. Your own question and qrels files are never modified. inspect-dataset counts questions still flagged needs_review. See methodology for the pooling bias.
  • ragbench auto: one command from a folder of documents to a decision. ragbench auto --docs ./my_docs [--questions q.jsonl] [--preset standard] [--profile balanced] [--max-cost 2.00] [--yes] [--mock] [--open] profiles the documents, writes --n-questions synthetic questions when you bring none (flagged needs_review), estimates the cost and asks before a live run above the confirmation threshold (refusing, not hanging, when nobody can answer), runs the preset, and prints the recommended system with the paths of winner.yaml, report.html and recommendation.md; everything lands in one results/auto_<time>/ directory with an auto.json that remembers how it was started. --max-cost is the budget for the whole command (what writing the questions cost is deducted from the run's cap). It is built from the existing pieces: the question generator, estimate, the run, the recommendation.
  • Resumable runs. A run given its own directory (what auto does) checkpoints every system that finishes cleanly to checkpoints/<system>.json, with the latency probe's measurements once they exist. ragbench auto --resume RUN_DIR loads the checkpoints whose context still matches (the system's config, every setting that changes its results, the dataset, the run mode; speed and budget settings do not count) and runs only the rest, so after a crash, Ctrl-C or a budget stop nothing finished is paid for again. A system that had failed questions is never checkpointed, so resuming retries it; a changed system, setting or dataset re-runs what it affects. The spending cap counts what the restored systems already cost. BenchmarkEvaluator / run_benchmark take run_dir=; ProgressListener gains system_restored.
  • Report v2 (docs/methodology.md). report.html is rebuilt as a decision document and generated from the files in the run directory (report_data.json is written next to it, and the same data is embedded in the page), so a report can be regenerated for any past run. Sections: run banner (mock or live, dataset, spend, versions, run warnings and a notice for unreviewed synthetic questions), recommendation hero with the winner, its ties, constraints and a copyable winner.yaml, quality-against-cost scatter (inline SVG, bubble = latency, Pareto frontier, emphasis colors instead of one hue per system), leaderboard with confidence whiskers, significance markers against the baseline, sorting and a column picker, category heatmap (answer score or recall), stage-cost stacked bars and p50 / p95 latency bars, agent / tool / routing panel, failure distribution, label audit and a reproducibility footer. Light and dark themes (follows the OS, with a toggle), a print stylesheet, texture instead of hue in print and forced-colors mode, keyboard-focusable marks with tooltips, a table view for every chart, no external resources, escaped text. The template, stylesheet and script live in reporting/templates/ (shipped as package data), charts are pure functions in reporting/charts.py. The categorical palette was checked with the dataviz validator in both themes.
  • Question explorer in the report. report.html gains a "Questions" section for debugging a run without opening per_question_results.jsonl: a table of every question with search, category, system, failure-type and "score below" (answer score, correctness or faithfulness) filters and a "systems disagree" switch, and a row that opens into the systems side by side: reference answer, each answer, the judge's scores and reasoning, the retrieved contexts (green for labeled-relevant documents, amber for unlabeled ones the label audit found probably good, dashed for chunks the generator never saw, with page and heading) and the step trace with latency, cost and tool calls. Keyboard: ↑ ↓, Enter, Esc, /. New report: {max_embedded_mb: 4} setting caps the embedded data; above it the page keeps a reduced copy and the complete data goes to report_questions.json, which the page loads by itself when served over http. Retrieved chunks in per_question_results.jsonl now carry page and heading when the chunker recorded them. All text is inserted as text, never as HTML (a test feeds it <script> in answers, questions, reasoning, contexts and traces). The page now has three scripts (report data, question data, behavior). The demo report's non-explorer part stays under 600 KB; the explorer's data is capped by report.max_embedded_mb.
  • CLI UX and run management. ragbench runs [list|show RUN|rm RUN --yes|index] lists the runs in a results folder (date, name, mode, systems, questions, winner, cost; --json), shows one, deletes one (only a directory that looks like a run, never one that holds your current folder, and only with --yes when nobody can be asked) and writes results/index.html, a static history page with a link to every report, the winner of each run and a score-over-time line per system name. ragbench report RUN [--open] rebuilds leaderboard.md, failures.md, qrels_audit.md, the recommendation and report.html from the files in a run directory, byte for byte, without running anything. ragbench compare-runs A B shows per-system metric deltas and flags a regression when an answer-score drop is significant (the paired bootstrap on the questions both runs answered, Holm-adjusted; without enough shared questions it falls back to --min-drop and says it was not tested); --fail-on-regression exits with status 1 for CI. RUN is a path, a name under --results-dir, latest or a unique part of a name. ragbench doctor [--config FILE] checks Python, the required packages, each optional extra (with its pip install), whether each API key is set (the value is never printed), that the results folder is writable, the cache size and the price table's date (a warning after 90 days), and every model in the config against the price table. ragbench completion show|install replaces Typer's --install-completion flags.
  • --only a,b and --skip a,b choose the systems of a run, compare and estimate (--systems still works as the old name of --only); unknown names get a suggestion and skipping everything is an error. --json on run, compare, recommend, runs, compare-runs and doctor prints one JSON document on stdout, with progress and messages on stderr. ragbench --debug (or RAGBENCH_DEBUG=1) shows the traceback of a failure.
  • Errors are messages. A failure is a one-line message and an exit code instead of a traceback: 1 when the command ran and ended badly (a run stopped, no system qualifies, you declined), 2 when it could not start because of its input or setup. An invalid config is a list of problems, each with its path and, when the validator knew one, a suggestion (systems[0]: ... unknown option 'top_kk' then → Did you mean 'top_k'?). Messages now go to stderr. The errors live in ragbench.errors (RagbenchError, UsageError, ConfigError, CommandFailed).
  • Tests and CI hardening. tests/test_e2e_matrix.py runs every registered system and every chunker end to end offline (it covers a component the moment it is registered) and checks that every output is written, that no promised number is missing, that each question's step costs add up to its cost, and that two runs give the same answers. tests/test_report_snapshot.py pins the structure of report.html and report_data.json (tests/golden/report_structure_v1.json; RAGBENCH_UPDATE_SNAPSHOTS=1 accepts a change), tests/test_ci_config.py validates the workflow, and tests/conftest.py gained tiny_dataset, pair_dataset and scripted_llm fixtures that replace the dataset builders several test files had copied. The matrix found no defects in the 20 systems and 7 chunkers.
  • CI: separate jobs for lint and types, generated docs and the docs audit, tests on Python 3.11 to 3.14 with coverage (3.14 is advisory), the full suite with every optional extra, a CLI smoke test that runs ragbench auto on the demo documents and the history commands, a package build with twine check --strict and a check of the wheel's contents and a clean install, and an advisory pip-audit. Coverage has a floor of 90% ([tool.coverage.report] fail_under; measured 96.9%); pytest-cov and twine join the dev extras.
  • docs/choosing-an-architecture.md, a decision guide: which systems to try for a given corpus, question mix, budget and privacy need, how to read the report in order, and a worked example on the demo dataset. scripts/audit_docs.py (also a test) fails on broken links and anchors, option tables and system descriptions outside their owners, metric definitions outside the methodology, stale counts and removed model names.
  • docs/system-wiki.md: a step-by-step explanation, with diagrams, of how every RAG system, chunker, reranker, vector backend and agent loop works.

Changed

  • ragbench.cli is now a package (cli/{app,common,run,runs,report,doctor,dataset,auto,tools}.py); ragbench.cli:app and python -m ragbench.cli work as before. The helpers that used to be private in cli.py are public names in ragbench.cli.common and ragbench.cli.run (fail, confirm_spend, execute_benchmark, print_run_summary, ...).
  • Model defaults and price table refreshed (2026-10-01, from OpenAI's and Anthropic's official pricing and model pages). Defaults move to the newest generation in each tier: generator and judge gpt-5.4-nano → gpt-6-luna ($0.10 / $0.50 per 1M tokens), the llm_heavy generator gpt-5.4-mini → gpt-6.1-sol ($2 / $10; GPT-6 has no mid-size model, and it undercuts gpt-5.6-terra), embeddings stay text-embedding-3-small (no newer generation exists). The price table now holds gpt-6-astra, gpt-6.1-sol, gpt-6-luna, both v3 embedding models and claude-fable-5-1, claude-opus-5-5, claude-sonnet-5-5, claude-haiku-4-5; gpt-5.4-* and gpt-4o* rows are gone, so configs naming them now report an "unknown price" banner until a pricing: entry is added. Mock-mode results are unaffected. Costs from 0.2.0 runs are not comparable. The release re-verified every row, and every model named in configs/*.yaml, against the same official pages on 2026-10-01 (nothing was stale).
  • dataset: is now validated strictly: an unknown key (for example a typo such as on_eror) is an error instead of being ignored. HTML documents keep paragraph and list line breaks instead of being flattened to one line, so their text (and chunks) differ from 0.2.0; Markdown, text and reST documents load byte-for-byte as before.
  • The reranker code is now the package ragbench.models.rerankers (one module per reranker; from ragbench.models.rerankers import create_reranker, Reranker, RerankResult still works), and it no longer breaks when imported first in a fresh interpreter (a circular import through rag_systems.base). RetrievedChunk lives in ragbench.documents.schema and is re-exported by ragbench.rag_systems.base. The LLM reranker now records original_rank like the others; reranker_model on a reranker that loads no model is rejected at config load.
  • OpenAILLM / OpenAIEmbeddingModel moved to ragbench.models.providers.openai; create_llm / create_embedding_model gained a providers argument and keep their other arguments.
  • estimate_model_cost resolves provider:model refs, and treats local: / openai_compatible: models as $0 unless pricing: prices them.
  • Embedding cost is now charged per text from one token estimate whether the text was a cache hit or had to be embedded. Before, the first system to embed a text was charged the provider's token count and later systems a tokenizer estimate, so a system's ingestion charge depended on run order (and, with parallel systems, on thread timing).
  • Chroma index builds take a lock (systems may now be built on several threads).
  • evaluation.max_workers must be at least 1 (0 used to be silently treated as 1).
  • The in-process embedding cache is now the first tier in front of the disk cache (its semantics and evaluation.embedding_cache are unchanged); the HTML report header counts reuse from both tiers.
  • Invalid configs now fail at load time: unknown options, unknown type/chunker/reranker names, unknown top-level system keys, vector_store values other than chroma/in_memory, and duplicate system names used to be silently ignored or merged. The CLI reports them as a short list instead of a traceback.
  • Systems share chunker/embedder/vector-store construction through rag_systems/components.py; ragbench list-systems and the leaderboard's "Best For" column read each system's SystemSpec (no behavior change in retrieval or scores: verified against the golden snapshot).
  • rerank now also accepts top_k as an alias for final_top_k, like the hybrid systems.
  • Leaderboard columns (console, leaderboard.md, report.html) are generated from one shared helper and follow k_values/primary_k instead of hard-coding @5/@10; missing values show — instead of 0.000. leaderboard.md headers are now the short names used elsewhere (Answer, Faithful, $/Q, Latency) and it gains an Errors column. Failure classification and the qrels audit use primary_k and only count chunks the generator actually saw.
  • Dependency floors raised to current majors: openai 3, pandas 3, numpy 2.3, scikit-learn 1.9, pydantic 2.13, typer 0.27, rich 15, tiktoken 0.14, chromadb 1.5, pytest 9, ruff 0.16, mypy 2.3.
  • Dependabot groups routine pip minor/patch bumps into a single PR; pre-commit hooks updated.
  • mypy no longer pins python_version (numpy 2.5 type stubs need a 3.12+ parser); CI runs it on each supported interpreter.
  • A present OPENAI_API_KEY with a failing OpenAI client now raises ModelInitError instead of silently producing mock scores (strict=False on create_llm/create_embedding_model restores the old fallback).
  • The Responses API fallback is used only when the model cannot be called via Chat Completions, not after any error (which previously doubled billable requests). Chat requests send max_completion_tokens, and models that reject temperature are called without it.
  • Models with no registered price log a warning and are flagged in reports rather than silently costing $0.
  • The demo dataset's source of truth is now the data files in data/demo (the embedded Python strings are gone); a setup.py build hook ships them in the wheel as ragbench/_data/demo. ragbench demo copies them, reports files you modified instead of rewriting them, and --overwrite restores the originals.
  • The original 50 demo questions keep their ids and text; their categories were folded into the new 12-category set, q_027 got a self-consistent reference answer, and a few old questions gained grade-1 qrels for newer documents that also state the answer. The golden snapshot (tests/golden/mock_metrics_v3.json) was regenerated because the dataset changed.
  • Failure classification uses structured signals (what the generator saw, the ranking, refusal detection, the judge's scores and flags, answer_type) instead of year and capitalization regexes; the failure type names are unchanged. bad_reranking is now produced (the evidence was ranked, but below what the generator reads), format_error marks an empty answer, wrong_date / wrong_entity need answer_type: date / entity (person, name, organization) and missing keywords, and a refusal with only part of the evidence in context is insufficient_context rather than over_refusal. A question whose retrieval cut-off primary_k missed the evidence but whose generator read it (context_k above primary_k) is no longer a retrieval_miss.
  • Refusals are recognized by one shared detector that no longer treats any answer containing "not available" as a refusal (it also accepts common paraphrases such as "the documents do not contain…"). The heuristic judge now reports judge: heuristic (was heuristic_fallback) with fallback: false when it scores by design. report.html shows the same warnings as leaderboard.md, including the concurrent-latency one.
  • Judge prompt v3: the judge is told whether a question is answerable and, for an answerable question with no reference answer, grades against the retrieved context alone (v2 read a missing reference as "unanswerable"). Scores are not comparable with runs from v2 (prompt_version is stored per judgment). Reference-less questions in mock runs are scored by the heuristic judge on context support instead of 0.
  • inspect-dataset prints more and, by default, runs the cost projection (a few seconds; --no-estimate skips it).
  • A question labeled only through the qrels file (no relevant_doc_ids) is now answerable. It used to read as unanswerable (the judge expected a refusal and the abstention metrics counted it wrongly), which a qrels-only dataset, including the merged file ragbench label --apply writes, would have hit on every question.
  • report.html is written last, after the recommendation and the manifest, because it reads them; run_summary.json now stores the run's notices and, under dataset, the counts of synthetic, needs_review and mock questions. The in-memory write_html_report function is replaced by write_report(run_dir).
  • Docs consolidated. The README is now a pitch, a quickstart, what you get, the generated systems table, a documentation index and the roadmap (the 0.1.0 results table and its hand-placed quadrant chart are gone: they described six systems on an older dataset); docs/extending.md lists every plugin entry-point group once and gained a Tools section and a run-flow diagram; the maintainer repository setup moved from docs/github-setup.md into CONTRIBUTING.md, which now holds the pre-push checks and the documentation ownership table; docs/release-checklist.md has the model-and-price refresh procedure, the live-run step and the package checks. The roadmap lists what is actually left (GraphRAG, pgvector, multilingual evaluation, MCP tools, a dashboard server).
  • CI uses actions/checkout@v7 and actions/setup-python@v7 (superseding the two Dependabot PRs, which no longer applied cleanly to the expanded workflow).

Deprecated

  • ragbench evaluate is deprecated: it still works, prints a deprecation notice, is hidden from --help and docs/cli.md, and will be removed. Use ragbench run (same options). compare stays as an alias of run.

Fixed

  • Two runs started within the same second used the same output directory and overwrote each other (plausible now that cached re-runs are fast); run ids get a numeric suffix when the directory exists.
  • Token estimation resolved tiktoken.encoding_for_model on every call and always failed for new model names, silently using a word-count heuristic; the encoding is now resolved once per model with an o200k_base fallback.
  • Leftover "RAGLab" naming in LICENSE and .env.example; stale version and release date in CITATION.cff.
  • pyproject.toml declared the license as a TOML table and with a license classifier, which setuptools deprecates and will stop building after 2027-02-18; it is now the SPDX expression MIT plus license-files (build requirement setuptools>=77).
  • docs/extending.md showed a loader plugin whose entry-point name had no leading dot, so it would never have matched a file extension; docs/configuration.md said every system except parent_doc has a chunker: section (six more have none) and listed vector-backend systems by hand.
  • The "estimated cost is above the confirmation threshold" error told ragbench run users to pass --max-cost, a flag only some other commands have; it now names evaluation.max_cost_usd as well. docs/release-checklist.md and docs/choosing-an-architecture.md made the same mistake.
  • The README results are now from a live run of configs/all.yaml on the demo dataset, kept with its result files in docs/example-run.md (ragbench 0.3.0, 1 Oct 2026, about $2.91 real spend) with refreshed report screenshots in docs/assets/; the 0.1.0 table was removed earlier in this release.
  • A clean pip install -e ".[dev]" could not run the tests or scripts/generate_docs.py: httpx (used by the wire-level tests) and click (used by the docs script) had only ever arrived through chromadb. httpx is now in the dev extra and the docs script uses the typer.Context that Typer ships instead of importing click.
  • Loading any .html / .htm file failed (TypeError: can only join an iterable) on Python releases whose HTMLParser keeps its own private _pending list: the HTML text extractor stored an integer under the same name. Its state now uses class-private names that cannot clash with the standard library's.

[0.2.0] - 2026-06-12

Added

  • Two new RAG systems, both dataset-agnostic and mock-safe:
  • hybrid_rerank: hybrid BM25 + vector RRF retrieval followed by a reranking pass.
  • hyde: Hypothetical Document Embeddings — the LLM writes a hypothetical answer passage used as the vector-search probe, fused with the raw-question ranking via RRF.
  • Shared embedding cache across systems in a run (evaluation.embedding_cache, on by default). Cache hits are still charged to each system at standalone prices so comparisons stay fair; actual API savings are reported in run_summary.json and the console.
  • Live progress display: per-system ingestion status and question-by-question progress bars during run / compare / evaluate.
  • Console leaderboard printed after each run with the best value per column highlighted.
  • Dataset validation: inspect-dataset now flags duplicate question ids, references to missing documents, qrels for unknown questions, relevant_doc_ids missing from qrels, and empty documents. Warnings are also embedded in run_summary.json.
  • PDF and reStructuredText document support (.pdf requires pip install 'ragbench[pdf]').
  • Weighted RRF for hybrid and hybrid_rerank (bm25_weight, vector_weight).
  • parent_doc child-score aggregation option (parent_score_aggregation: max | sum | mean).
  • Five paraphrase questions in the demo dataset that avoid document vocabulary, so semantic retrieval can differentiate from lexical matching (now 50 questions across 11 categories).
  • Configs configs/hyde.yaml and configs/hybrid_rerank.yaml; both new systems included in configs/all.yaml.

Changed

  • report.html redesigned: self-contained (no CDN), winner summary cards, sortable leaderboard, comparison bar charts, per-category quality, aggregated cost breakdown, light/dark mode.
  • BM25 retrieval no longer pads results with chunks that share no term with the query.
  • In-memory vector search uses partial sorting (argpartition) for top-k selection.

Fixed

  • llm_heavy rebuilt its reranker on every question; it is now constructed once at system init.
  • ragbench --version previously failed with "Missing command"; the option is now eager and works standalone.

[0.1.0] - 2026-05-09

Added

  • Evaluation-first RAG benchmark harness.
  • BM25, vector, hybrid, rerank, parent-document, and LLM-heavy RAG systems.
  • Chroma-backed vector search with in-memory fallback.
  • Mock mode for embeddings, LLM answers, and answer judging.
  • Retrieval metrics, answer judge, failure analysis, qrels audit, cost tracking, and reports.
  • Demo fictional insurance technology dataset.
  • Typer CLI with demo, run, compare, evaluate, inspect-dataset, and list-systems.