ragbench¶
RAGBench: evaluation-first RAG benchmark framework.
Exit codes: 0 success; 1 the command ran and ended badly (a run stopped, no system qualifies, you declined); 2 it could not start because of its input or setup (a flag, file, config, dataset, missing package or API key). Messages go to stderr, so a command with --json prints only JSON on stdout. --debug (before the command) shows the traceback of a failure.
Usage:
$ ragbench [OPTIONS] COMMAND [ARGS]...
Options:
--version: Show version and exit.--debug: Show the full traceback when a command fails (put it before the command:ragbench --debug run ...). [env var: RAGBENCH_DEBUG]--help: Show this message and exit.
Commands:
run: Run a config: every system on the same...compare: Run multiple systems from one config (the...estimate: Project what a run would cost and how long...recommend: Which system to deploy, from a finished...auto: From a folder of documents to a decision:...optimize: Propose → Probe → Compose: improve a...demo: Create or verify a bundled dataset: the...inspect-dataset: Profile a dataset: sizes and balance,...split: Split a dataset's questions into train /...generate-questions: Write questions for documents that have...label: Propose relevance labels for a run by...init: Start a benchmark of your own documents:...import: Convert a CSV, BEIR dataset or Markdown...chunk-preview: Show how a chunker cuts your documents:...list-systems: Print available RAG systems with their...cache: Inspect or clear the persistent cache of...completion: Shell completion for theragbenchcommand.report: Rebuild every report of a finished run...judge-audit: Calibrate the judge against a human: draw...compare-runs: How a later run moved against an earlier...contrast: Where two systems disagree: the questions...export-traces: Export a finished run's traces as...doctor: Check Python, packages, optional extras,...runs: List, inspect, remove and index the runs...models: List the models RAGBench has prices for,...datasets: Public benchmarks on demand: list them,...harness: Show a system's editable harness (prompts...library: The experience library: what `ragbench...
ragbench run¶
Run a config: every system on the same questions, then the report and a recommendation.
Usage:
$ ragbench run [OPTIONS]
Options:
-c, --config <path>: YAML config to run. Systems with asweep:are expanded into one system per combination.--preset <str>: Use a ready-made list of systems (quick, standard, thorough, agentic) instead of the config's own; the dataset comes from --docs/--questions or the config.--docs <path>: Documents folder for --preset.--questions <path>: Questions JSONL for --preset.--qrels <path>: Optional qrels JSONL for --preset.--only, --systems <str>: Only these systems (names or sweep base names, comma-separated or repeated), after sweeps are expanded.--skip <str>: Leave these systems out (names or sweep base names, comma-separated or repeated). Applied after --only.--mock: Force local mock mode even if OPENAI_API_KEY is set.--max-workers <int range>: Override evaluation.max_workers: questions answered at the same time within a system. [x>=1]--system-workers <int range>: Override evaluation.system_workers: systems evaluated at the same time. [x>=1]--no-cache: Do not read or write the persistent disk cache for this run.-y, --yes: Do not ask before a live run whose estimated cost is above evaluation.cost_confirm_threshold_usd.--run-dir <path>: Write the run into this directory instead of a new timestamped one. Running into it again resumes: systems that finished are kept, unfinished ones keep their answers and run only the missing or failed questions.--json: Print one JSON document on stdout when the run ends (progress and messages go to stderr), for scripts and CI.--help: Show this message and exit.
ragbench compare¶
Run multiple systems from one config (the same as run).
Usage:
$ ragbench compare [OPTIONS]
Options:
-c, --config <path>: YAML config to run. Systems with asweep:are expanded into one system per combination.--preset <str>: Use a ready-made list of systems (quick, standard, thorough, agentic) instead of the config's own; the dataset comes from --docs/--questions or the config.--docs <path>: Documents folder for --preset.--questions <path>: Questions JSONL for --preset.--qrels <path>: Optional qrels JSONL for --preset.--only, --systems <str>: Only these systems (names or sweep base names, comma-separated or repeated), after sweeps are expanded.--skip <str>: Leave these systems out (names or sweep base names, comma-separated or repeated). Applied after --only.--mock: Force local mock mode even if OPENAI_API_KEY is set.--max-workers <int range>: Override evaluation.max_workers: questions answered at the same time within a system. [x>=1]--system-workers <int range>: Override evaluation.system_workers: systems evaluated at the same time. [x>=1]--no-cache: Do not read or write the persistent disk cache for this run.-y, --yes: Do not ask before a live run whose estimated cost is above evaluation.cost_confirm_threshold_usd.--run-dir <path>: Write the run into this directory instead of a new timestamped one. Running into it again resumes: systems that finished are kept, unfinished ones keep their answers and run only the missing or failed questions.--json: Print one JSON document on stdout when the run ends (progress and messages go to stderr), for scripts and CI.--help: Show this message and exit.
ragbench estimate¶
Project what a run would cost and how long it would take, without spending anything.
Runs every system on the offline mock models (corpus indexing in full, a sample of the questions) to measure prompts and call
counts, then prices them with the configured models. Warns when the price table is old or a model has no price. With
--optimize SYSTEM it projects an optimization of that system instead: its probes, dev and test runs, and proposer requests.
Usage:
$ ragbench estimate [OPTIONS]
Options:
-c, --config <path>: YAML config to run. Systems with asweep:are expanded into one system per combination.--preset <str>: Use a ready-made list of systems (quick, standard, thorough, agentic) instead of the config's own; the dataset comes from --docs/--questions or the config.--docs <path>: Documents folder for --preset.--questions <path>: Questions JSONL for --preset.--qrels <path>: Optional qrels JSONL for --preset.--only, --systems <str>: Only these systems (names or sweep base names, comma-separated or repeated), after sweeps are expanded.--skip <str>: Leave these systems out (names or sweep base names, comma-separated or repeated). Applied after --only.--optimize <str>: Estimateragbench optimize --system NAMEinstead of a run (with --k, --mode, --finalists, --proposer, --test-trials).--k <int range>: With --optimize: the loop's k. [default: 2; x>=1]--mode <str>: With --optimize: parallel or sequential. [default: parallel]--finalists <int range>: With --optimize: candidates taken to dev. [default: 2; x>=1]--proposer <str>: With --optimize: the proposer's model ref (default: the system's generator).--test-trials <int range>: With --optimize: trials of each test question. [default: 1; x>=1]--help: Show this message and exit.
ragbench recommend¶
Which system to deploy, from a finished run: ranked by your constraints and priorities, with the reasons.
Starts from the run's own selection: settings; the options here override them. Exits with status 1 when no system qualifies.
Usage:
$ ragbench recommend [OPTIONS]
Options:
--run <path>: A finished run directory, e.g. results/. [required] --profile <str>: balanced, max_quality, cheapest_acceptable or lowest_latency. Default: the run'sselection.profile.--max-cost <float range>: Highest acceptable mean cost per question, in dollars. [x>=0]--max-latency <float range>: Highest acceptable p95 latency, in milliseconds. [x>=0]--min-faithfulness <float range>: Lowest acceptable mean faithfulness (0-5). [0<=x<=5]--min-answer-score <float range>: Lowest acceptable mean answer score (0-5). [0<=x<=5]--max-ingestion-cost <float range>: Highest acceptable one-off indexing cost, in dollars. [x>=0]--local-models: Only systems whose models all run on this machine.--no-network: Only systems that send no data off this machine (local models, no network tools).--export <path>: Write the winner's runnable config to this file.--json: Print the recommendation as JSON on stdout instead of the panel (the exit status is the same).--help: Show this message and exit.
ragbench auto¶
From a folder of documents to a decision: profile them, write questions if you have none, run the preset, and recommend a system.
Writes everything to one run directory (questions, results, recommendation.md, winner.yaml, report.html). If it stops (budget, a crash, Ctrl-C),
--resume RUN_DIR continues without paying again for the systems that finished or the questions already answered.
Usage:
$ ragbench auto [OPTIONS]
Options:
--docs <path>: Your documents: a folder, or one file.--questions <path>: Your questions JSONL. Without it, questions are written from your documents (flagged needs_review).--qrels <path>: Optional qrels JSONL for --questions.--preset <str>: Which systems to compare: quick, standard, thorough, agentic. Default: standard, or what the experience library suggests for a similar collection.--profile <str>: What the recommendation optimizes: balanced, max_quality, cheapest_acceptable or lowest_latency. [default: balanced]--n-questions <int range>: How many questions to write when you have none. [default: 50; x>=1]--seed <int>: Seed for writing questions. [default: 0]--max-cost <float range>: Total dollars to spend (writing questions plus the run). The run stops once it is reached. [x>=0]--model <str>: Model ref that writes the questions (default: the default generator model).-c, --config <path>: Take models, providers, pricing, evaluation and selection settings from this config (its systems and dataset are not used).--output-dir <path>: Where the run directory is created. [default: results]--resume <path>: Continue an earlier auto run in this directory: finished systems are kept, unfinished ones keep their answers and run only the missing or failed questions.--mock: Force local mock mode: nothing is paid for and the scores only validate the pipeline.--no-cache: Do not read or write the persistent disk cache.-y, --yes: Do not ask before spending more than evaluation.cost_confirm_threshold_usd.--no-library: Do not consult the experience library when choosing the preset.--open: Open report.html in your browser when done.--help: Show this message and exit.
ragbench optimize¶
Propose → Probe → Compose: improve a system's harness (prompts and knobs) on train, pick on dev, verify on test.
The proposer reads evidence from train runs only (never dev or test); a change is adopted only when the test split shows a
significant gain over the base (paired bootstrap, Holm). Writes optimized.yaml, optimization.md and ledger.json.
Usage:
$ ragbench optimize [OPTIONS]
Options:
-c, --config <path>: Experiment config holding the system and a dataset withsplits.json(ragbench split). [required]--system <str>: Name of the system whose harness to optimize. [required]--k <int range>: Width (parallel) or depth (sequential): 4k train probes and 3k proposer requests. [default: 2; x>=1]--mode <str>: parallel: k rollouts, k INV + k DOM candidates, k compositions. sequential: k rounds, carrying the best forward. [default: parallel]--feedback <str>: What the proposer sees: contrast (behaviour only), labels (train grades and references) or none. [default: contrast]--proposer <str>: Model ref of the proposer. Default: the system's own generator.--finalists <int range>: Candidates that beat the base on train and go on to dev. [default: 2; x>=1]--select <str>: How dev picks: score, arena (pairwise, label-free), or auto (arena on a label-free dataset). [default: auto]--test-trials <int range>: Times each test question is answered by the base and the pick. [default: 1; x>=1]--max-cost-usd <float range>: Spending cap for the whole optimization (default:evaluation.max_cost_usd). Required for a live run. [x>=0]--out <path>: Output folder (default: results/optimize__ --mock: Mock models and the scripted proposer: the whole loop offline, at no cost.--no-library: Do not read lessons from the experience library, and do not write one.--max-workers <int range>: Questions answered at the same time in each probe. [x>=1]-y, --yes: Do not ask before a live optimization whose estimate is aboveevaluation.cost_confirm_threshold_usd.--json: Print the outcome as JSON on stdout.--help: Show this message and exit.
ragbench demo¶
Create or verify a bundled dataset: the demo, or the hard suite with --suite hard.
Usage:
$ ragbench demo [OPTIONS]
Options:
-o, --output <path>: Directory where the dataset is written (default: data/). --overwrite: Overwrite files that differ from the bundled copy (by default they are kept and reported).--suite <str>: Which bundled dataset: demo or hard (generated to separate systems; run it with configs/hard.yaml). [default: demo]--help: Show this message and exit.
ragbench inspect-dataset¶
Profile a dataset: sizes and balance, label coverage, suggested chunk sizes, projected cost and likely problems.
Usage:
$ ragbench inspect-dataset [OPTIONS]
Options:
--docs <path>: Document folder. [required]--questions <path>: Questions JSONL file. [required]--qrels <path>: Optional qrels JSONL file.--include <str>: Only load files matching this glob (repeatable).--exclude <str>: Skip files matching this glob (repeatable).--on-error <str>:raise(fail on a file that cannot be loaded) orskip(warn and go on). [default: raise]--estimate / --no-estimate: Also project the cost of thestandardpreset (runs each system once on the mock models; a few seconds). [default: estimate]--help: Show this message and exit.
ragbench split¶
Split a dataset's questions into train / dev / test (stratified by category and answerability) and write splits.json. Never overwrites.
Usage:
$ ragbench split [OPTIONS] {dataset}
Arguments:
dataset: The questions JSONL file, or the dataset folder holdingquestions.jsonl. [required]
Options:
--train <float>: Share of the questions for train (what tuning features may learn from). [default: 0.5]--dev <float>: Share for dev (what they may check against). Test gets the rest. [default: 0.1]--seed <int>: Seed of the shuffle; the same seed on the same questions gives the same split. [default: 42]--out <path>: Where to write the splits file (default:splits.jsonnext to the questions).--json: Print the result as JSON on stdout.--help: Show this message and exit.
ragbench generate-questions¶
Write questions for documents that have none: single-hop, multi-hop, paraphrased, numeric and unanswerable, each flagged needs_review.
Usage:
$ ragbench generate-questions [OPTIONS]
Options:
--docs <path>: Your documents: a folder, or one file. [required]--out <path>: Questions JSONL to write, e.g. questions.jsonl. [required]--n <int range>: How many questions to write. [default: 100; x>=1]--mix <str>: Category shares, e.g. single_hop=0.4,multi_hop=0.2,paraphrase=0.15,numeric=0.1,unanswerable=0.15 (the default).--model <str>: Model ref that writes the questions (default: the default generator model).--seed <int>: Same seed, same documents sampled, same questions. [default: 0]-c, --config <path>: Readproviders:,pricing:andcache:from this config (needed for openai_compatible: models).--mock: Write template questions with no model (pipeline validation only). Also used when the model's API key is missing.--max-cost <float range>: Stop writing once this many dollars were charged (partial results are kept). [x>=0]--no-cache: Do not read or write the persistent disk cache.-y, --yes: Do not ask before a live run whose estimated cost is above the confirmation threshold.--force: Replace --out if it exists (it may hold questions you reviewed).--help: Show this message and exit.
ragbench label¶
Propose relevance labels for a run by pooling: every system's top documents, graded 0-3 by an LLM. Never changes your qrels.
Usage:
$ ragbench label [OPTIONS]
Options:
--run <path>: A finished run directory, e.g. results/. [required] --top-k <int range>: Pool the first K distinct documents each system retrieved, per question. [default: 10; x>=1]--judge-model <str>: Model ref that grades relevance (default: the run's judge model).--out <path>: Directory for the outputs (default: the run directory).--apply: Also write qrels.merged.jsonl: your labels plus the proposed grades for documents they do not mention.--mock: Grade with a word-overlap stand-in, no model (pipeline validation only). Also used when the model's API key is missing.--max-cost <float range>: Stop grading once this many dollars were charged (partial proposals are written). [x>=0]--max-workers <int range>: Documents graded at the same time. [default: 4; x>=1]--no-cache: Do not read or write the persistent disk cache.-y, --yes: Do not ask before a live run whose estimated cost is above the confirmation threshold.--force: With --apply: replace an existing qrels.merged.jsonl.--help: Show this message and exit.
ragbench init¶
Start a benchmark of your own documents: write a ready-to-run config and a questions file to edit.
Usage:
$ ragbench init [OPTIONS] [directory]
Arguments:
directory: Where to write ragbench.yaml (and questions.jsonl if you have none yet). [default: .]
Options:
--docs <path>: Your documents: a folder, or one file. [required]--questions <path>: Your questions JSONL. Without it a questions.jsonl template is created in DIRECTORY.--preset <str>: Which systems to compare: quick, standard, thorough, agentic. [default: standard]--force: Replace an existing ragbench.yaml (questions files are never replaced).--help: Show this message and exit.
ragbench import¶
Convert a CSV, BEIR dataset or Markdown Q/A file into RAGBench's questions (and qrels, documents).
Usage:
$ ragbench import [OPTIONS]
Options:
--format <str>: csv (question,answer,doc_ids), beir (corpus.jsonl + queries.jsonl + qrels/), or qa-md (MarkdownQ:/A:pairs). [required]--input <path>: The file (csv, qa-md) or folder (beir, qa-md) to import. [required]--output <path>: Directory to write questions.jsonl (and qrels.jsonl, docs/ for beir) into. [required]--docs <path>: Your documents folder, to check the imported doc ids against (csv, qa-md).--split <str>: beir: which qrels/.tsv to use. [default: test] --force: Replace the files of an earlier import into --output (for beir, its docs/ folder too).--help: Show this message and exit.
ragbench chunk-preview¶
Show how a chunker cuts your documents: size statistics plus the first chunks. Nothing is indexed or paid for (except semantic's sentence embeddings).
Usage:
$ ragbench chunk-preview [OPTIONS]
Options:
--docs <path>: Document file or folder. [required]--chunker <str>: Chunker section as YAML or JSON, e.g. '{type: markdown, chunk_size: 300}'. [default: {}]--doc <str>: Only this document id.-n, --limit <int range>: How many chunks to show. [default: 5; x>=0]--json: Print machine-readable JSON instead of tables.--mock: Use mock hashing embeddings for chunkers that embed text (semantic).--embedding-model <str>: Embedding model ref forsemantic(default: the benchmark default).--help: Show this message and exit.
ragbench list-systems¶
Print available RAG systems with their cost and latency profile.
Usage:
$ ragbench list-systems [OPTIONS]
Options:
--help: Show this message and exit.
ragbench cache¶
Inspect or clear the persistent cache of LLM responses and corpus embeddings.
Usage:
$ ragbench cache [OPTIONS] COMMAND [ARGS]...
Options:
--help: Show this message and exit.
Commands:
stats: Show what the persistent cache holds.clear: Delete cached entries.
ragbench cache stats¶
Show what the persistent cache holds.
Usage:
$ ragbench cache stats [OPTIONS]
Options:
--cache-dir <path>: Cache directory (default: $RAGBENCH_CACHE_DIR or .ragbench_cache).--help: Show this message and exit.
ragbench cache clear¶
Delete cached entries. Re-running a live benchmark afterwards pays for them again.
Usage:
$ ragbench cache clear [OPTIONS]
Options:
--cache-dir <path>: Cache directory (default: $RAGBENCH_CACHE_DIR or .ragbench_cache).--namespace <str>: Only clear this namespace:llmorembeddings.-y, --yes: Do not ask for confirmation.--help: Show this message and exit.
ragbench completion¶
Shell completion for the ragbench command.
Usage:
$ ragbench completion [OPTIONS] COMMAND [ARGS]...
Options:
--help: Show this message and exit.
Commands:
show: Print the completion script, to review it...install: Install tab completion for your shell...
ragbench completion show¶
Print the completion script, to review it or to source it yourself.
Usage:
$ ragbench completion show [OPTIONS]
Options:
--shell <str>: bash, zsh, fish, powershell, pwsh (default: the shell in $SHELL).--help: Show this message and exit.
ragbench completion install¶
Install tab completion for your shell (writes a script and, for bash and zsh, a line in your shell's startup file).
Usage:
$ ragbench completion install [OPTIONS]
Options:
--shell <str>: bash, zsh, fish, powershell, pwsh (default: the shell in $SHELL).--help: Show this message and exit.
ragbench report¶
Rebuild every report of a finished run (leaderboard.md, failures.md, qrels_audit.md, the recommendation, report.html) from its files.
Nothing is re-run and nothing is paid for: a run directory holds everything its reports are made of.
Usage:
$ ragbench report [OPTIONS] {run}
Arguments:
run: A run directory, its name under --results-dir,latest, or part of a name. [required]
Options:
--results-dir <path>: The folder that holds the run directories. [env var: RAGBENCH_RESULTS_DIR; default: results]--open: Open report.html in your browser when done.--help: Show this message and exit.
ragbench judge-audit¶
Calibrate the judge against a human: draw a blind sheet of answers to score by hand, then compare it with the judge.
Without --labels it writes judge_audit_sheet.csv into the run (fill in the five 0-5 axes; blank cells are skipped). With --labels it writes judge_audit.md (judge-human agreement per axis, bias per system) and links it from the report. Nothing is paid for.
Usage:
$ ragbench judge-audit [OPTIONS] {run}
Arguments:
run: A run directory, its name under --results-dir,latest, or part of a name. [required]
Options:
--results-dir <path>: The folder that holds the run directories. [env var: RAGBENCH_RESULTS_DIR; default: results]--sample <int range>: Answers to put on the sheet, drawn evenly from each system and judge-score band. [default: 50; x>=1]--seed <int>: Random seed of the draw: the same seed draws the same sheet. [default: 0]--labels <file>: The filled-in sheet: report how well the judge agrees with it.--force: Draw a new sheet over an existing one.--help: Show this message and exit.
ragbench compare-runs¶
How a later run moved against an earlier one, per system: metric changes and whether an answer-score drop is real.
Usage:
$ ragbench compare-runs [OPTIONS] {a} {b}
Arguments:
a: The earlier (baseline) run: a directory, a name under --results-dir, orlatest. [required]b: The later run. [required]
Options:
--results-dir <path>: The folder that holds the run directories. [env var: RAGBENCH_RESULTS_DIR; default: results]--min-drop <float range>: When two runs share too few questions to test, flag an answer-score drop larger than this. [default: 0.25; x>=0]--fail-on-regression: Exit with status 1 when any system regressed (for CI).--json: Print the comparison as JSON on stdout.--help: Show this message and exit.
ragbench contrast¶
Where two systems disagree: the questions one answers well and the other does not, and the first step at which their traces part.
Computed from the run's files (nothing is re-run or paid for). Unless --no-write, the pair is written into contrasts.jsonl and contrast_summary.json (replacing that pair's earlier lines) and the run's reports are rebuilt with a "Where systems disagree" section.
Usage:
$ ragbench contrast [OPTIONS] {run} {a} {b}
Arguments:
run: A run directory, its name under --results-dir,latest, or part of a name. [required]a: The first system. [required]b: The second system. [required]
Options:
--results-dir <path>: The folder that holds the run directories. [env var: RAGBENCH_RESULTS_DIR; default: results]--source <str>: What makes a system right:score(its answer score reaches --threshold) orverdicts(it wins the arena's verdict; needs evaluation.pairwise). [default: score]--threshold <float range>: Answer score (0-5) from which an answer counts as correct, for --source score. [default: 4.0; 0<=x<=5]--limit <int range>: Contrast questions to list per direction. [default: 5; x>=0]--no-write: Only print: leave contrasts.jsonl and the reports of the run as they are.--json: Print the summary and every contrast as JSON on stdout.--help: Show this message and exit.
ragbench export-traces¶
Export a finished run's traces as OpenTelemetry / OpenInference spans, to a file or to a trace backend.
One trace per answered question: a root span with the question, answer, system and score, and one child per step of the system
(retrieve, rerank, llm, tool, ...). Nothing is re-run or paid for. Sending needs pip install 'ragbench[otel]'. The reference
answers and the judge's reasoning are never exported.
Usage:
$ ragbench export-traces [OPTIONS] {run}
Arguments:
run: A run directory, its name under --results-dir,latest, or part of a name. [required]
Options:
--results-dir <path>: The folder that holds the run directories. [env var: RAGBENCH_RESULTS_DIR; default: results]--format <str>:jsonl(a file, offline),openinference(send to Phoenix, Langfuse, ... with OpenInference attributes) orotlp(send with OpenTelemetry GenAI attributes). [default: jsonl]--endpoint <str>: The OTLP/HTTP traces URL for openinference and otlp, e.g. http://localhost:6006/v1/traces (Phoenix). [env var: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT]--header <str>: A header to send, NAME=VALUE;${ENV_VAR}in the value is filled in and never printed. Repeatable.--project <str>: The project the backend files the traces under (default: the run's name).--system <str>: Only this system's traces. Repeatable.--question <str>: Only this question id's traces. Repeatable.--limit <int range>: Export only the first N traces. [x>=1]--out <path>: Where --format jsonl writes (default: traces.jsonl in the run directory).--json: Print a summary as JSON on stdout.--help: Show this message and exit.
ragbench doctor¶
Check Python, packages, optional extras, API keys (never shown), folders and the price table. Exits with status 1 if something is broken.
Usage:
$ ragbench doctor [OPTIONS]
Options:
-c, --config <path>: Also check every model this config calls against the price table and the credentials it needs.--results-dir <path>: The folder runs are written to. [env var: RAGBENCH_RESULTS_DIR; default: results]--cache-dir <path>: Cache directory (default: $RAGBENCH_CACHE_DIR or .ragbench_cache).--json: Print the checks as JSON on stdout.--help: Show this message and exit.
ragbench runs¶
List, inspect, remove and index the runs in a results folder.
Usage:
$ ragbench runs [OPTIONS] COMMAND [ARGS]...
Options:
--results-dir <path>: The folder that holds the run directories. [env var: RAGBENCH_RESULTS_DIR; default: results]--json: Print JSON on stdout instead of a table.--help: Show this message and exit.
Commands:
list: A table of runs, newest first: date, name,...show: One run in detail: its leaderboard,...rm: Delete a run directory.index: Write results/index.html: every run with...site: Many runs as one static, filterable...schema: The JSON Schema of a file a run writes...
ragbench runs list¶
A table of runs, newest first: date, name, mode, systems, questions, winner, cost.
Usage:
$ ragbench runs list [OPTIONS]
Options:
--results-dir <path>: The folder that holds the run directories. [env var: RAGBENCH_RESULTS_DIR; default: results]--json: Print JSON on stdout instead of a table.--help: Show this message and exit.
ragbench runs show¶
One run in detail: its leaderboard, recommendation and files.
Usage:
$ ragbench runs show [OPTIONS] {run}
Arguments:
run: A run directory, its name under --results-dir,latest, or part of a name. [required]
Options:
--results-dir <path>: The folder that holds the run directories. [env var: RAGBENCH_RESULTS_DIR; default: results]--json: Print the run as JSON on stdout.--help: Show this message and exit.
ragbench runs rm¶
Delete a run directory. Only a directory that looks like a run is ever removed.
Usage:
$ ragbench runs rm [OPTIONS] {run}
Arguments:
run: A run directory or its name under --results-dir (a part of a name is not enough for a delete). [required]
Options:
--results-dir <path>: The folder that holds the run directories. [env var: RAGBENCH_RESULTS_DIR; default: results]-y, --yes: Delete without asking.--help: Show this message and exit.
ragbench runs index¶
Write results/index.html: every run with its winner and a link to its report, and a score-over-time line per system.
Usage:
$ ragbench runs index [OPTIONS]
Options:
--results-dir <path>: The folder that holds the run directories. [env var: RAGBENCH_RESULTS_DIR; default: results]--open: Open the page in your browser.--help: Show this message and exit.
ragbench runs site¶
Many runs as one static, filterable leaderboard: every system of every complete run, with filters by run, mode, type and name and a sortable column each.
One HTML file with its data inside and no network requests, plus a copy of each run's report.html to link to. Mock runs are hidden by default when a live run is present. Nothing is re-run or paid for.
Usage:
$ ragbench runs site [OPTIONS] {folder}
Arguments:
folder: The folder that holds the run directories (a results folder, or docs/benchmarks). [required]
Options:
--out <path>: Where to write the site (default: asitefolder inside FOLDER).--title <str>: The page title.--open: Open the page in your browser.--json: Print a summary as JSON on stdout.--help: Show this message and exit.
ragbench runs schema¶
The JSON Schema of a file a run writes (run_summary.json, or one row of metrics_summary.csv), for tools that read RAGBench results.
Usage:
$ ragbench runs schema [OPTIONS] [name]
Arguments:
name: Which schema: run_summary, metrics_summary. Without a name, lists them.
Options:
--version <int>: The schema version (theschema_versionof a run's run_summary.json). [default: 1]--help: Show this message and exit.
ragbench models¶
List the models RAGBench has prices for, and sync OpenRouter's catalog (prices, tools, reasoning).
Usage:
$ ragbench models [OPTIONS] COMMAND [ARGS]...
Options:
--help: Show this message and exit.
Commands:
list: Models with a price, USD per 1M tokens:...sync: Download OpenRouter's model list (public,...
ragbench models list¶
Models with a price, USD per 1M tokens: the built-in table (OpenAI, Anthropic) and the OpenRouter catalog. Offline.
Usage:
$ ragbench models list [OPTIONS]
Options:
-p, --provider <str>: Only one provider: all, openai, anthropic, openrouter. [default: all]-s, --search <str>: Only refs containing this text (case-insensitive).--open-weights: Only OpenRouter models with published weights (a Hugging Face id).--json: Print JSON on stdout instead of a table.--help: Show this message and exit.
ragbench models sync¶
Download OpenRouter's model list (public, no key, free) and save prices, tool / JSON support and reasoning switches as a dated file.
Usage:
$ ragbench models sync [OPTIONS]
Options:
--from-file <path>: Read a savedGET /api/v1/modelsresponse instead of downloading it.-o, --output <path>: Where to write the catalog. Default: $RAGBENCH_OPENROUTER_CATALOG, else ~/.cache/ragbench/.--help: Show this message and exit.
ragbench datasets¶
Public benchmarks on demand: list them, pull one (checksum-verified, converted, optionally a stratified subset).
Usage:
$ ragbench datasets [OPTIONS] COMMAND [ARGS]...
Options:
--help: Show this message and exit.
Commands:
list: The datasetspullcan fetch, with...pull: Download a public benchmark, verify its...
ragbench datasets list¶
The datasets pull can fetch, with download size, license and kind of corpus.
Usage:
$ ragbench datasets list [OPTIONS]
Options:
--json: Print the datasets as JSON on stdout.--help: Show this message and exit.
ragbench datasets pull¶
Download a public benchmark, verify its checksum, convert it to RAGBench files and write docs/, questions.jsonl, qrels.jsonl and SOURCE.md.
Usage:
$ ragbench datasets pull [OPTIONS] {name}
Arguments:
name: A dataset fromragbench datasets list. [required]
Options:
-o, --output <path>: Where to write the dataset folder (default: data/NAME).--subset <int range>: Keep a stratified sample of N questions: every category and the unanswerable share are kept. [x>=1]--seed <int>: Seed of the subset sample. [default: 42]--force: Replace an earlier pull in the output folder.-y, --yes: Download without asking first.--help: Show this message and exit.
ragbench harness¶
Show a system's editable harness (prompts and knobs) and compare the harnesses of two configs. Offline.
Usage:
$ ragbench harness [OPTIONS] COMMAND [ARGS]...
Options:
--help: Show this message and exit.
Commands:
show: Print a system's harness as YAML: every...diff: Compare the harnesses of the systems two...
ragbench harness show¶
Print a system's harness as YAML: every prompt text it sends and its knobs, with what the config's patch changed marked.
Usage:
$ ragbench harness show [OPTIONS] {system}
Arguments:
system: A system type (its defaults), or with --config the name of a system in that config. [required]
Options:
-c, --config <path>: Config whose system to show, with itsharness:patch applied.--json: Print JSON on stdout instead of YAML.--help: Show this message and exit.
ragbench harness diff¶
Compare the harnesses of the systems two configs share (by name), field by field; list the systems only one of them has.
Usage:
$ ragbench harness diff [OPTIONS] {first} {second}
Arguments:
first: A config file. [required]second: Another config file (for example the optimized one). [required]
Options:
-s, --system <str>: Only this system (by name).--json: Print JSON on stdout instead of text.--help: Show this message and exit.
ragbench library¶
The experience library: what ragbench optimize learned on earlier collections. Offline; nothing here spends money.
Usage:
$ ragbench library [OPTIONS] COMMAND [ARGS]...
Options:
--help: Show this message and exit.
Commands:
list: List the entries, oldest first: id, date,...show: Print one entry in full: the collection's...export: Write the entries as JSONL (one entry per...clear: Delete entries for good (the library is...
ragbench library list¶
List the entries, oldest first: id, date, the system, its verified gain on held-out questions and the lesson.
Usage:
$ ragbench library list [OPTIONS]
Options:
--path <path>: Library folder (default:library.pathof --config, else RAGBENCH_LIBRARY, else ~/.ragbench/library).-c, --config <path>: Takelibrary.pathfrom this config.--fingerprint <str>: Only the entries of this collection key, or the start of one (showand--jsonprint it).--json: Print the entries as JSON on stdout.--help: Show this message and exit.
ragbench library show¶
Print one entry in full: the collection's features, the adopted harness, the verified gain and the lesson.
Usage:
$ ragbench library show [OPTIONS] {entry_id}
Arguments:
entry_id: An entry id, or the start of one. [required]
Options:
--path <path>: Library folder (default:library.pathof --config, else RAGBENCH_LIBRARY, else ~/.ragbench/library).-c, --config <path>: Takelibrary.pathfrom this config.--json: Print the entry as JSON on stdout.--help: Show this message and exit.
ragbench library export¶
Write the entries as JSONL (one entry per line, oldest first) to stdout, or to --out.
Usage:
$ ragbench library export [OPTIONS]
Options:
--path <path>: Library folder (default:library.pathof --config, else RAGBENCH_LIBRARY, else ~/.ragbench/library).-c, --config <path>: Takelibrary.pathfrom this config.--fingerprint <str>: Only the entries of this collection key, or the start of one (showand--jsonprint it).--out <path>: Write the JSONL here instead of stdout.--help: Show this message and exit.
ragbench library clear¶
Delete entries for good (the library is append-only otherwise). Asks first; with no terminal, or in CI, it needs --yes.
Usage:
$ ragbench library clear [OPTIONS]
Options:
--path <path>: Library folder (default:library.pathof --config, else RAGBENCH_LIBRARY, else ~/.ragbench/library).-c, --config <path>: Takelibrary.pathfrom this config.--fingerprint <str>: Remove only the entries of this collection key; without it, every entry.-y, --yes: Do not ask first.--help: Show this message and exit.