Where the results are¶
RAGBench publishes several runs, and each has its own page and folder. This page says which one is current, where each kind of result lives, and in what order to read a leaderboard.
Start here¶
The current benchmark is MultiHop-RAG (run on 26–27 October 2026). It is RAGBench's most recent and largest run: every RAG solution RAGBench can run, on one public dataset, with one generator and an independent judge. The README's published result is its short version. Older pages use the small demo dataset and are not comparable with it.
| You want | Go to |
|---|---|
| The current ranking of every system, with significance tests | MultiHop-RAG: Leaderboard |
| The system to pick, and the ones tied with it | MultiHop-RAG: Recommendation and Statistically tied with the best |
| The headline findings in a few lines | MultiHop-RAG: What it shows |
| Which question types are hard, why answers fail | MultiHop-RAG: Patterns across the runs |
| Claude Code, Codex and OpenCode as RAG systems | MultiHop-RAG: Agent harnesses |
| Which generator model to use, open-weight ones included | MultiHop-RAG: Open-weight generators |
| LangChain, LlamaIndex and Haystack out of the box | MultiHop-RAG: Frameworks |
| Every published run's table on one page, filterable | the docs site's Leaderboard and its explorer (built from the run folders when the site is built) |
| One run's files end to end, on the demo dataset | Example run (ragbench 0.4.0, October 2026) |
decision_gate measured on its own |
Decision gate evaluation (demo dataset) |
| How a number is computed, and what it cannot tell you | Methodology |
Which folder is which¶
Every published page is generated from the result files committed under docs/benchmarks/, never typed by hand (python scripts/check_results.py checks this in CI).
| Folder | What it holds | Explained on |
|---|---|---|
multihop-rag-run/ |
The main MultiHop-RAG run: every built-in system, the framework pipelines and GraphRAG on the same questions | MultiHop-RAG |
multihop-rag-harnesses-run/ |
The agent harnesses, on a stratified subset of those questions next to three systems of the main run; corrections.json corrects OpenCode's cost |
Agent harnesses |
multihop-rag-models-run/ |
Three systems with each generator model, on a smaller subset | Open-weight generators |
multihop-rag-optimize/ |
One ragbench optimize run on a train / test split of the questions |
Optimizing a system |
example-run/ |
Every built-in system on the demo dataset | Example run |
decision-gate-run/ |
decision_gate against its neighbours on the demo dataset |
Decision gate evaluation |
Each run folder has a leaderboard.md and a report.html rebuilt from its CSVs. They show that run alone; the page named in the last column puts it in context.
Reading a leaderboard¶
- Check what was run. The run table at the top of each page gives the dataset, the number of questions, the generator, the judge and the date. Compare systems only within one run, or within runs on the same questions.
- Read the recommendation and the tied list before the order. The tied list names the systems a paired significance test cannot tell apart from the best; among those, the cheapest or fastest is usually the better pick. The order inside a tie is noise, and so is a difference inside the bracketed 95% confidence intervals.
- Look at the categories that matter to you. An overall score can hide a weak category (on MultiHop-RAG, temporal questions and questions the documents cannot answer). The per-category table shows each system's strengths.
- Then cost and latency. Both are per question; the Pareto front lists the systems that no other system beats on answer score, cost and latency at once.
- Read the marks and the limitations. † marks a row from a run on fewer questions; a model-comparison row scored by a different judge is flagged; a cost the provider did not bill directly carries a correction. Each page ends with its limitations.
The metrics themselves (answer score, Recall@5, faithfulness, the confidence intervals and the significance tests) are defined in Methodology.
Your own runs¶
A run writes everything under results/<run>/: open report.html in a browser for the interactive report, leaderboard.md for the table and recommendation.md for the pick. ragbench runs site puts several runs on one filterable page. Result files and the site lists every file and its schema.