Leaderboard¶
Looking for the current benchmark? It is MultiHop-RAG, with its analysis; Where the results are says which run below is which and how to read them.
Every table below is generated from the result files of a run committed under docs/benchmarks/; nothing here is typed by hand. Systems are ordered by answer score; — means the run did not measure it, which is not the same as zero. How to read the numbers, and what they cannot tell you: methodology.
decision-gate-run¶
run, 2026-10-27, 9 systems. Files: metrics_summary.csv.
| System | Type | Answer score | Hit@5 | $/question | Latency p50 |
|---|---|---|---|---|---|
| decision_gate_no_sufficiency | decision_gate | 4.82 [4.72, 4.90] | 0.82 | $0.00086 | 2578 ms |
| decision_gate_default | decision_gate | 4.80 [4.70, 4.89] | 0.82 | $0.00094 | 2152 ms |
| corrective_default | corrective | 4.77 [4.67, 4.86] | 0.84 | $0.00060 | 3378 ms |
| hybrid_rerank_default | hybrid_rerank | 4.72 [4.59, 4.84] | 0.82 | $0.00053 | 1798 ms |
| decision_gate_llm_json | decision_gate | 4.71 [4.56, 4.84] | 0.82 | $0.00077 | 4270 ms |
| decision_gate_sufficiency_only | decision_gate | 4.69 [4.55, 4.81] | 0.82 | $0.00130 | 2279 ms |
| decision_gate_strict | decision_gate | 4.65 [4.49, 4.80] | 0.82 | $0.00091 | 1742 ms |
| decision_gate_per_chunk | decision_gate | 4.45 [4.27, 4.60] | 0.82 | $0.00101 | 2342 ms |
| no_retrieval_floor | no_retrieval | 1.54 [1.40, 1.68] | — | $0.00012 | 774 ms |
multihop-rag-models-run¶
live run, 2026-10-26, 100 questions, 24 systems. Files: metrics_summary.csv.
| System | Type | Answer score | Hit@5 | $/question | Latency p50 |
|---|---|---|---|---|---|
| hybrid_rerank_default[generator=anthropic:claude-haiku-5-5] | hybrid_rerank | 3.97 [3.76, 4.17] | 0.87 | $0.00123 | 3705 ms |
| agent_search_tools[generator=anthropic:claude-haiku-5-5] | agent_search | 3.97 [3.78, 4.15] | 0.82 | $0.00358 | 9933 ms |
| iterative_default[generator=anthropic:claude-haiku-5-5] | iterative | 3.84 [3.61, 4.04] | 0.81 | $0.00181 | 10033 ms |
| hybrid_rerank_default[generator=gpt-6-luna] | hybrid_rerank | 3.65 [3.47, 3.83] | 0.87 | $0.00108 | 2286 ms |
| agent_search_tools[generator=openrouter:z-ai/glm-5.3] | agent_search | 3.63 [3.43, 3.81] | 0.86 | $0.00474 | 6686 ms |
| hybrid_rerank_default[generator=openrouter:z-ai/glm-5.3] | hybrid_rerank | 3.61 [3.40, 3.80] | 0.87 | $0.00294 | 3175 ms |
| iterative_default[generator=openrouter:minimax/minimax-m3] | iterative | 3.60 [3.40, 3.79] | 0.82 | $0.00292 | 8130 ms |
| agent_search_tools[generator=openrouter:deepseek/deepseek-v4.1-flash] | agent_search | 3.60 [3.41, 3.78] | 0.85 | $0.00250 | 6676 ms |
| hybrid_rerank_default[generator=openrouter:xiaomi/mimo-v2.6-pro] | hybrid_rerank | 3.58 [3.38, 3.77] | 0.87 | $0.00246 | 7177 ms |
| iterative_default[generator=openrouter:moonshotai/kimi-k3] | iterative | 3.56 [3.34, 3.78] | 0.83 | $0.01098 | 8376 ms |
| agent_search_tools[generator=openrouter:xiaomi/mimo-v2.6-pro] | agent_search | 3.56 [3.35, 3.76] | 0.83 | $0.00513 | 22710 ms |
| iterative_default[generator=openrouter:z-ai/glm-5.3] | iterative | 3.54 [3.35, 3.72] | 0.82 | $0.00380 | 8126 ms |
| agent_search_tools[generator=openrouter:qwen/qwen3.8-flash] | agent_search | 3.51 [3.32, 3.69] | 0.86 | $0.00269 | 16879 ms |
| hybrid_rerank_default[generator=openrouter:qwen/qwen3.8-flash] | hybrid_rerank | 3.50 [3.27, 3.73] | 0.87 | $0.00131 | 4161 ms |
| iterative_default[generator=openrouter:xiaomi/mimo-v2.6-pro] | iterative | 3.50 [3.34, 3.67] | 0.83 | $0.00338 | 26323 ms |
| agent_search_tools[generator=openrouter:minimax/minimax-m3] | agent_search | 3.48 [3.27, 3.69] | 0.82 | $0.00491 | 10187 ms |
| hybrid_rerank_default[generator=openrouter:moonshotai/kimi-k3] | hybrid_rerank | 3.45 [3.19, 3.69] | 0.87 | $0.00408 | 2398 ms |
| iterative_default[generator=gpt-6-luna] | iterative | 3.44 [3.21, 3.65] | 0.80 | $0.00124 | 8587 ms |
| hybrid_rerank_default[generator=openrouter:minimax/minimax-m3] | hybrid_rerank | 3.43 [3.20, 3.64] | 0.87 | $0.00206 | 2524 ms |
| agent_search_tools[generator=gpt-6-luna] | agent_search | 3.42 [3.21, 3.64] | 0.87 | $0.00259 | 10793 ms |
| iterative_default[generator=openrouter:qwen/qwen3.8-flash] | iterative | 3.41 [3.18, 3.62] | 0.83 | $0.00157 | 15684 ms |
| agent_search_tools[generator=openrouter:moonshotai/kimi-k3] | agent_search | 3.16 [2.92, 3.41] | 0.50 | $0.01315 | 7755 ms |
| iterative_default[generator=openrouter:deepseek/deepseek-v4.1-flash] | iterative | 3.12 [2.87, 3.37] | 0.82 | $0.00146 | 5160 ms |
| hybrid_rerank_default[generator=openrouter:deepseek/deepseek-v4.1-flash] | hybrid_rerank | 2.82 [2.53, 3.11] | 0.87 | $0.00121 | 1461 ms |
multihop-rag-harnesses-run¶
live run, 2026-10-26, 150 questions, 6 systems. Files: metrics_summary.csv.
| System | Type | Answer score | Hit@5 | $/question | Latency p50 |
|---|---|---|---|---|---|
| hybrid_rerank_default | hybrid_rerank | 3.65 [3.50, 3.81] | 0.87 | $0.00109 | 2327 ms |
| opencode_glm | cli_agent | 3.64 [3.50, 3.77] | 0.83 | $0.01023 | 12196 ms |
| claude_code | cli_agent | 3.55 [3.35, 3.76] | 0.82 | $0.01353 | 20226 ms |
| agent_search_tools | agent_search | 3.44 [3.25, 3.61] | 0.87 | $0.00264 | 10922 ms |
| codex | cli_agent | 2.92 [2.73, 3.12] | 0.67 | $0.01598 | 27613 ms |
| no_retrieval_floor | no_retrieval | 1.95 [1.74, 2.17] | — | $0.00021 | 1357 ms |
multihop-rag-run¶
live run, 2026-10-26, 300 questions, 32 systems. Files: metrics_summary.csv.
| System | Type | Answer score | Hit@5 | $/question | Latency p50 |
|---|---|---|---|---|---|
| llm_heavy_default | llm_heavy | 3.83 [3.73, 3.94] | 0.86 | $0.00983 | 6401 ms |
| hybrid_rerank_default | hybrid_rerank | 3.73 [3.61, 3.83] | 0.87 | $0.00109 | 1840 ms |
| compression_llm_extract | compression | 3.72 [3.57, 3.85] | 0.84 | $0.00112 | 6280 ms |
| corrective_default | corrective | 3.68 [3.57, 3.79] | 0.85 | $0.00100 | 6836 ms |
| rerank_default | rerank | 3.65 [3.54, 3.76] | 0.86 | $0.00108 | 1990 ms |
| bm25_default | bm25 | 3.62 [3.50, 3.74] | 0.83 | $0.00104 | 2233 ms |
| decompose_default | decompose | 3.61 [3.49, 3.72] | 0.85 | $0.00087 | 3917 ms |
| contextual_default | contextual | 3.60 [3.48, 3.71] | 0.84 | $0.00104 | 2197 ms |
| rag_fusion_default | rag_fusion | 3.58 [3.47, 3.70] | 0.86 | $0.00085 | 4642 ms |
| adaptive_default | adaptive | 3.57 [3.44, 3.69] | 0.85 | $0.00088 | 3164 ms |
| decision_gate_default | decision_gate | 3.55 [3.40, 3.69] | 0.87 | $0.00181 | 1187 ms |
| agent_search_tools | agent_search | 3.53 [3.40, 3.64] | 0.87 | $0.00259 | 9450 ms |
| iterative_default | iterative | 3.52 [3.40, 3.63] | 0.83 | $0.00123 | 9772 ms |
| vector_default | vector | 3.48 [3.36, 3.59] | 0.81 | $0.00106 | 1909 ms |
| hierarchical_default | hierarchical | 3.45 [3.31, 3.58] | 0.79 | $0.00076 | 1787 ms |
| vector_mmr | vector | 3.44 [3.31, 3.57] | 0.81 | $0.00102 | 1570 ms |
| sentence_window_default | sentence_window | 3.44 [3.30, 3.56] | 0.82 | $0.00051 | 1895 ms |
| hyde_default | hyde | 3.44 [3.32, 3.56] | 0.83 | $0.00111 | 3410 ms |
| full_context_50k | full_context | 3.41 [3.29, 3.53] | 0.86 | $0.00581 | 2277 ms |
| late_interaction_rerank | late_interaction | 3.41 [3.28, 3.54] | 0.84 | $0.00076 | 1966 ms |
| parent_doc_default | parent_doc | 3.40 [3.26, 3.53] | 0.83 | $0.00123 | 1244 ms |
| raptor_collapsed | raptor | 3.36 [3.24, 3.49] | 0.81 | $0.00076 | 1982 ms |
| agent_search_plain | agent_search | 3.35 [3.22, 3.48] | 0.85 | $0.00226 | 8538 ms |
| grep_agent_default | grep_agent | 3.30 [3.15, 3.42] | 0.78 | $0.00309 | 8226 ms |
| graph_rag_local | graph_rag | 3.26 [3.13, 3.39] | 0.79 | $0.00092 | 1865 ms |
| hybrid_default | hybrid | 3.25 [3.12, 3.38] | 0.82 | $0.00106 | 2488 ms |
| langchain_default | python_callable | 2.85 [2.73, 2.98] | 0.83 | $0.00088 | 3632 ms |
| haystack_default | python_callable | 2.71 [2.57, 2.84] | 0.83 | $0.00115 | 4611 ms |
| decision_gate_strict | decision_gate | 2.60 [2.41, 2.78] | 0.87 | $0.00154 | 1151 ms |
| llamaindex_default | python_callable | 2.60 [2.46, 2.73] | 0.82 | $0.00176 | 4278 ms |
| graph_rag_global | graph_rag | 2.04 [1.88, 2.19] | 0.25 | $0.00061 | 2888 ms |
| no_retrieval_floor | no_retrieval | 1.90 [1.76, 2.05] | — | $0.00021 | 1164 ms |
example-run¶
live run, 2026-10-07, 163 questions, 24 systems. Files: metrics_summary.csv.
| System | Type | Answer score | Hit@5 | $/question | Latency p50 |
|---|---|---|---|---|---|
| iterative_default | iterative | 4.81 [4.71, 4.90] | 0.84 | $0.00057 | 4787 ms |
| decision_gate_default | decision_gate | 4.80 [4.70, 4.89] | 0.82 | $0.00094 | 2856 ms |
| grep_agent_default | grep_agent | 4.78 [4.69, 4.86] | 0.82 | $0.00109 | 7180 ms |
| full_context_ceiling | full_context | 4.78 [4.70, 4.84] | 0.80 | $0.00280 | 1710 ms |
| corrective_default | corrective | 4.77 [4.67, 4.86] | 0.84 | $0.00060 | 3493 ms |
| rag_fusion_default | rag_fusion | 4.75 [4.64, 4.86] | 0.85 | $0.00042 | 3515 ms |
| agent_search_tools | agent_search | 4.75 [4.63, 4.85] | 0.84 | $0.00082 | 2692 ms |
| rerank_default | rerank | 4.74 [4.63, 4.84] | 0.83 | $0.00052 | 1364 ms |
| hybrid_rerank_default | hybrid_rerank | 4.72 [4.59, 4.84] | 0.82 | $0.00053 | 1302 ms |
| decompose_default | decompose | 4.72 [4.60, 4.83] | 0.82 | $0.00041 | 2589 ms |
| llm_heavy_default | llm_heavy | 4.72 [4.60, 4.82] | 0.83 | $0.00512 | 4897 ms |
| parent_doc_default | parent_doc | 4.71 [4.60, 4.81] | 0.83 | $0.00049 | 1337 ms |
| adaptive_default | adaptive | 4.70 [4.57, 4.82] | 0.81 | $0.00044 | 2294 ms |
| contextual_default | contextual | 4.69 [4.57, 4.81] | 0.82 | $0.00046 | 1240 ms |
| bm25_default | bm25 | 4.69 [4.55, 4.81] | 0.79 | $0.00049 | 1155 ms |
| hyde_default | hyde | 4.68 [4.54, 4.80] | 0.83 | $0.00046 | 2662 ms |
| vector_default | vector | 4.67 [4.54, 4.79] | 0.83 | $0.00044 | 1588 ms |
| decision_gate_strict | decision_gate | 4.65 [4.49, 4.80] | 0.82 | $0.00091 | 1785 ms |
| hybrid_default | hybrid | 4.61 [4.46, 4.74] | 0.81 | $0.00048 | 1593 ms |
| agent_search_plain | agent_search | 4.58 [4.43, 4.72] | 0.83 | $0.00072 | 2523 ms |
| hierarchical_default | hierarchical | 4.56 [4.40, 4.70] | 0.79 | $0.00033 | 1224 ms |
| vector_mmr | vector | 4.54 [4.40, 4.68] | 0.81 | $0.00040 | 1358 ms |
| sentence_window_default | sentence_window | 4.42 [4.23, 4.59] | 0.82 | $0.00025 | 1157 ms |
| no_retrieval_floor | no_retrieval | 1.54 [1.40, 1.68] | — | $0.00012 | 977 ms |