RAG systems

Every system runs on the same dataset and is configured in the systems: list of an experiment file (see configuration.md). Each system accepts chunker:, retrieval:, models: and, where noted, llm_features: sections; unknown options are rejected when the config is loaded.

Profiles are relative: cost low = one generation call plus embeddings, medium = one extra short LLM call, high = several LLM calls per question or a prompt that carries the whole corpus, or LLM work at ingestion; latency fast = no LLM call besides generation. LLM in retrieval means the retrieval side itself calls an LLM.

Overview

Type	System	Cost	Latency	LLM in retrieval	Agentic	Best for
bm25	BM25	low	fast	no	no	Cheap baseline and exact-term matching
hybrid	Hybrid (BM25 + vector)	low	fast	no	no	Balanced lexical + semantic retrieval
hybrid_rerank	Hybrid + rerank	low	fast	no	no	Recall of hybrid plus rerank precision
no_retrieval	No retrieval	low	fast	no	no	Showing how much retrieval helps and how often the model makes things up without it
parent_doc	Parent document	low	fast	no	no	Better answer context with precise retrieval
rerank	Vector + rerank	low	fast	no	no	Higher precision context selection
sentence_window	Sentence window	low	fast	no	no	Precise matching with enough surrounding text to answer from
vector	Vector	low	fast	no	no	Semantic baseline
external_http	External HTTP endpoint	low	medium	no	no	Benchmarking a RAG you already run (any stack) against the built-in systems
late_interaction	Late interaction (ColBERT MaxSim)	low	medium	no	no	Questions whose wording must match precisely (names, terms of art) where one pooled vector per chunk blurs the detail
python_callable	Python callable	low	medium	no	no	Benchmarking your own code or a framework pipeline against the built-in systems without deploying it
compression	Contextual compression	medium	medium	yes	no	Long or noisy chunks where most of each passage is beside the point; smaller prompts on long contexts
decompose	Question decomposition	medium	medium	yes	no	Multi-part, comparison and multi-hop questions
hyde	HyDE	medium	medium	yes	no	Short or vaguely-worded questions
rag_fusion	RAG-Fusion	medium	medium	yes	no	Questions worded differently from the documents
contextual	Contextual retrieval	high	fast	no	no	Corpora where chunks lose their meaning out of context (many similar documents, pronouns, section-relative facts)
hierarchical	Hierarchical (summary-routed)	high	fast	no	no	"Which document?" questions and corpora too large to search chunk by chunk
raptor	RAPTOR (recursive summary tree)	high	fast	no	no	Questions that need the gist of several related passages, and corpora where one fact is spread over many chunks
adaptive	Adaptive router	high	medium	yes	no	Mixed workloads where no single architecture is best for every question
decision_gate	Decision-gated RAG	high	medium	yes	no	Corpora where many questions have no answer in the documents, and a confident wrong answer costs more than a refusal
full_context	Full context	high	medium	no	no	Small corpora, and measuring how much a retrieval pipeline loses against reading everything
graph_rag	GraphRAG (knowledge graph)	high	medium	no	no	Questions that connect entities across documents (local) and corpus-wide questions about themes and groups (global)
agent_search	Tool-using agent (search)	high	slow	yes	yes	Questions that need computation, exact lookups, or several searches
cli_agent	CLI agent harness	high	slow	no	yes	Measuring a coding-agent harness as a RAG system: how well, how fast and at what cost it finds the answer in your documents
corrective	Corrective RAG	high	slow	yes	yes	Corpora where the first search often misses and a bad context should be noticed rather than answered from
grep_agent	Grep agent (no index)	high	slow	yes	yes	Small or exact-match-heavy corpora, and testing whether you need retrieval infrastructure at all
iterative	Iterative multi-hop	high	slow	yes	yes	Multi-hop questions whose second search depends on what the first one found
llm_heavy	LLM-heavy	high	slow	yes	no	Higher-cost, quality-oriented experiments

Chunker options

All systems except no_retrieval, parent_doc, sentence_window, external_http, python_callable, adaptive, full_context, cli_agent, grep_agent use this chunker: section:

Option	Type	Default	Description
type	str	token	Chunker: token (real model tokens), word (whitespace words), fixed_char, recursive, sentence, semantic, or markdown.
chunk_size	int	unset	Maximum chunk size. Unit: tokens (token, recursive, sentence, semantic, markdown), words (word) or characters (fixed_char). Default: 500 (1200 for fixed_char).
chunk_overlap	int	unset	Overlap between consecutive chunks, in the unit of chunk_size; sentence and semantic count sentences instead. Default: 80 (150 for fixed_char, 1 for sentence, 0 for semantic).
min_chunk_size	int	unset	recursive, sentence, semantic, markdown: fold chunks (for markdown, sections) smaller than this many tokens into a neighbour while the result fits chunk_size. Default: 50 for markdown, off otherwise.
prefix_title	bool	false	Prepend the document title to every chunk's text, so it is embedded and searched with the chunk.
prefix_heading	bool	unset	markdown only: prepend the heading breadcrumb (Guide > Returns) to every chunk's text.
breakpoint_percentile	float	unset	semantic only: start a new chunk where the distance between consecutive sentences exceeds this percentile of the document's distances. Default: 90.

bm25

BM25. Lexical BM25 over chunks.

Best for: Cheap baseline and exact-term matching. Cost: low; latency: fast.

retrieval: options

Option	Type	Default	Description
top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size.

hybrid

Hybrid (BM25 + vector). BM25 + vector search fused with (optionally weighted) Reciprocal Rank Fusion.

Best for: Balanced lexical + semantic retrieval. Cost: low; latency: fast.

retrieval: options

Option	Type	Default	Description
diversity	none | mmr	none	mmr re-ranks a candidate pool by maximal marginal relevance so near-duplicate chunks stop crowding out other evidence. none keeps the plain ranking.
mmr_lambda	float	0.5	Relevance versus diversity when diversity: mmr: 1.0 reproduces the plain ranking, 0.0 only avoids repeats. Relevance is the retrieval score scaled so the best candidate is 1.
mmr_candidates	int	20	Candidates MMR chooses from when diversity: mmr (raised to the retrieval depth if smaller).
multi_query	bool	false	Retrieve with local (non-LLM) query variants to help multi-hop questions.
max_query_variants	int	4	Maximum number of query variants when multi_query is on.
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
bm25_top_k	int	20	BM25 candidates per query (raised to the retrieval depth if smaller).
vector_top_k	int	20	Vector candidates per query (raised to the retrieval depth if smaller).
bm25_weight	float	1.0	Weight of the BM25 ranking in weighted RRF; raise it to favor lexical evidence.
vector_weight	float	1.0	Weight of the vector ranking in weighted RRF; raise it to favor semantic evidence.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size, vector_weight.

hybrid_rerank

Hybrid + rerank. Hybrid BM25 + vector RRF retrieval followed by a reranking pass.

Best for: Recall of hybrid plus rerank precision. Cost: low; latency: fast.

retrieval: options

Option	Type	Default	Description
multi_query	bool	false	Retrieve with local (non-LLM) query variants to help multi-hop questions.
max_query_variants	int	4	Maximum number of query variants when multi_query is on.
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
bm25_top_k	int	20	BM25 candidates per query (raised to the retrieval depth if smaller).
vector_top_k	int	20	Vector candidates per query (raised to the retrieval depth if smaller).
bm25_weight	float	1.0	Weight of the BM25 ranking in weighted RRF; raise it to favor lexical evidence.
vector_weight	float	1.0	Weight of the vector ranking in weighted RRF; raise it to favor semantic evidence.
reranker_model	str	unset	Hugging Face model for reranker: cross_encoder (default BAAI/bge-reranker-base). Needs pip install 'ragbench[rerank]'.
candidate_top_k	int	30	Fused candidates handed to the reranker (raised to the retrieval depth if smaller).
reranker	str	local_relevance	simple_keyword_overlap, local_relevance (TF-IDF), cross_encoder (needs the rerank extra), or llm.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size, reranker, vector_weight.

no_retrieval

No retrieval. The model answers from its own knowledge with no documents at all (the floor baseline).

Best for: Showing how much retrieval helps and how often the model makes things up without it. Cost: low; latency: fast.

retrieval: options

This system has no retrieval: options.

This system takes no chunker: section.

This system retrieves nothing, so its retrieval columns (recall, MRR, nDCG) are blank in every report, never 0.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs none.

parent_doc

Parent document. Retrieve small child chunks, answer from their larger parent chunks.

Best for: Better answer context with precise retrieval. Cost: low; latency: fast.

retrieval: options

Option	Type	Default	Description
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
top_k_children	int	8	Child chunks searched (raised to twice the parent depth if smaller).
top_k_parents	int	4	Parent chunks returned; also how many the generator reads.
parent_score_aggregation	max | sum | mean	max	How child scores roll up to their parent: max keeps the best match, sum rewards parents hit by several children, mean averages.

chunker: options (replaces the shared chunker section)

Option	Type	Default	Description
parent_chunk_size	int	1000	Parent chunk size in words.
parent_chunk_overlap	int	150	Overlap between parent chunks in words.
child_chunk_size	int	250	Child chunk size in words.
child_chunk_overlap	int	50	Overlap between child chunks in words.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k.

rerank

Vector + rerank. Vector retrieval followed by a reranking pass.

Best for: Higher precision context selection. Cost: low; latency: fast.

retrieval: options

Option	Type	Default	Description
multi_query	bool	false	Retrieve with local (non-LLM) query variants to help multi-hop questions.
max_query_variants	int	4	Maximum number of query variants when multi_query is on.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
reranker_model	str	unset	Hugging Face model for reranker: cross_encoder (default BAAI/bge-reranker-base). Needs pip install 'ragbench[rerank]'.
candidate_top_k	int	30	Vector candidates handed to the reranker (raised to the retrieval depth if smaller).
reranker	str	simple_keyword_overlap	simple_keyword_overlap, local_relevance (TF-IDF), cross_encoder (needs the rerank extra), or llm.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size, reranker.

sentence_window

Sentence window. Search single sentences, then hand the generator each matched sentence with the sentences around it.

Best for: Precise matching with enough surrounding text to answer from. Cost: low; latency: fast.

retrieval: options

Option	Type	Default	Description
top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
window	int	2	Sentences added on each side of a matched sentence (0 = the sentence alone). Overlapping windows of one document merge.
candidate_top_k	int	30	Sentences searched before windows are built (raised to four times the retrieval depth if smaller).

This system takes no chunker: section.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k.

vector

Vector. Embedding search with cosine similarity over a pluggable vector backend (exact NumPy by default).

Best for: Semantic baseline. Cost: low; latency: fast.

retrieval: options

Option	Type	Default	Description
diversity	none | mmr	none	mmr re-ranks a candidate pool by maximal marginal relevance so near-duplicate chunks stop crowding out other evidence. none keeps the plain ranking.
mmr_lambda	float	0.5	Relevance versus diversity when diversity: mmr: 1.0 reproduces the plain ranking, 0.0 only avoids repeats. Relevance is the retrieval score scaled so the best candidate is 1.
mmr_candidates	int	20	Candidates MMR chooses from when diversity: mmr (raised to the retrieval depth if smaller).
top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size.

external_http

External HTTP endpoint. Your own RAG service: each question is sent to an HTTP endpoint and the JSON reply (answer, passages, usage, cost) is mapped with JSONPath.

Best for: Benchmarking a RAG you already run (any stack) against the built-in systems. Cost: low; latency: medium.

retrieval: options

Option	Type	Default	Description
model	str	unset	The model your system generates with (a model ref, e.g. gpt-6-luna). RAGBench never calls it: it prices reported token usage and keeps the judge independent of it. Unset: a judge of any family scores the answers.
url	str	required	The endpoint that answers one question. ${VAR} is replaced by the environment variable when the request is sent.
method	POST | GET	POST	POST sends the rendered request as a JSON body, GET as query parameters.
headers	dict	empty	HTTP headers. Put secrets in environment variables (Authorization: Bearer ${MY_RAG_TOKEN}): they are filled in per request and never logged or saved.
request	str	{"question": {{ question \| tojson }}, "top_k": {{ top_k }}}	Jinja template of the request, rendering a JSON object; it sees question and top_k (the retrieval depth). Use \| tojson for strings.
response	block	its fields below	Where the answer, passages, usage and cost are in the reply; see its fields.
response.answer	str	$.answer	The answer text. A reply without it fails the question.
response.contexts	str	$.contexts[*]	The passages the answer was written from, one value per passage. Unset when the endpoint returns none: its retrieval metrics are then blank.
response.text	str	$.text	A passage's text, read from one passage ($ when the passages are plain strings).
response.doc_id	str	$.doc_id	A passage's document id, read from one passage. It must name a corpus document (its id, path, file name or file stem); a question whose passages do not all map gets blank retrieval metrics, never zeros.
response.score	str	$.score	A passage's retrieval score, read from one passage (optional).
response.prompt_tokens	str	$.usage.prompt_tokens	Prompt tokens the endpoint's model used (optional).
response.completion_tokens	str	$.usage.completion_tokens	Completion tokens the endpoint's model used (optional).
response.cost_usd	str	$.cost_usd	What the request cost in USD (optional). Without it, cost is priced from the tokens at model's rate, else reported as unknown.
timeout_s	float	60.0	Seconds to wait for one reply.
retries	int	2	Extra attempts after a timeout, a connection error, HTTP 429 or a 5xx reply, with exponential backoff.
limits	block	its fields below	Throttles for this endpoint; see its fields.
limits.max_concurrent_requests	int	unset	Requests in flight at once, across all questions.
limits.requests_per_minute	int	unset	Requests started per rolling minute.
post_documents_to	str	unset	Unset: the endpoint already has the documents. Set to a URL to POST them there once before the questions ({"documents": [{doc_id, path, title, text}]}, 100 per request).

This system takes no chunker: section.

This system takes no harness: (its prompts are its routes' or live outside RAGBench).

late_interaction

Late interaction (ColBERT MaxSim). Per-token embeddings scored by MaxSim (ColBERT): reranks hybrid or BM25 candidates, or searches every chunk exactly.

Best for: Questions whose wording must match precisely (names, terms of art) where one pooled vector per chunk blurs the detail. Cost: low; latency: medium.

retrieval: options

Option	Type	Default	Description
diversity	none | mmr	none	mmr re-ranks a candidate pool by maximal marginal relevance so near-duplicate chunks stop crowding out other evidence. none keeps the plain ranking.
mmr_lambda	float	0.5	Relevance versus diversity when diversity: mmr: 1.0 reproduces the plain ranking, 0.0 only avoids repeats. Relevance is the retrieval score scaled so the best candidate is 1.
mmr_candidates	int	20	Candidates MMR chooses from when diversity: mmr (raised to the retrieval depth if smaller).
multi_query	bool	false	Retrieve with local (non-LLM) query variants to help multi-hop questions.
max_query_variants	int	4	Maximum number of query variants when multi_query is on.
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
bm25_top_k	int	20	BM25 candidates per query (raised to the retrieval depth if smaller).
vector_top_k	int	20	Vector candidates per query (raised to the retrieval depth if smaller).
bm25_weight	float	1.0	Weight of the BM25 ranking in weighted RRF; raise it to favor lexical evidence.
vector_weight	float	1.0	Weight of the vector ranking in weighted RRF; raise it to favor semantic evidence.
mode	rerank | full	rerank	rerank: MaxSim over the first stage's candidate_k candidates (cheap; a candidate is encoded the first time it is seen). full: exact MaxSim over every chunk, all encoded at ingestion and kept in memory (fp16); refused above max_index_vectors.
model	str	answerdotai/answerai-colbert-small-v1	ColBERT checkpoint (Hugging Face id or local folder; Stanford ColBERT or PyLate layout). Local, $0 API cost; needs pip install 'ragbench[colbert]'.
first_stage	hybrid | bm25	hybrid	The ranking rerank mode reorders: hybrid (BM25 + vector, RRF) or BM25 alone (no embedding cost).
candidate_k	int	30	First-stage candidates rerank mode scores (raised to the retrieval depth if smaller).
max_index_vectors	int	2000000	Most token vectors full mode will hold (about 2 bytes per dimension each); a larger corpus is refused with a pointer to rerank.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size, vector_weight.

python_callable

Python callable. Your own pipeline, called in-process: a function (question, top_k) returning a dict (answer, passages, usage, cost) or an AnswerResult.

Best for: Benchmarking your own code or a framework pipeline against the built-in systems without deploying it. Cost: low; latency: medium.

retrieval: options

Option	Type	Default	Description
model	str	unset	The model your system generates with (a model ref, e.g. gpt-6-luna). RAGBench never calls it: it prices reported token usage and keeps the judge independent of it. Unset: a judge of any family scores the answers.
target	str	required	The function that answers one question, as package.module:function or path/to/file.py:function (relative to the working directory). It is called as function(question, top_k) and returns a dict (answer, and optionally contexts, usage, cost_usd) or an AnswerResult.
ingest_target	str	unset	A function called once with the list of documents before the questions (same notation).

This system takes no chunker: section.

This system takes no harness: (its prompts are its routes' or live outside RAGBench).

compression

Contextual compression. Hybrid retrieval of candidate_k candidates, each cut down to the sentences that bear on the question (LLM quotes, an embedding filter or a decision model), then packed to a token budget.

Best for: Long or noisy chunks where most of each passage is beside the point; smaller prompts on long contexts. Cost: medium; latency: medium.

retrieval: options

Option	Type	Default	Description
diversity	none | mmr	none	mmr re-ranks a candidate pool by maximal marginal relevance so near-duplicate chunks stop crowding out other evidence. none keeps the plain ranking.
mmr_lambda	float	0.5	Relevance versus diversity when diversity: mmr: 1.0 reproduces the plain ranking, 0.0 only avoids repeats. Relevance is the retrieval score scaled so the best candidate is 1.
mmr_candidates	int	20	Candidates MMR chooses from when diversity: mmr (raised to the retrieval depth if smaller).
multi_query	bool	false	Retrieve with local (non-LLM) query variants to help multi-hop questions.
max_query_variants	int	4	Maximum number of query variants when multi_query is on.
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
bm25_top_k	int	20	BM25 candidates per query (raised to the retrieval depth if smaller).
vector_top_k	int	20	Vector candidates per query (raised to the retrieval depth if smaller).
bm25_weight	float	1.0	Weight of the BM25 ranking in weighted RRF; raise it to favor lexical evidence.
vector_weight	float	1.0	Weight of the vector ranking in weighted RRF; raise it to favor semantic evidence.
candidate_k	int	20	Hybrid candidates compressed per question (also the ranking scored for retrieval, raised to the retrieval depth if smaller).
compressor	llm_extract | embedding_filter | decision	llm_extract	llm_extract: one generator call quotes the sentences of every candidate that bear on the question; a quote not found word for word in its chunk is dropped. embedding_filter: keep sentences whose embedding is at least similarity_threshold similar to the question (no LLM). decision: a decision model judges each sentence (the interface decision_gate uses), keeping those at relevance_threshold or above.
context_token_budget	int	1500	Tokens of compressed text the generator reads: compressed candidates in rank order, each that fits.
similarity_threshold	float	0.3	Cosine similarity to the question a sentence needs under embedding_filter.
relevance_threshold	float	0.5	Probability a sentence needs under decision; one the model does not give is kept.
decision_model	block	its fields below	The decision model backend for compressor: decision; see its fields.
decision_model.provider	str	openai_decisions	openai_decisions (OpenAI's Decisions API, POST /v1/decisions, needs OPENAI_API_KEY), llm_json (the same questions put to a chat model that replies with JSON; the like-for-like control for openai_decisions), mock (deterministic word-overlap probabilities, no key), or a registered third-party backend. Mock runs (--mock, or no key) always use mock.
decision_model.model	str	unset	Model the backend calls: for openai_decisions a Decisions model (default gpt-6-luna, the only one served today), for llm_json a chat model ref (default gpt-6-luna).

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size, vector_weight.

decompose

Question decomposition. The LLM splits a complex question into sub-questions; each is searched (optionally in sequence, feeding earlier answers forward), then one answer is synthesized from all the evidence.

Best for: Multi-part, comparison and multi-hop questions. Cost: medium; latency: medium.

retrieval: options

Option	Type	Default	Description
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
retriever	hybrid | vector	hybrid	Search used for every query: hybrid (BM25 + vector, fused with RRF) or vector alone.
per_query_top_k	int	20	Candidates per query before the rankings are merged (raised to the retrieval depth if smaller).
max_subquestions	int	4	Most sub-questions the LLM may split a question into; extras are dropped.
sequential	bool	false	Answer the sub-questions in order, each from its own evidence, and add the earlier findings to the next search (for questions whose later parts depend on earlier answers). Costs one extra LLM call per sub-question; off, they are searched independently.
chunks_per_subquestion	int	3	Chunks of a sub-question's ranking the LLM reads to answer it when sequential is on.
include_original	bool	false	Also search the original question, as a safety net against a bad split.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, query_rewrite_prompt, few_shot; knobs top_k, chunk_size.

hyde

HyDE. Hypothetical Document Embeddings: the LLM writes a hypothetical answer used as the search probe.

Best for: Short or vaguely-worded questions. Cost: medium; latency: medium.

retrieval: options

Option	Type	Default	Description
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
probe_top_k	int	unset	Results fetched per ranking before fusion. Default: twice the retrieval depth, at least 10.
fuse_with_question	bool	true	Fuse the hypothetical-document ranking with the raw-question ranking via RRF.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, query_rewrite_prompt, few_shot; knobs top_k, chunk_size.

rag_fusion

RAG-Fusion. The LLM writes alternative queries; each is searched (hybrid or vector) and the rankings are merged with Reciprocal Rank Fusion.

Best for: Questions worded differently from the documents. Cost: medium; latency: medium.

retrieval: options

Option	Type	Default	Description
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
retriever	hybrid | vector	hybrid	Search used for every query: hybrid (BM25 + vector, fused with RRF) or vector alone.
per_query_top_k	int	20	Candidates per query before the rankings are merged (raised to the retrieval depth if smaller).
num_queries	int	4	Alternative queries the LLM writes for each question.
include_original	bool	true	Also search the original question and merge it in (standard RAG-Fusion).

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, query_rewrite_prompt, few_shot; knobs top_k, chunk_size.

contextual

Contextual retrieval. An LLM situates each chunk within its document at ingestion; the context is embedded and BM25-indexed with the chunk, then hybrid RRF retrieval.

Best for: Corpora where chunks lose their meaning out of context (many similar documents, pronouns, section-relative facts). Cost: high; latency: fast.

retrieval: options

Option	Type	Default	Description
diversity	none | mmr	none	mmr re-ranks a candidate pool by maximal marginal relevance so near-duplicate chunks stop crowding out other evidence. none keeps the plain ranking.
mmr_lambda	float	0.5	Relevance versus diversity when diversity: mmr: 1.0 reproduces the plain ranking, 0.0 only avoids repeats. Relevance is the retrieval score scaled so the best candidate is 1.
mmr_candidates	int	20	Candidates MMR chooses from when diversity: mmr (raised to the retrieval depth if smaller).
multi_query	bool	false	Retrieve with local (non-LLM) query variants to help multi-hop questions.
max_query_variants	int	4	Maximum number of query variants when multi_query is on.
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
bm25_top_k	int	20	BM25 candidates per query (raised to the retrieval depth if smaller).
vector_top_k	int	20	Vector candidates per query (raised to the retrieval depth if smaller).
bm25_weight	float	1.0	Weight of the BM25 ranking in weighted RRF; raise it to favor lexical evidence.
vector_weight	float	1.0	Weight of the vector ranking in weighted RRF; raise it to favor semantic evidence.
document_max_chars	int	12000	Characters of the source document shown to the LLM when it writes each chunk's context. A longer document is cut to a window around the chunk. Raise it for long documents whose chunks need context from far away; every call pays for the whole excerpt.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size, vector_weight.

hierarchical

Hierarchical (summary-routed). Choose documents first from LLM-written summaries and titles, then retrieve chunks only inside the chosen documents.

Best for: "Which document?" questions and corpora too large to search chunk by chunk. Cost: high; latency: fast.

retrieval: options

Option	Type	Default	Description
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
docs_k	int	5	Documents the first stage selects (by BM25 and embedding similarity over each document's title and summary).
chunks_per_doc	int	2	Chunks taken from each selected document in the second stage; the best of them overall are returned.
summary_max_chars	int	6000	Characters of each document shown to the LLM when it writes the summary.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size.

raptor

RAPTOR (recursive summary tree). Chunks clustered and summarised by an LLM, recursively, into a tree; the question is matched against chunks and summaries of every level.

Best for: Questions that need the gist of several related passages, and corpora where one fact is spread over many chunks. Cost: high; latency: fast.

retrieval: options

Option	Type	Default	Description
top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
mode	collapsed | tree	collapsed	collapsed: search the chunks and every summary as one flat index (the paper's better variant). tree: start at the top summaries and descend, keeping branch_k nodes per level.
max_levels	int	3	Most summary levels built above the chunks.
min_cluster_nodes	int	5	A level with fewer nodes than this is not clustered again: the tree stops there.
max_clusters	int	50	Most clusters per level; the number is chosen by BIC among 2 and this (at most half the level's nodes).
pca_dims	int	10	Dimensions the embeddings are reduced to (PCA) before Gaussian-mixture clustering.
summary_input_chars	int	12000	Most characters of members one summary prompt carries; a larger cluster is clustered again within itself until its parts fit, so members go in whole.
branch_k	int	3	Nodes kept per level in tree mode.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size.

adaptive

Adaptive router. Routes each question to the best of several configured pipelines (exact identifiers to BM25, comparisons to decomposition, calculations to a tool agent, the rest to the default).

Best for: Mixed workloads where no single architecture is best for every question. Cost: high; latency: medium.

retrieval: options

Option	Type	Default	Description
router	heuristic | llm	heuristic	heuristic routes with fixed rules and costs nothing: exact identifiers go to lexical, arithmetic, percentages and date calculations to computation, comparisons and questions with several linked parts to multi_hop, everything else to default. llm asks the system's generator model to choose among the routes (one short call per question), which also allows route names of your own.
routes	dict	required	The pipelines a question can be sent to, by route name; each is a full system config (type, chunker, retrieval, tools, ...). default is required. With router: heuristic the other names must be lexical, computation or multi_hop; a role you leave out falls back to default. Routes inherit the adaptive system's models unless they set their own.
route_descriptions	dict	empty	router: llm only: what each route is for, shown to the model. Known role names have a built-in description.

This system takes no chunker: section.

This system takes no harness: (its prompts are its routes' or live outside RAGBench).

decision_gate

Decision-gated RAG. Hybrid + rerank retrieval, then a typed decision model drops chunks that do not help answer the question and abstains when the rest is not enough.

Best for: Corpora where many questions have no answer in the documents, and a confident wrong answer costs more than a refusal. Cost: high; latency: medium.

retrieval: options

Option	Type	Default	Description
multi_query	bool	false	Retrieve with local (non-LLM) query variants to help multi-hop questions.
max_query_variants	int	4	Maximum number of query variants when multi_query is on.
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
bm25_top_k	int	20	BM25 candidates per query (raised to the retrieval depth if smaller).
vector_top_k	int	20	Vector candidates per query (raised to the retrieval depth if smaller).
bm25_weight	float	1.0	Weight of the BM25 ranking in weighted RRF; raise it to favor lexical evidence.
vector_weight	float	1.0	Weight of the vector ranking in weighted RRF; raise it to favor semantic evidence.
reranker_model	str	unset	Hugging Face model for reranker: cross_encoder (default BAAI/bge-reranker-base). Needs pip install 'ragbench[rerank]'.
candidate_top_k	int	30	Fused candidates handed to the reranker (raised to the retrieval depth if smaller).
reranker	str	local_relevance	simple_keyword_overlap, local_relevance (TF-IDF), cross_encoder (needs the rerank extra), or llm.
decision_model	block	its fields below	The decision model backend; see its fields.
decision_model.provider	str	openai_decisions	openai_decisions (OpenAI's Decisions API, POST /v1/decisions, needs OPENAI_API_KEY), llm_json (the same questions put to a chat model that replies with JSON; the like-for-like control for openai_decisions), mock (deterministic word-overlap probabilities, no key), or a registered third-party backend. Mock runs (--mock, or no key) always use mock.
decision_model.model	str	unset	Model the backend calls: for openai_decisions a Decisions model (default gpt-6-luna, the only one served today), for llm_json a chat model ref (default gpt-6-luna).
answerability_threshold	float	0.5	A retrieved chunk is kept only when its answerability probability is at least this. 0 keeps every chunk, so the gate only decides sufficiency.
sufficiency_threshold	float	0.5	The kept chunks must be judged sufficient with at least this probability, or the system abstains: no generator call, the answer is "I could not find the answer in the provided documents." 0 turns the sufficiency check off (and saves its request), so the gate only filters chunks.
judge_mode	batched | per_chunk	batched	batched judges all candidates in one request (one predicate per passage, all passages in the input: cheapest and fastest, but a passage is read next to the others). per_chunk sends one request per candidate (isolated, five at a time, one request's cost each).
max_candidates_to_judge	int	10	Top-ranked candidates the answerability question is asked about, drawn from the retrieval depth. Kept chunks are the ones above the threshold, in rank order, up to the generator's context size, so a deeper candidate can replace a dropped one.
answerability_instructions	str	unset	Overrides the per-chunk predicate. {passage} stands for the passage under judgment and is required in batched mode. Default: "Does {passage} help answer the query?" The wording sets how many chunks survive: a stricter predicate (see configs/decision_gate.yaml) drops chunks that carry only part of an answer.
sufficiency_instructions	str	unset	Overrides the set-level predicate. Default: "Is this evidence enough to answer the query from the documents alone?" A stricter wording abstains more often, on answerable questions too.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size, reranker, vector_weight.

full_context

Full context. Puts whole documents (the whole corpus when it fits) in the prompt, most relevant first by BM25, up to a token budget (the ceiling baseline: do you need retrieval at all?).

Best for: Small corpora, and measuring how much a retrieval pipeline loses against reading everything. Cost: high; latency: medium.

retrieval: options

Option	Type	Default	Description
context_token_budget	int	100000	Most tokens of document text put in the prompt (counted with the chunking tokenizer; the small [doc_id] headers are not counted). Documents are ordered by BM25 relevance to the question; when they do not all fit, the first one that does not is cut at a token boundary and the rest are dropped (truncated: true in the retrieval metadata). You pay for every token you put in.

This system takes no chunker: section.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs none.

graph_rag

GraphRAG (knowledge graph). An LLM extracts entities and relations from every chunk at ingestion and writes a report per graph community; local follows the question's entities to their chunks, global answers by map-reduce over the community reports.

Best for: Questions that connect entities across documents (local) and corpus-wide questions about themes and groups (global). Cost: high; latency: medium.

retrieval: options

Option	Type	Default	Description
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
mode	local | global	local	local: match the question to entities, follow their relations, and return the chunks they came from, with the entities and relations as notes for the generator. global: rank the community reports and answer by map-reduce over them (one extra LLM call per report the generator reads); top_k then counts reports.
entities_k	int	10	Entities matched to the question in local mode (BM25 and embedding similarity over each entity's name and description).
neighbours_per_entity	int	10	Related entities followed from each matched entity in local mode, strongest relation first (a hub entity would otherwise pull in everything).
max_relations	int	20	Relations around the matched entities shown to the generator in local mode, strongest first.
notes_max_chars	int	4000	Characters of graph notes (entities, relations, the main community report) the generator reads in local mode.
max_entities_per_chunk	int	15	Most entities the LLM may extract from one chunk at ingestion.
min_community_size	int	2	Entities a community needs to get a report; smaller ones (a lone entity) get none.
community_max_entities	int	40	Entities shown to the LLM when it writes a community's report: the best connected ones of a larger community.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, few_shot; knobs top_k, chunk_size.

agent_search

Tool-using agent (search). A function-calling agent that searches the corpus and calls its configured tools (calculator, date_calc, corpus_grep, ...), then writes the cited answer.

Best for: Questions that need computation, exact lookups, or several searches. Cost: high; latency: slow.

retrieval: options

Option	Type	Default	Description
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
retriever	hybrid | vector	hybrid	Search used for every query: hybrid (BM25 + vector, fused with RRF) or vector alone.
per_query_top_k	int	20	Candidates per query before the rankings are merged (raised to the retrieval depth if smaller).
max_cost_usd	float	unset	Stop the agent's extra work once this question has cost this much (USD, at standalone prices); the answer is then written from the evidence so far. No cap by default.
max_tokens	int	unset	Stop the agent's extra work once this question has used this many LLM tokens (prompt + completion). No cap by default.
agent_mode	native | react_json	native	native uses the provider's function calling. react_json describes the tools in the prompt and parses one JSON action per reply, for models without function calling.
max_steps	int	6	Most model turns (each may call several tools). When it is reached the agent is asked to answer with what it has.
max_tool_calls	int	unset	Most tool calls per question; further calls are refused and the agent is asked to answer. No cap by default.

This system accepts a tools: list (see tools.md).

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, tool_guidance, few_shot; knobs top_k, chunk_size, max_steps.

cli_agent

CLI agent harness. An agent harness (Claude Code, Codex, OpenCode or any command) answers each question by searching a read-only copy of the corpus in a fresh workspace.

Best for: Measuring a coding-agent harness as a RAG system: how well, how fast and at what cost it finds the answer in your documents. Cost: high; latency: slow.

retrieval: options

Option	Type	Default	Description
model	str	unset	The model your system generates with (a model ref, e.g. gpt-6-luna). RAGBench never calls it: it prices reported token usage and keeps the judge independent of it. Unset: a judge of any family scores the answers.
preset	claude_code | codex | opencode | custom	required	The harness: claude_code (claude -p), codex (codex exec), opencode (opencode run), or custom (your own command). A preset builds the command line, picks the event parser and switches the harness's network tools off.
command	list	unset	The command of a custom harness, one argument per item; {prompt}, {model}, {workspace} and {task_file} are filled in. With a preset it replaces the preset's command line (the parser stays).
harness_model	str	unset	The model name as the harness spells it. Unset: derived from model (anthropic:claude-haiku-5-5 -> claude-haiku-5-5; for opencode, openrouter:z-ai/glm-5.3 -> openrouter/z-ai/glm-5.3). Neither set: the harness's own default model.
output_format	claude_code | codex | opencode | text	unset	How to read the harness's stdout: the event stream of claude_code, codex or opencode, or text (stdout is the answer, no trajectory). Unset: the preset's own (text for custom).
answer_file	str	unset	A file in the workspace the harness writes its answer to (the task says so). Unset, or not written: the answer is the harness's final message.
system_prompt	str	unset	Text added to the harness's system prompt (claude_code) or to the instructions file (the others).
instructions	str	unset	A file copied into the workspace as the harness's instructions file (AGENTS.md; claude_code reads it as an appended system prompt).
files	dict	empty	More harness files: workspace path -> a file or directory to copy there (a skills directory, a settings file, ...).
network	off | on	off	off: the harness's web tools are switched off (Claude Code's WebFetch/WebSearch, Codex's sandbox network, OpenCode's webfetch). A custom harness cannot be switched off; RAGBench warns.
timeout_s	float	300.0	Seconds one question may take; the harness is then stopped and the question fails.
max_steps	int	30	Tool calls per question; the harness is stopped at the next one and its last message is the answer.
env	dict	empty	Extra environment variables for the harness; ${VAR} is filled from the environment.
limits	block	its fields below	How many harnesses run at once (max_concurrent_requests) or start per minute.
limits.max_concurrent_requests	int	unset	Requests in flight at once, across all questions.
limits.requests_per_minute	int	unset	Requests started per rolling minute.

This system takes no chunker: section.

This system takes no harness: (its prompts are its routes' or live outside RAGBench).

corrective

Corrective RAG. Retrieve, grade each chunk with the LLM, and when too little is relevant rewrite the query and widen the search; optional answer self-check.

Best for: Corpora where the first search often misses and a bad context should be noticed rather than answered from. Cost: high; latency: slow.

retrieval: options

Option	Type	Default	Description
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
retriever	hybrid | vector	hybrid	Search used for every query: hybrid (BM25 + vector, fused with RRF) or vector alone.
per_query_top_k	int	20	Candidates per query before the rankings are merged (raised to the retrieval depth if smaller).
max_cost_usd	float	unset	Stop the agent's extra work once this question has cost this much (USD, at standalone prices); the answer is then written from the evidence so far. No cap by default.
max_tokens	int	unset	Stop the agent's extra work once this question has used this many LLM tokens (prompt + completion). No cap by default.
max_rounds	int	2	Retries after the first retrieval when too little relevant evidence was found (0 = grade only, never retry).
grade_top_k	int	5	Chunks of the ranking the LLM grades per round (graded in one call).
min_relevant	int	1	Chunks graded relevant needed to stop retrying.
expand_to	hybrid | full_doc | none	hybrid	How a retry widens the search besides rewriting the query: hybrid searches with BM25 + vector at twice the depth and grades twice as many chunks, full_doc also adds every chunk of the two best documents (right document, wrong chunk), none only rewrites.
self_check	bool	false	After answering, ask the LLM whether the answer is supported by the context and, if not, regenerate once from the supported claims. Costs one or two extra calls.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, query_rewrite_prompt, few_shot; knobs top_k, chunk_size, max_rounds.

grep_agent

Grep agent (no index). An index-free agent: it lists, greps and reads the raw documents with tools. No chunking, no embeddings, no vector store.

Best for: Small or exact-match-heavy corpora, and testing whether you need retrieval infrastructure at all. Cost: high; latency: slow.

retrieval: options

Option	Type	Default	Description
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
max_cost_usd	float	unset	Stop the agent's extra work once this question has cost this much (USD, at standalone prices); the answer is then written from the evidence so far. No cap by default.
max_tokens	int	unset	Stop the agent's extra work once this question has used this many LLM tokens (prompt + completion). No cap by default.
agent_mode	native | react_json	native	native uses the provider's function calling. react_json describes the tools in the prompt and parses one JSON action per reply, for models without function calling.
max_steps	int	6	Most model turns (each may call several tools). When it is reached the agent is asked to answer with what it has.
max_tool_calls	int	unset	Most tool calls per question; further calls are refused and the agent is asked to answer. No cap by default.

This system takes no chunker: section.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, tool_guidance, few_shot; knobs top_k, max_steps.

iterative

Iterative multi-hop. Search, ask the LLM what is known and what is missing, search for the missing part, and repeat until it says it is done or the hop limit is reached.

Best for: Multi-hop questions whose second search depends on what the first one found. Cost: high; latency: slow.

retrieval: options

Option	Type	Default	Description
rrf_k	int	60	Reciprocal Rank Fusion smoothing constant.
final_top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
top_k	int	unset	Alias for final_top_k; final_top_k wins when both are set.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
retriever	hybrid | vector	hybrid	Search used for every query: hybrid (BM25 + vector, fused with RRF) or vector alone.
per_query_top_k	int	20	Candidates per query before the rankings are merged (raised to the retrieval depth if smaller).
max_cost_usd	float	unset	Stop the agent's extra work once this question has cost this much (USD, at standalone prices); the answer is then written from the evidence so far. No cap by default.
max_tokens	int	unset	Stop the agent's extra work once this question has used this many LLM tokens (prompt + completion). No cap by default.
max_hops	int	3	Most search rounds; each asks the LLM what is known, what is missing, and what to search next.
chunks_per_hop	int	3	Top chunks of each round the LLM sees as evidence (the final answer reads the merged ranking).

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, query_rewrite_prompt, few_shot; knobs top_k, chunk_size, max_hops.

llm_heavy

LLM-heavy. LLM-driven ingestion metadata, query rewriting, and reranking.

Best for: Higher-cost, quality-oriented experiments. Cost: high; latency: slow.

retrieval: options

Option	Type	Default	Description
top_k	int	unset	Chunks given to the generator. Default 5; when unset the evaluator uses evaluation.context_k.
vector_store	str	numpy	Vector backend: numpy (exact, no extra), faiss (exact) or faiss_hnsw (approximate) with pip install 'ragbench[faiss]', chroma (approximate) with ragbench[chroma], qdrant (exact, local mode) with ragbench[qdrant]. in_memory is a deprecated alias of numpy. A missing library is an error, never a silent fallback.
persist_directory	str	unset	Directory for a persistent store (chroma, qdrant). Default: in memory.
reranker_model	str	unset	Hugging Face model for reranker: cross_encoder (default BAAI/bge-reranker-base). Needs pip install 'ragbench[rerank]'.
per_query_top_k	int	10	Candidates per rewritten query (raised to the retrieval depth if smaller).
reranker	str	simple_keyword_overlap	Reranker used when llm_features.enable_llm_rerank is off: simple_keyword_overlap, local_relevance (TF-IDF), cross_encoder (needs the rerank extra), or llm.

llm_features: options

Option	Type	Default	Description
enable_llm_ingestion	bool	false	Enrich every chunk at ingestion with an LLM summary, entities, and hypothetical questions.
enable_query_rewrite	bool	true	Rewrite each question into several search queries with the LLM.
enable_llm_rerank	bool	false	Rerank candidates with an LLM call.

harness: (configuration): prompts system_prompt, grounding, refusal_policy, answer_instructions, citation_style, query_rewrite_prompt, few_shot; knobs top_k, chunk_size, reranker.