Skip to content

Contributing to RAGBench

Thanks for helping improve RAGBench. This project is a practical, evaluation-first RAG benchmark framework, so contributions should preserve reproducibility, local mock-mode execution, and clear reports.

Development setup

Requires Python 3.11 or newer.

git clone https://github.com/alexandre0sheva/ragbench.git
cd ragbench
uv venv --python 3.12 && source .venv/bin/activate      # or: python -m venv .venv && source .venv/bin/activate
uv pip install -e ".[dev,anthropic,pdf,docx,hub]"           # the extras CI installs, so the SDK and loader tests run instead of skipping
uv pip install -e ".[langchain,llamaindex,haystack]"         # optional: the framework adapter tests (`tests/test_framework_adapters.py`) skip without them; the `frameworks` CI job runs them
uv pip install -e ".[mcp]"                                   # optional: the MCP SDK tests (`tests/test_mcp_sdk.py`) skip without it; the `mcp` CI job runs them
uv pip install -e ".[otel]"                                  # optional: the OTLP export tests (`tests/test_trace_export.py`) skip their sending half without it; the `otel` CI job runs them
uv pip install -e ".[docs]"                                  # optional: build the docs site locally (`python scripts/build_site.py`); the `site` CI job builds it on every change
ragbench demo
pytest -m "not slow" -n auto                      # the fast loop (under a minute on a laptop); plain `pytest` runs everything

Every dependency floor in pyproject.toml is the latest release the day it was last checked, so a fresh environment is the supported one: uv pip list --outdated should print nothing in it (the one exception is a package that is itself pinned by another, such as mpmath under PyTorch). Raise the floors, never pin an old major; an upper bound needs a comment naming the incompatibility. GitHub Actions and pre-commit hooks follow the same rule (pre-commit autoupdate), and Dependabot opens one grouped pull request a week for pip and for github-actions.

To try the container instead: docker build -t ragbench . then docker run --rm ragbench demo (the image has no key and runs as a non-root user; CI builds it on every push).

RAGBench must work without OPENAI_API_KEY. Tests must not need the network, a paid API or a model download: use the mock models and the fixtures in tests/conftest.py (a tiny dataset, a scripted LLM), and tests/fake_openai_server.py for code that talks to a real SDK. Every ragbench command and option is in docs/cli.md.

The checks a change must pass

CI runs these; run them before you push.

ruff check .
mypy
pytest -n auto --cov                         # everything, in parallel; fails below the coverage floor in pyproject.toml; raise the floor when coverage rises, never lower it
python scripts/generate_docs.py --check      # docs/systems.md, docs/cli.md, docs/tools.md and the README system table match the code
python scripts/audit_docs.py                 # no duplicated or broken documentation (see below)
ragbench compare --config configs/all.yaml --mock --max-workers 4
  • Slow tests. A test that takes several seconds even in parallel (a full parallel run, the live path) carries @pytest.mark.slow. CI runs the fast tests (-m "not slow") on every Python version and the whole suite once, with coverage, so mark a new test slow rather than skipping it; --strict-markers is on, so a misspelt marker is an error. mypy runs with check_untyped_defs: annotate new code, and do not add blanket ignores.
  • Golden snapshot. tests/golden/mock_metrics_v3.json pins what every system retrieves and answers in a mock run. It must not change unless your change is meant to alter retrieval or answers; then run python scripts/update_golden.py, say so in the pull request, and note it in the changelog. RAGBENCH_GOLDEN_STRICT=1 pytest tests/test_golden_mock.py is the strict check.
  • Report structure. tests/golden/report_structure_v1.json pins the sections and data keys of report.html. After an intended change: RAGBENCH_UPDATE_SNAPSHOTS=1 pytest tests/test_report_snapshot.py.
  • Every system and chunker is tested end to end by tests/test_e2e_matrix.py as soon as it is registered: it must run, write every output, trace its cost, and be reproducible. Fix what it finds rather than excluding a component.

Guidelines

  • Keep changes modular and dataset-agnostic; match the style of the code around you (from __future__ import annotations, Pydantic v2, line length 140).
  • Do not add required external services for default runs, and preserve mock mode.
  • Add or update tests for behavior changes (tests first for logic such as metrics, parsing, routing, caching and statistics).
  • Keep demo data fictional and safe to publish. data/hard is generated: change scripts/generate_hard_suite.py and run it, never the files.
  • Do not commit results/, .env, caches or local vector-store state.
  • Every change adds a bullet to CHANGELOG.md, under a ## [Unreleased] heading above the latest release (add the heading if it is not there).

Good first issues

Small, self-contained and testable offline. Each names where to start; open an issue (or comment on one) before you begin so two people do not build the same thing. Maintainers tag these good first issue.

  • An agent-harness preset. cli_agent knows Claude Code, Codex and OpenCode (CliPreset in src/ragbench/rag_systems/options.py, parsers in cli_trajectory.py). Add another harness (Gemini CLI, Aider, Goose) with a fake binary like tests/fake_cli_agent.py, and document it in benchmarking-external-systems.md.
  • A dataset for ragbench datasets pull. One HubDataset in src/ragbench/datasets/hub/ with its files' checksums, licence and a reader; tests/test_hub_catalog.py and tests/test_hub_pull.py show the checks (no download in tests).
  • An agent tool. A typed function such as a unit converter or a table lookup, registered like the tools in src/ragbench/tools/ and described in docs/tools.md, with a test in the style of tests/test_tools.py.
  • A chunker for a format that has structure (source code, HTML, subtitles): docs/extending.md and tests/test_chunkers.py.
  • A harder question type for the bundled hard suite. Add a category to scripts/generate_hard_suite.py, regenerate data/hard, and extend tests/test_hard_suite.py.
  • A price or a model that moved. ragbench models sync, the price table in src/ragbench/models/cost.py and the shortlist in docs/models.md must agree; fix whichever drifted and say where you read the price.
  • Raise the coverage of one module. pytest --cov lists the modules furthest below the floor; add tests for the behaviour they miss, then raise fail_under if the total allows.
  • A walk-through for your own corpus. A short guide in the style of choosing-an-architecture.md for a kind of data it does not cover yet (support tickets, source code, legal text), from a real mock run.

Larger, and wanted (help wanted): a Postgres / pgvector vector backend (extending.md) and multilingual evaluation.

Issue labels

.github/labels.yml is the list; python scripts/sync_labels.py --apply (maintainers, needs the gh CLI) creates or updates them on the repository, and without --apply it prints what it would do.

Label Use
bug, enhancement, question What kind of issue it is (the issue templates set the first two)
good first issue, help wanted Starting points for new contributors; larger wanted work
area: systems, area: evaluation, area: models, area: adapters, area: optimize, area: reporting, area: docs Which part of the code it touches
cost, mock-mode Spends or may spend money; breaks or lacks the deterministic mock path
needs repro, needs decision, wontfix Waiting for a reproduction or a maintainer decision; closed as out of scope

Commit messages

One short subject, then a body only when the why is not obvious from the diff.

  • Subject: imperative, at most 72 characters, no trailing full stop, optionally prefixed with the area (optimize: keep the base when dev disagrees). The areas are the area: labels above without the prefix.
  • Body: wrap at 80 characters; say what changed for the user and why, mention a golden or report-snapshot change, and give any API-cost implication. Reference an issue as Fixes #123.
  • One change per commit, with its tests and its CHANGELOG.md bullet. Do not mix a refactor with a behaviour change.

Documentation

Each piece of knowledge has one owner; everywhere else links to it. scripts/audit_docs.py (also a test) fails on broken links, option tables and system descriptions outside their owners, metric definitions outside the methodology, and counts that will go stale.

Knowledge Owner
Pitch, install, quickstart, links README.md
Systems, their options, cost and latency profile docs/systems.md (generated)
How each system, chunker, reranker and agent works (algorithms, diagrams) docs/system-wiki.md
Commands and options docs/cli.md (generated)
Config sections, presets, sweeps, budgets, caching, providers docs/configuration.md
Which models to use, model shortlists and prices, OpenRouter docs/models.md
Dataset formats and onboarding docs/dataset-format.md
Metrics, the judge, statistics, cost accounting, limitations docs/methodology.md
Tools and agents docs/tools.md (tool table generated)
Writing a new system, chunker, tool, provider or plugin docs/extending.md
Which architecture to try and how to read the result docs/choosing-an-architecture.md
A measured comparison of decision_gate with the systems it replaces, and when to try it docs/decision-gate-evaluation.md
Exporting a run's traces (ragbench export-traces): formats, backends, what is and is not exported docs/observability.md
Result-file schemas, the cross-run page (ragbench runs site), the docs site and its Leaderboard docs/results.md
Optimizing a system's harness (ragbench optimize): the loop, its budget, outputs, resuming docs/optimization.md
How RAGBench differs from RAGAS, ARES, TruLens, DeepEval, promptfoo and Harness² docs/comparison.md
Release history CHANGELOG.md
Releasing docs/release-checklist.md

After adding or changing a system, option, command or tool, run python scripts/generate_docs.py.

Adding a RAG system

See docs/extending.md: declare the options and a SystemSpec, register the class, then regenerate the docs. Add a config example and at least one focused test; the end-to-end matrix covers the rest.

Pull requests

Include a concise explanation of the evaluation impact and mention any API-cost or latency implications. The pull request template asks for the same.

Maintainers: repository setup

  • Enable GitHub Actions and Dependabot (.github/dependabot.yml groups routine bumps).
  • Enable private vulnerability reporting (see SECURITY.md).
  • Protect main once there are outside contributors, and require the CI jobs to pass before merging: Lint and types, Generated docs are current, Tests (Python 3.11 to 3.14), Tests with every optional extra, CLI smoke test and Package build. Tests (Python 3.15) and the dependency audit are advisory.
  • Never commit .env, results/, dist/, build/, *.egg-info/ or caches (git status --short before a first push).