fairbench.aiIndependent model evaluationOpen source
Menu

Run it yourself

Challenge the score with a rerun.

FairBench is a dependency-free Node runner plus plain JSON manifests. Point it at an OpenAI-compatible local endpoint. No hosted account, telemetry, or FairBench API key is required.

Clone, inspect, then run

The suite and harness files are meant to be read before they are run. Node 22 or newer is the only runner dependency.

git clone https://github.com/memekr/fairbench.git
cd fairbench
npm install
npm test

Run one model through one harness

Start your local server, then provide its OpenAI-compatible base URL. The runner saves after every item, so an interrupted run can resume.

FAIRBENCH_BASE_URL=http://127.0.0.1:8080/v1 \
npm run fairbench -- run \
  --model qwen2.5:1.5b \
  --harness minimal-answer/v1 \
  --suite core-v1 \
  --out results/qwen-minimal.json
Use FAIRBENCH_API_KEY only when your own endpoint requires it. The runner never writes the key into the result bundle.

Validate before you compare

Validation checks the schema, suite and harness hashes, required provenance, complete task coverage, item input hashes, and final bundle digest.

npm run fairbench -- validate results/qwen-minimal.json

A pull request starts the replay

  1. Validate locally

    Include the unedited result JSON and pass the repository test suite.

  2. Disclose provenance

    Identify your relationship to the model and any benchmark-specific tuning you know about.

  3. Open a pull request

    CI validates structure and hashes. Passing CI makes a result eligible, not verified.

  4. Independent replay

    A maintainer reruns the pinned artifact before promoting the result to verified.

Open a pull request ↗Read result grades →

Keep the comparison legible

  • Never replace or rewrite a published result bundle.
  • Do not omit timeouts, refusals, parse failures, or unfavorable tasks.
  • Version a harness change even when it looks cosmetic.
  • Keep provider adapters separate from harness logic.
  • Describe limitations in the same pull request as the score.