Open source · Inspectable · Zero ads
Measure the model.
Hold everything else still.
FairBench runs every model through the exact same tasks and harness, then swaps the harness and runs it again. No sponsored rank. No hidden weighting. Every score comes with receipts.
24original tasks
2reference models
2pinned harnesses
24/24task records kept
The founding matrix
Same suite. Two honest comparisons.
Read down a column to compare models under one harness. Read across a row to see what the harness changes for the same model.
Showing FairBench Core v1 results for Minimal answer v1.
Swipe or scroll horizontally to inspect every column →
Scores are macro-averaged across four categories. Missing tasks, malformed output, and timeouts count as zero.
Open the full explorer →The fairness contract
A score is only as fair as its conditions.
FairBench treats the model, harness, suite, and runtime as separate versioned inputs. Change one, and it is a new result.
- 01
Same constraints
Identical tasks, token limits, sampling settings, retry policy, and scoring rules.
- 02
Pinned harnesses
Every prompt, parser, and adapter is versioned and hashed. The wrapper is part of the result.
- 03
Receipts, not screenshots
Scores ship with item outputs, parser outcomes, latency, recorded runtime fields, and a recorded-configuration command template.
- 04
Uncertainty stays visible
Coverage, category breakdowns, failures, and confidence intervals live next to the headline score.
The benchmark is public
A plan to seal the next answers.
The proposal combines open methodology, rotating seasonal suites, timestamped manifest commitments, and contamination canaries. The aim is to make scoreboard-only optimization harder without blocking audit.
We cannot make post-release optimization disappear. We can label it, preserve old runs, and make provenance impossible to hand-wave away.
Run it yourself
Your machine. The same protocol.
The dependency-free runner speaks to any OpenAI-compatible local endpoint. It records coverage and hashes the complete result bundle; catalog inclusion remains a separate maintainer decision.
$ npm run fairbench -- run \ --model qwen2.5:1.5b \ --harness minimal-answer/v1 \ --suite core-v1 \ --out result.jsonBecause intelligence should be measured fairly.