Fair means fixing the conditions.

The unit of comparison

A FairBench score belongs to a complete run tuple—not to a model name floating by itself. Any change to the tuple creates a new run and cannot silently overwrite the old one.

Model artifact

Model ID plus the revision, artifact digest, and quantization declared by the operator.

Harness

System prompt, answer contract, parser, retry policy, and manifest hash.

Suite

Task inputs, deterministic graders, weights, category map, and manifest hash.

Inference

Endpoint, request settings, timeout, and limited runner-host fields recorded in the bundle.

A small suite you can inspect

Core v1 contains 24 original tasks across reasoning, mathematics, structured extraction, and instruction compliance. It is deliberately small enough to read end-to-end and fast enough to run on a laptop.

The founding suite is a calibration release. It exercises the protocol and runner before seasonal, pre-committed evaluation shards are added. No copyrighted benchmark questions are copied into the suite.

Deterministic graders only

Core v1 uses exact-match normalization, regular expressions, JSON field checks, and deterministic value comparisons. There is no hidden LLM judge and no repair pass after malformed output.

• Exact-match and regular-expression tasks receive either 0 or 1.
• json_fields tasks split one point equally across expected leaf fields. Each matching leaf earns fractional credit; when extra root fields are forbidden, any extra or missing root field makes the task score 0.
• A task is counted as passed only when it earns full credit. The passed-task count therefore differs from the headline score whenever a json_fields task earns partial credit.
• Parser failure, timeout, refusal, and missing output score 0.
• Category scores average task credit first; the headline score is the macro-average of the four category scores.
• Unattempted tasks cannot be removed from the denominator.

Ranks do not erase uncertainty

FairBench reports the score with task count, coverage, category breakdown, failures, and a 95% bootstrap confidence interval. When intervals overlap, the interface avoids pretending that a tiny numeric difference proves a stable ordering.

Core v1 is a 24-task calibration suite, so its interval will be wide. That is evidence about the measurement—not a visual defect to hide.

Every result carries receipts

A result bundle stores the exact task input hash, stored model output, parsed answer, grade, status, and latency for every item. It also stores runner, suite, harness, model, and recorded inference descriptors. If an endpoint echoes the exact configured bearer-token string, that value is redacted before the output is persisted and hashed. Transformed, split, or unrelated secrets cannot be detected, so the endpoint must be trusted.

Reference

Run by maintainers against a pinned artifact and published with its complete bundle.

Verified

Reserved for a future independently replayed tier. The v0.1 catalog rejects this label until replay linkage, tolerance, and operator-independence rules are enforced.

Community

Schema and hashes validate, but operator provenance and execution context are self-reported and no independent replay is implied.

What this score cannot say

It cannot declare an absolute “smartest model,” compare modalities it does not test, or fully detect training-data contamination. Latency on different hardware is recorded for diagnosis, not ranked as a universal speed result.