Proposed controls · Open threat model v0.1
Make gaming costly. Make claims auditable.
Benchmaxxing is optimization for a visible scoreboard without a corresponding gain in general capability. An open benchmark cannot eliminate it. FairBench is designed to expose its fingerprints.
What we are defending against
The threat is not improvement itself. It is a score that looks general while depending on privileged knowledge, cherry-picked coverage, a model-specific wrapper, or repeated exposure to the exact answers.
- Contamination
- Evaluation items or near-duplicates appear in training or tuning data.
- Harness tuning
- A wrapper is optimized for one model and presented as a neutral comparison.
- Selective runs
- Only favorable seeds, tasks, or completed outputs are submitted.
- Artifact swapping
- The published model label does not identify the exact weights that were run.
- Score laundering
- Self-reported results appear next to independently replayed runs without a visible distinction.
Proposed controls, not magic
These controls are design commitments for a future sealed season. Core v1 is a public calibration suite and does not claim pre-release contamination protection.
Pre-commit the evaluation
Publish a timestamped hash of the suite manifest and rules before the sealed tasks are used.
Rotate seasonal shards
Retire exposed suites without deleting their history; new headline runs use a newly committed shard.
Bind every artifact
Require model, runner, harness, and suite digests plus explicit serving-environment descriptors in every accepted bundle.
Keep failures in the denominator
Partial submissions, parser failures, and timeouts remain visible and score zero.
Probe for leakage
Use canaries, paraphrase families, and suspicious phrase matching as evidence—not as infallible detection.
Replay before verification
A result becomes verified only after an independent run reproduces it within the published tolerance.
A season has a paper trail
- Commit
- Suite hash, scoring rules, supported harnesses, model list, and replay tolerance are timestamped.
- Evaluate
- Maintainer reference runs execute against the sealed shard with frozen artifacts.
- Reveal
- Tasks, manifests, complete result bundles, and provenance are released together.
- Replay
- Independent maintainers reproduce candidate runs; disputes stay attached to the run.
- Retire
- Exposure risk is recorded, the season leaves the headline view, and its history remains public.
Labels prevent false equivalence
- Pre-release reference: run before task reveal under the committed protocol.
- Verified replay: independently reproduced from the same artifact.
- Post-release: run after the suite became public, regardless of who submitted it.
- Community: validates structurally but has not received an independent replay.
- Contested: a provenance, contamination, or reproduction concern is open.
The honest limit
Once a suite is public, post-release optimization and training-data exposure can be detected imperfectly but never ruled out completely. Private tasks reduce auditability; fully public tasks increase gaming pressure. The proposed seasonal design uses temporary secrecy followed by full disclosure to preserve a clean evaluation window and long-term scrutiny.