reference run · structured-json/v1

SmolLM2 1.7B Instruct

2.8%

0 of 24 tasks earned full credit. The headline score averages task credit, including fractional JSON-field credit from 1 task. Every stored item record below is part of bundle 949fd9833aa0…f96f47cf.

← Back to explorer Download raw bundle ↗

Score (%): 2.8%
Fully passed tasks: 0/24
95% bootstrap CI: 0.0%–8.3%
Coverage: 24/24
Parse failures: 21
Harness: Structured JSON v1
Model revision: Q4_K_M
Mean latency: 60.6 ms
Completed: 2026-08-03 17:37:58 UTC
Model digest: decd2598bc2c…a30d4f33
Suite digest: 069366e2e1fd…c11c0b0b
Harness digest: 577ade87bc3d…6c68fca5
Runner digest: b4eb95ad9a05…69bd09d5

Category breakdown

The average does not hide the shape.

Each category has six tasks. Category scores average numerical task credit—including fractional json_fields credit—then the headline macro-averages the four categories.

information extraction11.1%

instruction following0.0%

logical reasoning0.0%

quantitative reasoning0.0%

Recorded configuration

Start from the recorded settings.

This template selects the recorded model descriptor, endpoint, request settings, suite ref, and harness ref in your current checkout. Those refs and the current runner may differ from the bundle's embedded definitions. Compare them and match runner SHA-256 b4eb95ad9a05…69bd09d5 before calling the run a protocol replay.

Recorded-configuration command

$ npm run fairbench -- run \
  --suite 'core/v1' \
  --harness 'structured-json/v1' \
  --model 'smollm2-1.7b-instruct' \
  --endpoint 'http://127.0.0.1:8082/v1/chat/completions' \
  --revision 'Q4_K_M' \
  --artifact-digest 'decd2598bc2c8ed08c19adc3c8fdd461ee19ed5708679d1c54ef54a5a30d4f33' \
  --quantization 'GGUF-Q4_K_M' \
  --timeout-ms 120000 \
  --seed 17 \
  --max-tokens 256 \
  --temperature 0 \
  --top-p 1 \
  --out 'result.json'

Task receipts

All 24 task records, including the misses.

Expand a row to inspect the prompt, stored response, parsed value, grader outcome, and item hash. The exact configured bearer-token string is redacted if an endpoint echoes it; transformed or unrelated secrets are not detectable.

01quant-01-cratesquantitative reasoningMiss49.3 ms
Prompt
A depot receives 18 crates with 7 lamps in each crate. Nine lamps are removed for inspection. How many lamps remain? Answer with the integer only.
Stored model output
```
answer = 12
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 57ed14599e2359421bbac363091903f53c72950036c4b37c06162722acff4455
02quant-02-ratioquantitative reasoningMiss32.2 ms
Prompt
Red and blue tokens are in the ratio 3:5. There are 64 tokens altogether. How many are red? Answer with the integer only.
Stored model output
```
answer = 32
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: e86a1c406e7af17044ea608e06d672be91bd48cefa8a49d3a3cc14c9e7414ea8
03quant-03-discountquantitative reasoningMiss32.0 ms
Prompt
A notebook costs 45 dollars before a 20 percent discount. What is the discounted price in whole dollars? Answer with the integer only.
Stored model output
```
answer = 36
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 818463be6c0bc7fc37005ae9be33e4cf321a30def3cf1e07f63d379034b0e92a
04quant-04-averagequantitative reasoningMiss31.3 ms
Prompt
Four readings are 14, 19, 22, and 25. What is their arithmetic mean? Answer with the number only.
Stored model output
```
answer = 19
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 7f3d7b0f5f793aede5f2f00ce8a08f97c4c8816797de18a236c6e213f52bd53d
05quant-05-clockquantitative reasoningMiss63.0 ms
Prompt
A workshop begins at 09:40 and lasts 2 hours 35 minutes. Give the ending time in 24-hour HH:MM format.
Stored model output
```
{
  "answer": "21:15"
}
```
Parsed answer
```
"21:15"
```
Grader receipt
```
{
  "details": {
    "actual": "21:15",
    "flags": "",
    "pattern": "^12:15$"
  },
  "grader": "regex",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 7e50bd62e398aea77d2f863c0124bf7eccd1db23c9ae5603280a451ea07c8bb8
06quant-06-areaquantitative reasoningMiss36.7 ms
Prompt
A rectangle is 12.5 meters long and 8 meters wide. What is its area in square meters? Answer with the number only.
Stored model output
```
answer = 100
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 19a432843dd1d47d569a89094d8ce986b3b8c41b53426f8799871c169a62072b
07logic-01-rillslogical reasoningMiss24.4 ms
Prompt
Every Rill is a Tam. No Tam is a Vek. Can any Rill be a Vek? Answer yes or no.
Stored model output
```
Answer: no
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: ed2385dcce176c8f48337ef333c5334d96e8fe2c2b6bb0fc58559afd3958d353
08logic-02-orderlogical reasoningMiss19.8 ms
Prompt
Mina arrives before Oren, and Oren arrives before Pia. Who arrives first? Answer with one name.
Stored model output
```
Oren
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 887f5986ba43fcae476e0c89c929cc360be12a2256e419e1ae8509e10a5c3d25
09logic-03-switchlogical reasoningMiss18.0 ms
Prompt
A lamp is on if and only if its switch is up. The lamp is off. Is the switch up or down? Answer with one word.
Stored model output
```
down
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 86d62455d0ec0028f643ece6bc7d219736d0c445eeb84478e520b1d34c387118
10logic-04-schedulelogical reasoningMiss24.6 ms
Prompt
Ada works Tuesday and Thursday. Ben works Monday and Thursday. Cora works Thursday and Friday. Which weekday do all three work? Answer with the weekday only.
Stored model output
```
answer = Tuesday
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 4b5f954cc0bb4b82b1611a9fc1bd89fc5aabef172b7029570dd8120c9e4e098d
11logic-05-badgelogical reasoningMiss16.1 ms
Prompt
Rule: every triangular badge is blue. A particular badge is not blue. Can that badge be triangular? Answer yes or no.
Stored model output
```
no
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "regex",
  "passed": false,
  "score": 0
}
```
Input SHA-256: a89096bb1b64b85311c3c5b3c28cced674ad3b0f602648b47b00b0e3cbd670f7
12logic-06-positionlogical reasoningMiss45.9 ms
Prompt
K is east of J. L is west of J. List the three letters from west to east, separated by commas with no spaces.
Stored model output
```
answer = [L, K, J]
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: c7567d53511d94e2d9abf6db1871d5bcc22e80a4533b7730fb0e19654c961954
13extract-01-shipmentinformation extractionMiss113.3 ms
Prompt
Shipping note: Batch Q7 contains 14 crates and is headed to Busan. Return a JSON object with fields batch, crates, and destination. crates must be a number.
Stored model output
```
{
  "batch": "Q7",
  "crates": 14,
  "destination": "Busan"
}
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "json_fields",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 949965a08e5f301dff120b349502146918236829d3f79e5954e90f93e4382620
14extract-02-eventinformation extractionMiss154.7 ms
Prompt
Event card: The Lantern Review is scheduled for 2026-09-12 at 18:30. Return a JSON object with fields title, date, and time.
Stored model output
```
{
  "title": "The Lantern Review",
  "date": "2026-09-12",
  "time": "18:30"
}
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "json_fields",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 5a0772b1217500be8a0e8a53489b5aae07f77d4b93c6ce3008ea04a2ea17b09d
15extract-03-sensorinformation extractionMiss146.6 ms
Prompt
Sensor log: Unit pine-4 recorded 18.6 degrees Celsius and reported status stable. Return a JSON object with fields unit, value, scale, and status. value must be a number and scale must be C.
Stored model output
```
{
  "unit": "Unit pine-4",
  "value": 18.6,
  "scale": "C",
  "status": "stable"
}
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "json_fields",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 0d918c4b6321def1d0cbcb528bfd2a4e5cb837224f829838c73b7e8461420099

16extract-04-recipeinformation extractionPartial150.7 ms

Prompt

Recipe note: Use 240 grams of chickpeas; the dish is vegan. Return a JSON object with fields ingredient, grams, and vegan. grams must be a number and vegan a boolean.

Stored model output

{
  "answer": {
    "ingredient": "240 grams of chickpeas",
    "grams": 240,
    "vegan": true
  }
}

Parsed answer

{
  "grams": 240,
  "ingredient": "240 grams of chickpeas",
  "vegan": true
}

Grader receipt

{
  "details": {
    "matched": 2,
    "mismatches": [
      "ingredient"
    ],
    "total": 3
  },
  "grader": "json_fields",
  "passed": false,
  "score": 0.6666666666666666
}

Input SHA-256: f045f31f38e6aaee5b5608bd6a3f320b5ae830d03b0f52868657ffa48476d3bc

17extract-05-cataloginformation extractionMiss132.9 ms
Prompt
Catalog line: Glass Orchard, published in 2031, carries the tags climate and mystery in that order. Return a JSON object with fields title, year, and tags. year must be a number and tags an array.
Stored model output
```
{
  "title": "Glass Orchard",
  "year": 2031,
  "tags": ["climate", "mystery"]
}
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "json_fields",
  "passed": false,
  "score": 0
}
```
Input SHA-256: dfc6547940005d5f926691bd3ae2c5ead7096d148b17b9136b5ce9da4ea4224b
18extract-06-weatherinformation extractionMiss136.9 ms
Prompt
Weather brief: Daejeon expects a high of 29 and a low of 18, with no advisory. Return a JSON object with fields city, high, low, and advisory. Temperatures must be numbers and advisory must be null.
Stored model output
```
{
  "city": "Daejeon",
  "high": 29,
  "low": 18,
  "advisory": null
}
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "json_fields",
  "passed": false,
  "score": 0
}
```
Input SHA-256: a354272b45f595b5de6f24035f2215be1e93ab857f433048bfe7fab81a9f921d
19follow-01-lowercaseinstruction followingMiss21.5 ms
Prompt
Write the word FAIR in lowercase. Output that word only.
Stored model output
```
faire
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 0e9de6ef7276e328dd1d1fcbba6aa3fda7e25d4150af56fd0803a72fdb85cf3d
20follow-02-codeinstruction followingMiss53.5 ms
Prompt
Output the code FB-207 exactly, including the hyphen, and nothing else.
Stored model output
```
{"answer":{"FB-207"}}
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "regex",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 5cf4f859cc67feac7697d3f94177b0de5c381a3d325a26d8afb8757f6406fee0
21follow-03-third-wordinstruction followingMiss23.2 ms
Prompt
From the phrase 'cedar amber river silver', output only the third word.
Stored model output
```
answer = silver
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 0ffff63da206b3899a63e9570aad9120cae24928e5cec18d2f442750d4aac485
22follow-04-sortinstruction followingMiss42.0 ms
Prompt
Sort mango, apple, and kiwi alphabetically. Output the result exactly as lowercase words separated by a vertical bar with one space on each side.
Stored model output
```
answer = "mango apple kiwi"
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "exact",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 5ad5fe99cc776c7a6bd94c469c17df729d877f56e2fbb97db120a149e419d2fa
23follow-05-objectinstruction followingMiss50.5 ms
Prompt
Return a JSON object with exactly two fields: left must be SUN and right must be MOON.
Stored model output
```
{
  "answer": "SUN"
}
```
Parsed answer
```
"SUN"
```
Grader receipt
```
{
  "details": {
    "error": "Unexpected token 'S', \"SUN\" is not valid JSON",
    "matched": 0,
    "total": 2
  },
  "grader": "json_fields",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 24b88c139429763b6eb9803da8db48b4799c02609f9493a0f13c9b2f3ee6b1d5
24follow-06-title-hypheninstruction followingMiss34.6 ms
Prompt
Transform 'quiet blue harbor' by capitalizing the first letter of each word and joining the words with hyphens. Output only the result.
Stored model output
```
quiet-blue-harbor
```
Parsed answer
```
null
```
Grader receipt
```
{
  "details": {
    "reason": "parse_error"
  },
  "grader": "regex",
  "passed": false,
  "score": 0
}
```
Input SHA-256: 26f24b024c3e7fa8e8ef32131a37c2db3064982c536e44f8cf74d2401409261b