Leaderboard
task — the correct value appears somewhere in the reply. strict — the whole reply is exactly that value, no prose, no fences. Δ — what the worked example changed. Click a model for its per-case detail; click a column to sort.
| # | Model | Size | Task · plain | Strict · plain | Task · ex | Strict · ex | Δ |
|---|
Score against model size
Best of the two prompt modes. Models with undisclosed parameter counts are omitted.
Where models struggle
Task score per conversion direction, top 20 models. Darker is better.
What each constraint costs
Task score on cases carrying each constraint, averaged over every model, against the no-constraint baseline. Each one is an independent instruction the model has to honour while reformatting.
Does a worked example help?
Change in task score when one solved example of the same direction is prepended to the prompt. Blue = the example helped, red = it hurt.
Explore the test set
Every one of the 2,025 cases, with the reference answer and how each model did on it.
What is measured
Each case gives the model data in one format and asks for it in another, plus a set of transformation rules. The reply is graded on two axes:
- task — the correct value appears anywhere in the reply. Prose and code fences are tolerated. This asks: did the conversion work?
- strict — the entire trimmed reply parses as the target format and equals the expected value. This asks: can it follow "reply with only the value"?
Comparison is structural, not textual: object keys are order-insensitive, arrays are ordered, and types
are strict — "22" is not 22. A reply that says {"rating": "5"} when
the answer is {"rating": 5} is wrong, because preserving types is the whole job.
The axes
Directions (9)
text→json · text→yaml · text→csv · json→yaml · json→csv · yaml→json · yaml→csv · csv→json · csv→yaml
The CSV→* directions are the hardest, and that is not an accident: CSV erases types, so the model has to reconstruct which cells were numbers, which were booleans, and which were strings that merely look numeric.
Structural levels (5)
Constraints (7)
Each is an orthogonal instruction, combined in tiers from zero to all-applicable:
Domains (8)
Prompt modes
Every model is measured twice. plain sends the instructions alone. explained prepends one solved example of the same direction, drawn from a train split that shares no inputs with the test set. The difference between them is the Δ column.
Limitations
enum constraint is partly unguessable in v1.0. It asks the model to normalise
category text to a canonical form, giving examples like IN STOCK → in_stock. But some
expected values cannot be derived from the instruction: Pre-order → preorder drops the
underscore that the given example implies, and clear → cleared is a different word, not a
case normalisation. On those cases the benchmark measures vocabulary guessing rather than
instruction-following. The dataset is frozen for v1.0 so that all published scores stay comparable;
the leaderboard therefore also carries a task score with enum cases removed. v1.1 will state the
allowed vocabulary in the instruction — a prompt-only change that leaves every expected answer identical.
inkling explained went 8 → 62, and
nemotron-3-ultra-550b on Kilo went 26 → 73, landing exactly where the same model already
scored through a different provider. A truncated reply is now a retryable error, never a zero.
Reproducing this
The harness talks to any OpenAI-compatible endpoint:
git clone https://github.com/hadestructhor/format-bench cd format-bench && bun install # prove the harness agrees with the reference answers (should print 2025/2025) bun bench.ts gold # score your own model bun bench.ts run --base-url http://localhost:8000/v1 --api-key sk-… \ -m my-model --save-responses bun bench.ts run --base-url http://localhost:8000/v1 --api-key sk-… \ -m my-model --explain --save-responses