Benchmark Registry
Immutable, content-hashed benchmark versions. Each task records its scoring method, difficulty, contamination provenance, and tags.
Yuu Golden Regression Suite
goldenThe permanent regression suite run against every Yuu release. Deterministic-first coverage across the capability taxonomy. Immutable at this version.
| Task | Category | Scoring | Judge | Difficulty | Tags |
|---|---|---|---|---|---|
| g-reason-01 | Reasoning | numeric-tolerance | deterministic | medium | arithmetic-trap, word-problem |
| g-reason-02 | Reasoning | exact-match | deterministic | easy | syllogism |
| g-math-01 | Math | numeric-tolerance | deterministic | easy | order-of-operations |
| g-math-02 | Math | numeric-tolerance | deterministic | medium | rates |
| g-code-01 | Coding | unit-test | deterministic | easy | array, implementation |
| g-code-02 | Coding | unit-test | deterministic | medium | stack, parsing |
| g-debug-01 | Debugging | unit-test | deterministic | medium | off-by-one |
| g-sql-01 | SQL | sql-result | deterministic | medium | group-by, aggregate |
| g-tool-01 | Tool Use | numeric-tolerance | deterministic | easy | single-tool |
| g-tool-02 | Tool Use | numeric-tolerance | deterministic | medium | tool-selection |
| g-longctx-01 | Long Context | exact-match | deterministic | medium | needle, single-hop |
| g-longctx-02 | Long Context | exact-match | deterministic | medium | needle, single-hop, shallow |
| g-longctx-03 | Long Context | exact-match | deterministic | medium | multi-hop |
| g-longctx-04 | Long Context | exact-match | deterministic | medium | contradiction, recency |
| g-longctx-05 | Long Context | exact-match | deterministic | medium | temporal-order |
| g-instr-01 | Instruction Following | regex | deterministic | easy | format-constraint |
| g-instr-02 | Instruction Following | exact-match | deterministic | easy | exact-format |
| g-struct-01 | Structured Output | json-schema | deterministic | medium | json, strict-format |
| g-halluc-01 | Hallucination | regex | deterministic | medium | unanswerable, refuse-to-fabricate |
| g-plan-01 | Planning | regex | deterministic | easy | ordering, dependencies |
| g-agentic-01 | Agentic | numeric-tolerance | deterministic | hard | multi-step, tool-chain |
| g-safety-01 | Safety | regex | deterministic | medium | harmful-request, must-refuse |
| g-safety-02 | Safety | regex | deterministic | easy | benign, over-refusal-check |
| g-data-01 | Data Analysis | numeric-tolerance | deterministic | easy | descriptive-stats |
Coding Depth
Algorithmic and implementation tasks graded by unit tests in the sandbox.
| Task | Category | Scoring | Judge | Difficulty | Tags |
|---|---|---|---|---|---|
| code-fizzbuzz | Coding | unit-test | deterministic | easy | classic |
| code-dedupe-stable | Coding | unit-test | deterministic | easy | array |
| code-lru | Coding | unit-test | deterministic | hard | data-structure, cache |
SQL Analytics
Result-set-graded SQL over in-memory fixtures.
| Task | Category | Scoring | Judge | Difficulty | Tags |
|---|---|---|---|---|---|
| sql-region-revenue | SQL | sql-result | deterministic | easy | group-by |
| sql-having | SQL | sql-result | deterministic | medium | having, aggregate-filter |
| sql-avg-revenue | SQL | sql-result | deterministic | medium | aggregate, round |
Structured Output
Schema adherence and semantic correctness of JSON responses.
| Task | Category | Scoring | Judge | Difficulty | Tags |
|---|---|---|---|---|---|
| struct-contact | Structured Output | json-schema | deterministic | medium | extraction, nested |
| struct-enum | Structured Output | json-schema | deterministic | easy | enum, classification |
| struct-nested-tool | Structured Output | json-schema | deterministic | hard | nested-json, tool-result-synthesis |
Long Context
Retrieval, multi-hop, temporal ordering, and summarisation over generated documents.
| Task | Category | Scoring | Judge | Difficulty | Tags |
|---|---|---|---|---|---|
| lc-temporal | Long Context | exact-match | deterministic | hard | temporal-order |
| lc-summary-fidelity | Long Context | numeric-tolerance | deterministic | medium | summarization, fidelity |
| lc-deep-needle | Long Context | exact-match | deterministic | expert | needle, deep, 24k |
Research (LLM-judged)
Open-ended research answers graded by an LLM judge panel. Scores here are model-graded and reported separately from deterministic scores.
| Task | Category | Scoring | Judge | Difficulty | Tags |
|---|---|---|---|---|---|
| research-tradeoffs | Research | llm-judge | model-graded | medium | architecture, open-ended |
| research-explain | Research | llm-judge | model-graded | easy | explanation |
Benchmark creator
New benchmarks are authored as immutable data with the defineSuite builder (see src/lib/benchmarks), or imported as JSONL/CSV. AI-generated candidate tasks are supported but must carry generated: true provenance so a generated task is never mistaken for a hand-verified one. Publishing a change to an existing benchmark creates a new version rather than mutating tasks in place — the content hash enforces this.