> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark packs

> Declare an evaluation set: what to run, which numbers to read, and how to read them.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench new my-bench            # scaffold a pack
tp plugin publish ./my-bench     # publish it
tp bench ls                      # it is now in the catalog
tp bench run my-bench -m run:run-4f2a91
```

A benchmark pack answers three questions in YAML, and contains no code: **what to run**, **which
numbers to read**, and **how to run it by default**. The thing that actually executes is a runner
that ships with the SDK.

## Layout

```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
my-bench/
├── plugin.yaml       # capability: benchmark
├── benchmark.yaml    # the declaration
└── README.md
```

## benchmark.yaml

```yaml benchmark.yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
schema: tuneplane/benchmark/v1
name: gsm8k
title: GSM8K · Grade-school math word problems
category: math
summary: >
  8.5K grade-school math word problems requiring 2-8 reasoning steps. The usual entry
  benchmark for math post-training: a clean signal, fast to run, and well correlated with
  the RL reward curve, which makes it a good spot check mid-training.

runner: lm-eval
suites: [gsm8k]
num_fewshot: 5
batch_size: auto
keywords: [math, reasoning, cot]

metrics:
  - key: exact_match,strict-match
    label: Exact match (strict)
    primary: true
  - key: exact_match,flexible-extract
    label: Exact match (flexible)

i18n:
  zh:
    title: GSM8K · 小学数学应用题
    summary: >
      8.5K 道小学数学应用题，需要 2-8 步推理。
```

### Fields

**name** `string` *required*

Leaf name. A published pack is referenced as `<owner>.<name>`; built-ins keep the bare name.

**runner** `lm-eval | evalscope | rtl | judge | safety` *required*

Which harness executes the evaluation. See the table below.

**suites** `string[]` *required*

The suite names the runner understands. For `lm-eval` these are its task names.

**category** `string` (default: general)

Dashboard grouping — `math`, `code`, `knowledge`, `chinese`, `instruction`, and so on.

**num\_fewshot, batch\_size, limit, extra\_args** `mixed`

Defaults for running it. `batch_size` defaults to `auto`, `limit` caps the number of samples. A
user can override all of these at submission; what you declare is what happens when they do not.

This is the point of the pack: nobody should have to remember that GSM8K wants 5-shot or that
HumanEval needs `--confirm_run_unsafe_code`.

**rubric** `\<owner\>/\<name\>`

Only read by the `judge` runner, and for that runner it *is* the benchmark's definition. Swap the
rubric and it is a different benchmark — which is why it is a declared field rather than a flag
buried in `extra_args` that nobody can see from the catalog.

### Metrics

Each entry says which number to pull out of the runner's report and how to present it.

**key** `string` *required*

The raw metric name the runner emits, verbatim — `exact_match,strict-match` for lm-eval. This is
the machine-facing name and it is not translatable.

**label** `string`

What the dashboard column says. Defaults to `key`, which is fine for a machine and poor for a
human.

**direction** `higher | lower | neutral` (default: higher)

Which way is better. The comparison view colours regressions by this.

**ratio** `bool` (default: true)

Whether the value is a 0–1 proportion, so the dashboard renders it as a percentage.

**primary** `bool` (default: false)

The metric cross-run comparison defaults to. Exactly one should be primary; if none is, the first
is used.

> [!TIP]
>
> Declare more than one metric where the difference between them is diagnostic. GSM8K's strict and
> flexible matches are the standard example: a low strict score with a high flexible score means the
> model can do the arithmetic but is not formatting the answer as asked — a prompt problem, not a
> capability problem. One number would hide that.

## The five runners

| Runner | What it runs | Use it for |
| - | - | - |
| `lm-eval` | EleutherAI lm-evaluation-harness | Academic standard benchmarks. The default choice |
| `evalscope` | ModelScope evalscope | Chinese benchmarks — C-Eval, CMMLU — which it covers better |
| `rtl` | An RTL/hardware harness pinned in the evalkit image | Verilog and hardware-design suites |
| `judge` | Generation with vLLM, scored by the platform judge | Anything a rule cannot score. Needs a `rubric` |
| `safety` | A refusal suite scored against a rubric | Whether the model declines what it should |

> [!NOTE]
>
> `lm-eval` and `evalscope` decide correctness with a rule — an exact match, a parsed answer. `judge`
> and `safety` decide it with a rubric your team wrote. The platform supplies neither the suite's
> policy nor the rubric's content for the last two: both belong to the deployment, and a built-in
> would be the platform declaring a verdict it has no standing to declare.

## Bilingual display copy

Write the manifest in English and carry translations in an `i18n:` block, at the pack level and per
metric:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
metrics:
  - key: exact_match,strict-match
    label: Exact match (strict)
    i18n:
      zh:
        label: 精确匹配（严格）
    primary: true
```

Only display copy is translatable. The id, the suites and the metric keys are the contract and must
read the same in every language.

## Check it worked

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench ls                                    # your pack, with its runner and metrics
tp bench run my-bench -m run:run-4f2a91        # score a finished run
```

Scores land on the console's Benchmarks page, in the same matrix as every built-in benchmark, so
your pack's numbers are directly comparable with GSM8K and MMLU on the same run.

## Externally scored benchmarks

When the scoring happens somewhere the platform cannot reach — a licensed harness, a physical test
rig — report the numbers back instead of running them here:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench external create -m run:run-4f2a91 --harness my-rig --train-run run-4f2a91
tp bench external submit run-9c3e10 --file scores.json
```

`create` prints a run id, and `submit` reports the score file against it. `--train-run` is what
lets the evaluation gate and the model registry treat these scores as authoritative.

The scores enter the same matrix and are labelled as externally produced. See
[running benchmarks](https://docs.tuneplane.ai/en/guides/benchmarks.md) for the workflow around it.

## What ships built in

`gsm8k`, `math`, `humaneval`, `mmlu`, `ceval`, `ifeval`, plus the RTL packs (`rtl-repo`,
`rtllm-v2`, `verilogeval-v2`, `verilogeval-v2-completion`, `cvdp`).

Those axes are chosen deliberately: math, code, knowledge regression, Chinese, and instruction
following. Post-training most often improves one thing while quietly breaking another, and the point
of running several is to make that visible on one screen.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.