> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run a benchmark

> Score a finished run against GSM8K, MMLU, C-Eval and the rest, on one comparable dashboard.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp new my-bench --method evalkit/benchmark      # once per project
tp bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu
```

Scores land on the console's Benchmarks page, in the same matrix as every other run, comparable
across runs, methods and base models.

![TunePlane benchmarks page](https://mintcdn.com/starforge/GatXR2rI5-_Vm4_H/images/console/benchmarks.png?fit=max\&auto=format\&n=GatXR2rI5-_Vm4_H\&q=85\&s=4c566bdcc2e13ced6378a9d975671356)

The benchmark matrix: runs down, benchmarks across, best score highlighted.

## What you can score

| `-m` accepts | Example |
| - | - |
| A run | `run:run-4f2a91`, which resolves that run's exported weights |
| A hub model | `Qwen/Qwen2.5-7B` |
| A path on shared storage | `/mnt/shared/models/qwen-support-v3` |

Scoring a run needs it to have been exported. `--then export` at submission saves you
remembering.

## Pick benchmarks

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench ls
tp bench ls --category chinese     # math | code | knowledge | chinese | instruction | hardware | safety
```

Built in: `gsm8k`, `math`, `humaneval`, `mmlu`, `ceval`, `ifeval`, plus the RTL packs. Anything
published as a [benchmark pack](https://docs.tuneplane.ai/en/extend/benchmark-packs.md) appears here too.

Run several across different categories. Post-training reliably improves one axis and quietly
damages another, and that only becomes visible if you measure more than the one you were
optimising.

## Options worth knowing

| Flag | Use it for |
| - | - |
| `--suites` | Comma-separated benchmark names |
| `--runner` | `lm-eval` or `evalscope`, which covers Chinese benchmarks better. The benchmark declares its own default |
| `--limit` | Cap samples per benchmark. Smoke tests only |
| `--repeat` | Run the evaluation N times to measure its own variance. An evaluation gate needs this before it can call a regression |
| `--profile` | Resources, exactly as for training |
| `--extra-args` | Passed straight through to the runner |

> [!WARNING]
>
> A `--limit` score is not comparable with a full one, and nothing downstream marks it as
> truncated. Use it to check the pipeline works, then run without it.

Both runners produce the same score contract, so the same benchmark is comparable whichever ran
it.

## Score every run automatically

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile h200:8 --then eval
```

Running it by hand is fine once. Making it happen on every run is what stops a regression
reaching production. An evaluation gate in the experiment config goes further: a run that falls
below a threshold is marked as failing its gate. See the `eval:` block in your method's
template.

## Report scores from a harness that cannot run here

A licensed tool, or a physical test rig:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench external create -m run:run-4f2a91 --harness cvdp --train-run run-4f2a91
tp bench external submit run-9c3e10 --file scores.json
```

`create` returns a run id, and `submit` reports the score file against it. `--train-run` is what
lets the evaluation gate and the model version treat these scores as authoritative; without it
they are recorded but not.

They enter the same matrix, labelled as externally produced.

## Check it worked

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp job ls
tp job logs
```

Then open the Benchmarks page. A row for your run with a score in each requested column means it
worked. An empty column means that suite failed, and the job log names which.

Scores are ingested idempotently: re-reporting the same run and benchmark updates the row rather
than adding a second one.

## Reading the matrix

Each metric carries a declared direction, so the comparison knows which way is better without
guessing. Where a benchmark reports more than one metric, the gap between them is often the
useful part: GSM8K's strict and flexible matches diverging means the model can do the arithmetic
but is not formatting the answer as asked.

## Next

[Write a benchmark pack](https://docs.tuneplane.ai/en/extend/benchmark-packs.md) to declare your own evaluation set ·
[Rubrics](https://docs.tuneplane.ai/en/guides/rubrics.md) to score what no rule can decide ·
[Comparing runs](https://docs.tuneplane.ai/en/console/review.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.