> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# tp bench

> Run standard benchmarks against a run or any model, and report externally produced scores.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench ls                                       # what can be evaluated here
tp bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu
```

`<EXP>` is an evaluation experiment, created once with `tp new my-bench --method evalkit/benchmark`
and reused. The workflow around these commands is in [Run a benchmark](https://docs.tuneplane.ai/en/guides/benchmarks.md).

## What `tp bench ls` prints

Built-in benchmarks ship with the platform; published ones arrive through `tp recipe sync`. Each row
says which runner it uses, its primary metric, and its category.

```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
math
  gsm8k      built-in  lm-eval    exact match (strict)  GSM8K · grade-school word problems
  math       built-in  lm-eval    exact match           MATH · competition mathematics
code
  humaneval  built-in  lm-eval    pass@1                HumanEval · code generation
```

## The runner is usually not yours to pick

> [!NOTE]
>
> Which runner, how many few-shot examples, and the sample cap are declared by the benchmark itself —
> so you do not have to remember that GSM8K wants 5-shot or that HumanEval needs
> `--confirm_run_unsafe_code`. `--runner` is only for a raw suite name the platform has no pack for.
>
> If the benchmarks you select span two runners — say `gsm8k` on lm-eval and `ceval` on evalscope —
> the submission is refused and tells you to split it in two. One evaluation job is one process, and
> quietly splitting it would be worse than saying so.

## Examples

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp new my-bench --method evalkit/benchmark              # once per project
tp bench run my-bench -m run:run-4f2a91 --suites gsm8k,mmlu
tp bench run my-bench -m Qwen/Qwen2.5-7B --suites ceval
tp bench run my-bench -m run:run-4f2a91 --suites gsm8k --limit 20   # smoke test only
```

## `tp bench`

Run standard benchmarks

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench <subcommand>
```

| Command | Description |
| - | - |
| `tp bench external` | Report scores from an external harness |
| `tp bench ls` | List the available benchmarks |
| `tp bench new` | Scaffold a safety benchmark pack |
| `tp bench run` | Submit a benchmark evaluation job |

### `tp bench external`

Report scores from an external harness

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench external <subcommand>
```

| Command | Description |
| - | - |
| `tp bench external create` | Open an external evaluation |
| `tp bench external submit` | Report a score file |

#### `tp bench external create`

Open an external evaluation

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench external create [options]
```

| Option | Description |
| - | - |
| `--model`, `-m` | Reference of the model under evaluation (deployment id / model version / HF id) (**required**) |
| `--harness` | Which harness is used, e.g. cvdp |
| `--train-run` | Which training run's artifact is evaluated; only with this do the gate and the model version accept these scores |
| `--note` | Note |

#### `tp bench external submit`

Report a score file

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench external submit <RUN_ID> [options]
```

| Argument | Description |
| - | - |
| `RUN_ID` | The run\_id that external create returned (**required**) |

| Option | Description |
| - | - |
| `--file`, `-f` | Score JSON file (**required**) |
| `--model`, `-m` | Model reference (used to group the variance) |
| `--harness` | Harness name |

### `tp bench ls`

List the available benchmarks

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench ls [options]
```

| Option | Description |
| - | - |
| `--category` | One category only: math\|code\|knowledge\|chinese\|instruction\|hardware\|safety |

### `tp bench new`

Scaffold a safety benchmark pack

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench new <NAME> [options]
```

| Argument | Description |
| - | - |
| `NAME` | Pack name, e.g. refusal-policy (**required**) |

| Option | Description |
| - | - |
| `--owner` | Namespace segment for the local catalog; defaults to your OS user. Not an identity claim -- user packs are synced down from the platform, not pushed up from here |
| `--rubric` | \<owner>/\<name> of the rubric that decides what counts as a refusal |
| `--title` | Display title; defaults to the name |

### `tp bench run`

Submit a benchmark evaluation job

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp bench run <EXP> [options]
```

| Argument | Description |
| - | - |
| `EXP` | The evaluation experiment (create it with tp new \<name> --method evalkit/benchmark) (**required**) |

| Option | Description |
| - | - |
| `--model`, `-m` | HF model id / absolute path on shared storage / run:\<run\_id> (takes that run's hf\_export) (**required**) |
| `--suites` | Comma-separated benchmarks (e.g. gsm8k,mmlu). Run `tp bench ls` for the options (**required**) |
| `--runner` | Evaluation backend: lm-eval \| evalscope. Declared by the benchmark itself by default, rarely needed |
| `--limit` `<int>` | Sample cap per benchmark (for smoke runs) |
| `--extra-args` | Extra arguments passed straight to the runner |
| `--profile` | Target hardware and resources in one argument: h200 (the registry default shape), h200:4 (4 GPUs), h200:16 (2 full nodes). Repeat it to split pools per role (the heterogeneous extension point): --profile train=h200:8 --profile rollout=h100:2. Comma-separate roles that share a pool: --profile actor,rollout=h200:8 --profile teacher=h200:8 (repeatable) |
| `--allow-dirty` | Allow uncommitted changes in the working tree |
| `--repeat` `<int range>` | Repeat the evaluation this many times to measure the evaluation's own variance (the gate needs it to call a regression) (default `1`) |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.