> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Submit a training job

> Pick hardware, override hyperparameters, point at data, and chain what happens after training.

The console accepts recipe, source-package and image submissions. Review the resolved Fleet, backend, GPU count, warnings and expiry before confirming. Preview does not queue a job; commit creates it. Expand resolved details when checking configuration or provenance.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile h200:8
```

Validates locally, packs the working tree and uploads it, then shows a preview for confirmation. Committing queues the job and prints its id.
Everything below changes what that command does. The full flag list is on
[`tp submit`](https://docs.tuneplane.ai/en/cli/submit).

## Train from the console

Open **Jobs → Submit job**, or use **Submit job** in a recipe's catalog detail. The default **Recipe and configuration** form needs no JobSpec JSON:

1. Enter the project, then select a training framework, recipe and supported framework version.
2. Set the experiment name, base-model reference and training/validation dataset references and splits.
3. Request GPU series, nodes and GPUs per node. CPU and memory budgets are per node; the scheduler selects a compatible Fleet.
4. Adjust the recipe's declared parameters. Expand optimization, tuning or other groups, or search by parameter name. Empty controls retain recipe defaults. Framework-specific options such as LoRA appear only where the recipe declares them.
5. Resolve the preview, review the chosen Fleet, runtime details, GPU count and warnings, then confirm. Preview alone creates no training job.

Changing framework or recipe clears incompatible inputs. Changing framework version resets parameter overrides. Any edit invalidates the previous preview. A retry after an uncertain confirmation uses the same preview to avoid creating a second job.

The advanced runtime section accepts an optional digest-pinned image override. The registered runtime must exist for the selected framework/version; selecting a form option does not install a training framework. Source-package and image submission modes remain available for custom projects.

## Pick the hardware

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile h200          # the registry's default shape
tp submit my-grpo --profile h200:4        # 4 GPUs
tp submit my-grpo --profile h200:16       # 16 GPUs, two full nodes
```

Split roles across GPU types by repeating the flag. Roles that share a pool are
comma-separated:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile train=h200:8 --profile rollout=h100:2
tp submit my-distill --profile actor,rollout=h200:8 --profile teacher=h200:8
```

Multiple pools need a `kubernetes` or `slurm` Fleet. See
[the resource model](https://docs.tuneplane.ai/en/concepts/resources.md).

## Override a hyperparameter for one run

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile h200:8 \
  -s policy.optimizer.kwargs.lr=1e-6 \
  -s loss_fn.reference_policy_kl_penalty=0.05
```

Repeatable. Types and ranges are checked against the method declaration before anything is
uploaded. This affects the submission only; your experiment files are untouched.

## Point at a model and data

**NeMo-RL**

Model and data come from the config — a base plus a model fragment. Nothing extra is
usually needed at submit time.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile h200:8
```

**verl and TRL**

`--model` and `--train-data` are required.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-verl --profile h200:4 \
  --model Qwen/Qwen3.5-9B \
  --train-data data/train.parquet \
  --validation-data data/val.parquet
```

**A platform dataset**

Reference it as `<owner>/<name>[@version]`. The job pulls it into the shared cache at
startup and gets `<NAME>_DATA_DIR` in its environment, so `--train-data` is a path
*inside* the dataset.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-verl --profile h200:4 \
  --model Qwen/Qwen3.5-9B \
  --train-dataset alice/gsm8k-zh@v2 \
  --train-data train.parquet
```

Prefer declaring it as `data.train.dataset` in the config. The flag is a temporary
override. See [datasets](https://docs.tuneplane.ai/en/guides/datasets.md).

## Mount material the job may read but not download

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-sft --profile h200:8 --volume acme/support-transcripts
```

A [Volume](https://docs.tuneplane.ai/en/guides/volumes.md) arrives read-only at `$VOLUMES_DIR/<name>`. It has no version:
its current contents are its contents. `tp volume ls` lists the ones you can reference.
Repeatable, and better declared as `data.volumes` in the config.

## Publish what the run produces

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-prep --profile h200:1 --output-dataset acme/support-sft@v1
```

The job writes into `$TUNEPLANE_DATASET_OUT_DIR` and the platform publishes it when the run
succeeds. The job never holds a registry credential, and the published dataset inherits the
classification of the material it read.

## Run against an agent environment

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-agent --profile h200:8 --environment acme/web-shop@v3
```

The platform resolves the version into the job runtime and mounts it read-only. `tp env ls`
lists what you can reference. See [agent environments](https://docs.tuneplane.ai/en/guides/agent-envs.md).

## Use a different image

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-custom --profile h200:8 --image registry.example.com/train:v1
```

Required for `custom`, optional elsewhere, where it overrides the framework's configured
default. Resolution order is `--image`, then the console's framework default, then the runtime
registry, then the catalogue. Tags work; pin a digest in production. The registry has to be on
the server's allowlist — see [custom images](https://docs.tuneplane.ai/en/guides/custom-images.md).

## Chain steps after training

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile h200:8 --then export --then eval
```

Runs automatically once training succeeds. Each one becomes a Step of the same Pipeline. See
[pipelines](https://docs.tuneplane.ai/en/guides/pipelines.md).

## Continue from another run

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-dpo --profile h200:8 --init-from run/run-4f2a91/checkpoint@step=500
```

This is how SFT → DPO → GRPO is expressed. The new run records a Derivation back to the old
one, and the lineage shows up on both job pages.

## Move to a newer method version

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile h200:8 --upgrade-recipe
tp submit my-grpo --profile h200:8 --upgrade-recipe --framework-version 0.8.0
```

`--framework-version` only does something together with `--upgrade-recipe`. On its own it does
not rewrite the lock file.

## What gets uploaded

Three directories: `experiments/<exp>/`, `common/` and `configs/`. In a git project, git's own
ignore rules apply, plus a shared exclude list:

| Pattern | Why |
| - | - |
| `.tuneplane-data/**` | A locally run console's database and credentials |
| `.git/**` | The commit is recorded in provenance; the history is not shipped |
| `**/outputs/**` | Local leftovers. Cluster output is `$TUNEPLANE_OUT_DIR` |
| `datasets/**/raw/**`, `datasets/**/data/**` | Raw data. Use a dataset or a hub id instead |
| `**/__pycache__/**` | Bytecode |
| `**/*.key`, `**/secrets.env` | Secrets belong on the server |

> [!WARNING]
>
> A file that looks like a credential **refuses to package**, rather than being skipped
> quietly: `.env`, `.envrc`, `credentials.json`, `token.txt`, `api_key.txt`, `*.pem`, `*.key`,
> `id_rsa*`, `secrets.*`, `hf_token*`, `*_credentials.json`. Move it out of the three packed
> directories. Secrets reach a job through the platform's own injection, never through the
> package.

## Provenance

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile h200:8 --allow-dirty   # submit with uncommitted changes
tp submit my-grpo --profile h200:8 --no-validate   # skip the local validation
```

A dirty working tree is refused by default, so a result traces back to an exact commit.
Untracked files are listed as warnings either way.

## Check it worked

`tp submit` prints a job id and a state of `QUEUED` or `SUBMITTED`. Then:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp job ls                 # your jobs
tp job logs               # follow the latest one
tp job status <JOB_ID>    # one job in detail
```

## When the server refuses or waits

A refusal names the gate. A job that was admitted but is waiting names it too, on the job page
and in `tp status`.

| Gate | What to do |
| - | - |
| `admission` | The profile is not in the hardware registry, or the pool's series disagrees with the profile's. Express the shape as `--profile h200:4`, not by editing the series |
| `userQuota` | Over your concurrent GPU, job or daily GPU-hour limit, or the profile is not in your allowed set. Waiting is normal; the job dequeues on its own |
| `capacity` | Not enough free cards in the series. `capacity.neverFits` means the series does not have that many cards at all |
| `offlineSeries` | No nodes of that series are in the cluster. Use a profile for a series that is online |
| `scheduleWindow` | The series' availability window is closed. The message names when it next opens |
| `storageQuota`, `diskWatermark` | Reclaim runs with `tp clean`, then retry |
| `maintenance` | The cluster is draining. The job starts on its own afterwards |

A stale `recipe.lock.json` is refused before any of these: re-submit with `--upgrade-recipe`.

## Next

[Sweeps](https://docs.tuneplane.ai/en/guides/sweep.md) · [Pipelines](https://docs.tuneplane.ai/en/guides/pipelines.md) ·
[Job states](https://docs.tuneplane.ai/en/concepts/job-lifecycle.md) · [Troubleshooting](https://docs.tuneplane.ai/en/troubleshooting.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.