Submit a training job

The console accepts recipe, source-package and image submissions. Review the resolved Fleet, backend, GPU count, warnings and expiry before confirming. Preview does not queue a job; commit creates it. Expand resolved details when checking configuration or provenance.

tp submit my-grpo --profile h200:8

Validates locally, packs the working tree and uploads it, then shows a preview for confirmation. Committing queues the job and prints its id. Everything below changes what that command does. The full flag list is on tp submit.

Train from the console

Open Jobs → Submit job, or use Submit job in a recipe’s catalog detail. The default Recipe and configuration form needs no JobSpec JSON:

Enter the project, then select a training framework, recipe and supported framework version.
Set the experiment name, base-model reference and training/validation dataset references and splits.
Request GPU series, nodes and GPUs per node. CPU and memory budgets are per node; the scheduler selects a compatible Fleet.
Adjust the recipe’s declared parameters. Expand optimization, tuning or other groups, or search by parameter name. Empty controls retain recipe defaults. Framework-specific options such as LoRA appear only where the recipe declares them.
Resolve the preview, review the chosen Fleet, runtime details, GPU count and warnings, then confirm. Preview alone creates no training job.

Changing framework or recipe clears incompatible inputs. Changing framework version resets parameter overrides. Any edit invalidates the previous preview. A retry after an uncertain confirmation uses the same preview to avoid creating a second job. The advanced runtime section accepts an optional digest-pinned image override. The registered runtime must exist for the selected framework/version; selecting a form option does not install a training framework. Source-package and image submission modes remain available for custom projects.

Pick the hardware

tp submit my-grpo --profile h200          # the registry's default shape
tp submit my-grpo --profile h200:4        # 4 GPUs
tp submit my-grpo --profile h200:16       # 16 GPUs, two full nodes

Split roles across GPU types by repeating the flag. Roles that share a pool are comma-separated:

tp submit my-grpo --profile train=h200:8 --profile rollout=h100:2
tp submit my-distill --profile actor,rollout=h200:8 --profile teacher=h200:8

Multiple pools need a kubernetes or slurm Fleet. See the resource model.

Override a hyperparameter for one run

tp submit my-grpo --profile h200:8 \
  -s policy.optimizer.kwargs.lr=1e-6 \
  -s loss_fn.reference_policy_kl_penalty=0.05

Repeatable. Types and ranges are checked against the method declaration before anything is uploaded. This affects the submission only; your experiment files are untouched.

Point at a model and data

NeMo-RL
verl and TRL
A platform dataset

Model and data come from the config — a base plus a model fragment. Nothing extra is usually needed at submit time.

tp submit my-grpo --profile h200:8

--model and --train-data are required.

tp submit my-verl --profile h200:4 \
  --model Qwen/Qwen3.5-9B \
  --train-data data/train.parquet \
  --validation-data data/val.parquet

Reference it as <owner>/<name>[@version]. The job pulls it into the shared cache at startup and gets <NAME>_DATA_DIR in its environment, so --train-data is a path inside the dataset.

tp submit my-verl --profile h200:4 \
  --model Qwen/Qwen3.5-9B \
  --train-dataset alice/gsm8k-zh@v2 \
  --train-data train.parquet

Prefer declaring it as data.train.dataset in the config. The flag is a temporary override. See datasets.

Mount material the job may read but not download

tp submit my-sft --profile h200:8 --volume acme/support-transcripts

A Volume arrives read-only at $VOLUMES_DIR/<name>. It has no version: its current contents are its contents. tp volume ls lists the ones you can reference. Repeatable, and better declared as data.volumes in the config.

Publish what the run produces

tp submit my-prep --profile h200:1 --output-dataset acme/support-sft@v1

The job writes into $TUNEPLANE_DATASET_OUT_DIR and the platform publishes it when the run succeeds. The job never holds a registry credential, and the published dataset inherits the classification of the material it read.

Run against an agent environment

tp submit my-agent --profile h200:8 --environment acme/web-shop@v3

The platform resolves the version into the job runtime and mounts it read-only. tp env ls lists what you can reference. See agent environments.

Use a different image

tp submit my-custom --profile h200:8 --image registry.example.com/train:v1

Required for custom, optional elsewhere, where it overrides the framework’s configured default. Resolution order is --image, then the console’s framework default, then the runtime registry, then the catalogue. Tags work; pin a digest in production. The registry has to be on the server’s allowlist — see custom images.

Chain steps after training

tp submit my-grpo --profile h200:8 --then export --then eval

Runs automatically once training succeeds. Each one becomes a Step of the same Pipeline. See pipelines.

Continue from another run

tp submit my-dpo --profile h200:8 --init-from run/run-4f2a91/checkpoint@step=500

This is how SFT → DPO → GRPO is expressed. The new run records a Derivation back to the old one, and the lineage shows up on both job pages.

Move to a newer method version

tp submit my-grpo --profile h200:8 --upgrade-recipe
tp submit my-grpo --profile h200:8 --upgrade-recipe --framework-version 0.8.0

--framework-version only does something together with --upgrade-recipe. On its own it does not rewrite the lock file.

What gets uploaded

Three directories: experiments/<exp>/, common/ and configs/. In a git project, git’s own ignore rules apply, plus a shared exclude list:

Pattern	Why
`.tuneplane-data/**`	A locally run console’s database and credentials
`.git/**`	The commit is recorded in provenance; the history is not shipped
`/outputs/`	Local leftovers. Cluster output is `$TUNEPLANE_OUT_DIR`
`datasets//raw/`, `datasets//data/`	Raw data. Use a dataset or a hub id instead
`/__pycache__/`	Bytecode
`*/.key`, `**/secrets.env`	Secrets belong on the server

A file that looks like a credential refuses to package, rather than being skipped quietly: .env, .envrc, credentials.json, token.txt, api_key.txt, *.pem, *.key, id_rsa*, secrets.*, hf_token*, *_credentials.json. Move it out of the three packed directories. Secrets reach a job through the platform’s own injection, never through the package.

Provenance

tp submit my-grpo --profile h200:8 --allow-dirty   # submit with uncommitted changes
tp submit my-grpo --profile h200:8 --no-validate   # skip the local validation

A dirty working tree is refused by default, so a result traces back to an exact commit. Untracked files are listed as warnings either way.

Check it worked

tp submit prints a job id and a state of QUEUED or SUBMITTED. Then:

tp job ls                 # your jobs
tp job logs               # follow the latest one
tp job status <JOB_ID>    # one job in detail

When the server refuses or waits

A refusal names the gate. A job that was admitted but is waiting names it too, on the job page and in tp status.

Gate	What to do
`admission`	The profile is not in the hardware registry, or the pool’s series disagrees with the profile’s. Express the shape as `--profile h200:4`, not by editing the series
`userQuota`	Over your concurrent GPU, job or daily GPU-hour limit, or the profile is not in your allowed set. Waiting is normal; the job dequeues on its own
`capacity`	Not enough free cards in the series. `capacity.neverFits` means the series does not have that many cards at all
`offlineSeries`	No nodes of that series are in the cluster. Use a profile for a series that is online
`scheduleWindow`	The series’ availability window is closed. The message names when it next opens
`storageQuota`, `diskWatermark`	Reclaim runs with `tp clean`, then retry
`maintenance`	The cluster is draining. The job starts on its own afterwards

A stale recipe.lock.json is refused before any of these: re-submit with --upgrade-recipe.

Sweeps · Pipelines · Job states · Troubleshooting

Get started

Core concepts

Train a model

Evaluate and serve

Agent environments

Extend the platform

Collaborate

Integrations

Reference

Help

Submit a training job

Train from the console

Pick the hardware

Override a hyperparameter for one run

Point at a model and data

Mount material the job may read but not download

Publish what the run produces

Run against an agent environment

Use a different image

Chain steps after training

Continue from another run

Move to a newer method version

What gets uploaded

Provenance

Check it worked

When the server refuses or waits

Next

​Train from the console

​Pick the hardware

​Override a hyperparameter for one run

​Point at a model and data

​Mount material the job may read but not download

​Publish what the run produces

​Run against an agent environment

​Use a different image

​Chain steps after training

​Continue from another run

​Move to a newer method version

​What gets uploaded

​Provenance

​Check it worked

​When the server refuses or waits

​Next

Train from the console

Pick the hardware

Override a hyperparameter for one run

Point at a model and data

Mount material the job may read but not download

Publish what the run produces

Run against an agent environment

Use a different image

Chain steps after training

Continue from another run

Move to a newer method version

What gets uploaded

Provenance

Check it worked

When the server refuses or waits

Next