Troubleshooting

tp status                 # your account, quota and active jobs
tp job status <JOB_ID>    # one job, including the gate holding it
tp job logs               # follow the latest job

Those three answer most of it. The tables below are for when they do not.

Logging in

Symptom	Fix
`tp login` cannot open a browser	Add `--device-flow`. The CLI prints a short code; open the URL it gives you on any device and enter it
In CI	`tp login --token <service token>`
401, or an expired token	Run `tp login` again. Credentials live in `~/.tuneplane/`, per machine. A disabled account also reads as 401, so check the account first
Every `tp` command dies before printing anything	A non-ASCII character in `~/.tuneplane/credentials.json` on a machine whose locale encoding is not UTF-8. Upgrade the CLI

The submission was refused

Message names	Cause	Fix
A recipe contract or `runtime_id` mismatch	Your lock has drifted from the server’s catalogue	`tp recipe status <exp>`, then `tp recipe upgrade <exp>`, or submit with `--upgrade-recipe`
Uncommitted changes	A submission has to trace to an exact commit	`git commit`, or `--allow-dirty` if you accept the loss of provenance
Config validation failed	A misspelled key, an out-of-range value, or batch sizes that do not divide	Fix each one, then confirm with `tp validate <exp>` before resubmitting
A Hugging Face preflight failure	A gated repo you are not approved for, or a misspelled `org/name`	Request access on the hub, then link your account
The image registry is not on the allowlist	`--image`’s host is not in `TUNEPLANE_ALLOWED_IMAGE_REGISTRIES`. An empty list refuses user images entirely	Ask an administrator. See custom images
`custom external observability requires spec.framework.observability_url`	The published `custom/custom` recipe declares `external`	Pass `--observability-url`. See custom training
An unknown profile, or a pool series that disagrees with the profile’s	The profile is not in the hardware registry, or you edited the series instead of the shape	Write the shape as `--profile h200:4`
A packaging refusal naming a file	The packer found something that looks like a credential	Move it out of `experiments/`, `common/` and `configs/`

The job will not start

State	Check, in this order
`QUEUED` and staying there	The gate is named on the job page and in `tp status`. Quota first, then free cards in the series, then whether the schedule window is closed, then maintenance mode
`QUEUED` on a series with no nodes	`offlineSeries`: no machines of that series are in the cluster. Use a profile for one that is online
`PENDING` for minutes	A slow image pull, no node matching the selector, or not enough cards. The job’s Config and Logs tabs carry the backend’s own events. Past the pre-running deadline the job is marked failed
`FAILED` immediately, with a scaffold message	`train.sh` was never edited. The scaffold exits 1 and says so

QUEUED never means “on the cluster waiting for a GPU”. That is PENDING. See job states.

It is running but the charts are empty

Check in this order:

The Logs tab. Weights may still be downloading. A 9B model on a cold cache takes minutes.
Whether anything reports. A catalogue method wires metrics for you. A custom job has to call tuneplane.report — see custom training.
Whether ingest is reachable. TUNEPLANE_JOB_ENDPOINT inside the container must be an address the GPU node can reach, which a deployment configures as TUNEPLANE_INGEST_URL. If it points at 127.0.0.1 on the console host, nothing arrives and the logs look fine.

Other training problems

Symptom	What it is
A new job appears after a `FAILED` one	Automatic retry, within a budget and after a cooldown. The count is on the job page. Stop the job if you do not want another attempt
OOM, or the container was killed	Lower `train_micro_batch_size`, turn on activation checkpointing, lower vLLM’s `gpu_memory_utilization`, or move to a larger profile. The Diagnosis tab usually names the change
The job reports running but no step advances	A hung collective, a shared mount that went away, or a process that ran out of memory without exiting. Start with Logs, then System
Validation samples are empty	Normal early on. Samples appear at a validation round, so `val_period` has to be above zero and a round has to have been reached
The reward is flat at exactly one value	Suspect the verifier before the model. A verifier returning a constant looks identical to a task nothing can do. See verifiers
`train/token_mult_prob_error` above 1–2% (NeMo-RL)	Weight sync between the trainer and the rollout engine is broken. A reward curve that looks fine cannot be trusted at that point
Checkpoints vanished	They were written outside `$TUNEPLANE_OUT_DIR`

Logs

Symptom	Fix
`tp job logs` keeps dropping	The CLI reconnects and resumes on its own. Persistent drops are usually a reverse proxy buffering `/api` — nginx needs buffering off
You want the whole history	`tp job logs <JOB_ID> -n 0`

Still stuck

Open the job in the console and run an AI diagnosis from the Diagnosis tab. It reads the log tail, the metric trends, the validation samples and the config snapshot, and proposes changes including a resubmit plan. If that does not answer it, hand the run id to an administrator: the server-side logs carry what the job page cannot show.

FAQ · Job states · Errors

Get started

Core concepts

Train a model

Evaluate and serve

Agent environments

Extend the platform

Collaborate

Integrations

Reference

Help

Logging in

The submission was refused

The job will not start

It is running but the charts are empty

Other training problems

Logs

Still stuck

Next

​Logging in

​The submission was refused

​The job will not start

​It is running but the charts are empty

​Other training problems

​Logs

​Still stuck

​Next

Logging in

The submission was refused

The job will not start

It is running but the charts are empty

Other training problems

Logs

Still stuck

Next