> ## Documentation Index > Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt > Use this file to discover all available pages before exploring further. # Troubleshooting > Symptom, cause and fix for the ways a submission, a queue wait or a training run goes wrong. ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp status # your account, quota and active jobs tp job status # one job, including the gate holding it tp job logs # follow the latest job ``` Those three answer most of it. The tables below are for when they do not. ## Logging in | Symptom | Fix | | - | - | | `tp login` cannot open a browser | Add `--device-flow`. The CLI prints a short code; open the URL it gives you on any device and enter it | | In CI | `tp login --token ` | | 401, or an expired token | Run `tp login` again. Credentials live in `~/.tuneplane/`, per machine. A disabled account also reads as 401, so check the account first | | Every `tp` command dies before printing anything | A non-ASCII character in `~/.tuneplane/credentials.json` on a machine whose locale encoding is not UTF-8. Upgrade the CLI | ## The submission was refused | Message names | Cause | Fix | | - | - | - | | A recipe contract or `runtime_id` mismatch | Your lock has drifted from the server's catalogue | `tp recipe status `, then `tp recipe upgrade `, or submit with `--upgrade-recipe` | | Uncommitted changes | A submission has to trace to an exact commit | `git commit`, or `--allow-dirty` if you accept the loss of provenance | | Config validation failed | A misspelled key, an out-of-range value, or batch sizes that do not divide | Fix each one, then confirm with `tp validate ` before resubmitting | | A Hugging Face preflight failure | A gated repo you are not approved for, or a misspelled `org/name` | Request access on the hub, then [link your account](https://docs.tuneplane.ai/en/integrations/huggingface.md) | | The image registry is not on the allowlist | `--image`'s host is not in `TUNEPLANE_ALLOWED_IMAGE_REGISTRIES`. An empty list refuses user images entirely | Ask an administrator. See [custom images](https://docs.tuneplane.ai/en/guides/custom-images.md) | | `custom external observability requires spec.framework.observability_url` | The published `custom/custom` recipe declares `external` | Pass `--observability-url`. See [custom training](https://docs.tuneplane.ai/en/guides/custom-training.md) | | An unknown profile, or a pool series that disagrees with the profile's | The profile is not in the hardware registry, or you edited the series instead of the shape | Write the shape as `--profile h200:4` | | A packaging refusal naming a file | The packer found something that looks like a credential | Move it out of `experiments/`, `common/` and `configs/` | ## The job will not start | State | Check, in this order | | - | - | | `QUEUED` and staying there | The gate is named on the job page and in `tp status`. Quota first, then free cards in the series, then whether the schedule window is closed, then maintenance mode | | `QUEUED` on a series with no nodes | `offlineSeries`: no machines of that series are in the cluster. Use a profile for one that is online | | `PENDING` for minutes | A slow image pull, no node matching the selector, or not enough cards. The job's Config and Logs tabs carry the backend's own events. Past the pre-running deadline the job is marked failed | | `FAILED` immediately, with a scaffold message | `train.sh` was never edited. The scaffold exits 1 and says so | `QUEUED` never means "on the cluster waiting for a GPU". That is `PENDING`. See [job states](https://docs.tuneplane.ai/en/concepts/job-lifecycle.md). ## It is running but the charts are empty Check in this order: 1. **The Logs tab.** Weights may still be downloading. A 9B model on a cold cache takes minutes. 2. **Whether anything reports.** A catalogue method wires metrics for you. A custom job has to call `tuneplane.report` — see [custom training](https://docs.tuneplane.ai/en/guides/custom-training.md). 3. **Whether ingest is reachable.** `TUNEPLANE_JOB_ENDPOINT` inside the container must be an address the GPU node can reach, which a deployment configures as `TUNEPLANE_INGEST_URL`. If it points at `127.0.0.1` on the console host, nothing arrives and the logs look fine. ## Other training problems | Symptom | What it is | | - | - | | A new job appears after a `FAILED` one | Automatic retry, within a budget and after a cooldown. The count is on the job page. Stop the job if you do not want another attempt | | OOM, or the container was killed | Lower `train_micro_batch_size`, turn on activation checkpointing, lower vLLM's `gpu_memory_utilization`, or move to a larger profile. The Diagnosis tab usually names the change | | The job reports running but no step advances | A hung collective, a shared mount that went away, or a process that ran out of memory without exiting. Start with Logs, then System | | Validation samples are empty | Normal early on. Samples appear at a validation round, so `val_period` has to be above zero and a round has to have been reached | | The reward is flat at exactly one value | Suspect the verifier before the model. A verifier returning a constant looks identical to a task nothing can do. See [verifiers](https://docs.tuneplane.ai/en/extend/verifiers.md) | | `train/token_mult_prob_error` above 1–2% (NeMo-RL) | Weight sync between the trainer and the rollout engine is broken. A reward curve that looks fine cannot be trusted at that point | | Checkpoints vanished | They were written outside `$TUNEPLANE_OUT_DIR` | ## Logs | Symptom | Fix | | - | - | | `tp job logs` keeps dropping | The CLI reconnects and resumes on its own. Persistent drops are usually a reverse proxy buffering `/api` — nginx needs buffering off | | You want the whole history | `tp job logs -n 0` | ## Still stuck Open the job in the console and run an AI diagnosis from the **Diagnosis** tab. It reads the log tail, the metric trends, the validation samples and the config snapshot, and proposes changes including a resubmit plan. If that does not answer it, hand the run id to an administrator: the server-side logs carry what the job page cannot show. ## Next [FAQ](https://docs.tuneplane.ai/en/faq.md) · [Job states](https://docs.tuneplane.ai/en/concepts/job-lifecycle.md) · [Errors](https://docs.tuneplane.ai/en/api-reference/errors.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.