> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Operational troubleshooting

> The failures that look like something other than what they are.

Ordered by how often the symptom points somewhere other than the cause.

## Jobs queue forever with free GPUs on the dashboard

| Check | Because |
| - | - |
| The user's quota | Concurrent GPUs, concurrent jobs, or daily GPU-hours. A queued job was admitted; a refused one was not |
| Scheduling windows | `TUNEPLANE_SCHEDULE_ENFORCE` with a closed window for that series |
| Disk watermark | `TUNEPLANE_DISK_WATERMARK_PCT` stops dequeuing entirely to protect running jobs |
| Starvation | A large job repeatedly denied while small ones behind it take the freed capacity. `TUNEPLANE_SCHED_RESERVE_AFTER_S` is what stops this; if it is 0 the queue starves large jobs indefinitely |

The starvation case is the one that looks most like a bug. The console shows "waiting for capacity"
forever, and the platform itself generates the stream that causes it — lifecycle jobs and Playground
sessions each take one GPU, so a four-GPU job is skipped every pass.

## Training starts but is absurdly slow, or hangs in all-reduce

`TUNEPLANE_K8S_NCCL_SOCKET_IFNAME` is unset on a multi-NIC machine, so NCCL picked the wrong interface.
There is no error — only bad throughput or a hang. Common values are `bond0`, `eth0`, `ib0`.

## The cluster comes up but workers never register

`TUNEPLANE_KUBERAY_RAY_VERSION` does not match the Ray inside the training image. The only clue is one
version-mismatch warning line, easily lost in startup output.

## Jobs OOM on anything but a tiny batch

`/dev/shm` is at the container default of 64MB. Ray's object store lives there.
`TUNEPLANE_K8S_SHM_SIZE` or `TUNEPLANE_NODE_SHM_SIZE`, and remember it is RAM-backed — it plus the memory
limit must stay under physical memory.

## A multi-node run cannot be resumed

`TUNEPLANE_K8S_STORAGE_SHARED` is true but the PVC is RWO. An RWO claim does not fail the job — it gives
each pod its own volume, so checkpoint shards split across nodes. The executor's health check
verifies the declaration against the live PVC; read it.

## Every background task is stopped

No Redis. Background roles need a shared lock so several replicas do not run the same task at once.
Single instance: acceptable. Multiple replicas: storage accounting, diagnosis and the daily report
are not happening.

## Weight downloads stall, or fail on a 400 with only a URL in the body

The mirror, or Xet. See [air-gapped deployments](https://docs.tuneplane.ai/en/ops/airgapped.md).

## Logs stream but charts stay empty

For SDK reporting, check `TUNEPLANE_INGEST_URL` from the worker and verify that
[`tuneplane.report`](https://docs.tuneplane.ai/en/api-reference/python-sdk) is called. For native outputs, check
source declarations, file ownership, collector startup, gateway trust and credentials,
and the selected attempt/source in Charts. Follow [native telemetry troubleshooting](https://docs.tuneplane.ai/en/ops/telemetry.md).

## Everyone is logged out after a restart

`TUNEPLANE_WEB_JWT_SECRET` is unset, so each process generates its own. Set a fixed strong random value.
With more than one replica this also means tokens signed by one are rejected by the others.

## GPUs are permanently "unschedulable"

Health checks found pending ECC retirements, uncorrected errors, or hardware throttling — or a
process outside the platform is holding the card. The capacity view names which. These faults raise
no error anywhere; they produce bad results or a silent slowdown, which is why the check exists.

## Storage quotas report "not measured"

`TUNEPLANE_STORAGE_SCAN_INTERVAL_S` is 0, or the scan role is not running. A full scan takes minutes on a
large tree, which is why it is a background role rather than computed on request.

## An environment is refused at admission

Either it needs a sandbox and none is configured, or it is `openenv` (`deployment: container`) — which this deployment
does not host, deliberately. Refusing at admission rather than in the launcher means you find out
before an allocation is spent.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.