Skip to main content
The state the server keeps is the one the CLI and the console show. A Kubernetes pod phase or a Slurm PENDING is mapped onto the list below. Nothing is reported as succeeded because it could not be observed. A job's states: QUEUED, SUBMITTED, PENDING, RUNNING, the three terminal states, and PAUSED — with which of them hold GPUs, occupy a job slot, and count GPU-hours. A job's states: QUEUED, SUBMITTED, PENDING, RUNNING, the three terminal states, and PAUSED — with which of them hold GPUs, occupy a job slot, and count GPU-hours.

​
What each state means

QUEUED does not mean “on Slurm, waiting for a GPU”. That is PENDING. When a job sits in QUEUED, check quota and the schedule window before you check kubectl get pods — the job page names the gate that is holding it.

​
Pause, resume and retry

Resume is not a reattach to the old container. The state goes back to QUEUED and the scheduler submits a fresh launch, so quota, capacity and the schedule window all apply again. Retry keeps the same run id, so it writes to the same output directory and the framework continues from its latest checkpoint. One row, one run id, retry count incremented. Automatic retry only fires for a job that reached RUNNING. A job that died before that usually died of its configuration, and a retry would reproduce it. There is a retry budget and a cooldown, both set by the deployment; the count is on the job page. Stop the job if you do not want another attempt. When the retry button is greyed out, its tooltip says why. Usually the working-directory snapshot is gone: it is the only copy of the code as submitted, it is cleared when the job row is deleted, and after that the job has to be submitted again.

​
A Pipeline rolls up to one word

A submission with --then export --then eval is one Pipeline of several Steps. The job list shows the Pipeline’s state, not each Step’s. partial is the one worth knowing: training worked and the export or evaluation did not.

​
What GPU time is counted

Queued time is free. Metering starts when the container is up and the job holds cards. The figure is gpu_seconds — wall time × card count — and it appears on the console’s Usage page.

​
Next

Resources for the quota and window rules that hold a job in QUEUED. Pipelines for chaining steps. Troubleshooting for a job that will not start.