> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Job states

> What QUEUED, PENDING and PAUSED mean, how pause and retry work, and when GPU time is counted.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp job status <JOB_ID>    # one job
tp job ls                 # your jobs, newest first
tp job logs               # follow the latest job
```

The state the server keeps is the one the CLI and the console show. A Kubernetes pod phase or a
Slurm `PENDING` is mapped onto the list below. Nothing is reported as succeeded because it
could not be observed.

![A job's states: QUEUED, SUBMITTED, PENDING, RUNNING, the three terminal states, and PAUSED — with which of them hold GPUs, occupy a job slot, and count GPU-hours.](https://mintcdn.com/starforge/WYu6FCkjTqnjH1Kc/images/job-states.svg?fit=max\&auto=format\&n=WYu6FCkjTqnjH1Kc\&q=85\&s=ef8ae91186a6b00dcc656f65e362c69d)

## What each state means

| State | On the cluster | Why it is there |
| - | - | - |
| `QUEUED` | No | Admitted. Waiting for quota, cards, a schedule window, or a relaunch after resume |
| `SUBMITTED` | Launch in flight | Quota is already reserved. The backend has not returned a handle |
| `PENDING` | Yes, not training | Image pull, node selector, Slurm allocation, Ray cluster coming up |
| `RUNNING` | Yes | The entrypoint is executing |
| `SUCCEEDED` | Terminal | Exit 0. Any `--then` step may still start |
| `FAILED` | Terminal, unless retried | Non-zero exit, OOM, an ingest hard failure, a deadline before `RUNNING` |
| `STOPPED` | Terminal, unless retried | You or an administrator stopped it |
| `PAUSED` | No | The process is stopped and the checkpoint is kept |

`QUEUED` does not mean "on Slurm, waiting for a GPU". That is `PENDING`. When a job sits in
`QUEUED`, check quota and the schedule window before you check `kubectl get pods` — the job
page names the gate that is holding it.

## Pause, resume and retry

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp job pause <JOB_ID>     # stop the process, keep the checkpoint
tp job resume <JOB_ID>    # back to QUEUED, then relaunched
tp job retry <JOB_ID>     # requeue a FAILED or STOPPED job
```

Resume is not a reattach to the old container. The state goes back to `QUEUED` and the
scheduler submits a fresh launch, so quota, capacity and the schedule window all apply again.

Retry keeps the **same run id**, so it writes to the same output directory and the framework
continues from its latest checkpoint. One row, one run id, retry count incremented.

Automatic retry only fires for a job that reached `RUNNING`. A job that died before that
usually died of its configuration, and a retry would reproduce it. There is a retry budget and
a cooldown, both set by the deployment; the count is on the job page. Stop the job if you do
not want another attempt.

When the retry button is greyed out, its tooltip says why. Usually the working-directory
snapshot is gone: it is the only copy of the code as submitted, it is cleared when the job row
is deleted, and after that the job has to be submitted again.

## A Pipeline rolls up to one word

A submission with `--then export --then eval` is one **Pipeline** of several **Steps**. The job
list shows the Pipeline's state, not each Step's.

| Pipeline state | Meaning |
| - | - |
| `queued`, `running` | At least one Step is waiting or live |
| `succeeded` | Every declared Step finished |
| `partial` | The Root Step succeeded and a Step declared after it did not happen |
| `failed`, `stopped`, `paused` | The Root Step's own outcome, which beats its Steps' |

`partial` is the one worth knowing: training worked and the export or evaluation did not.

## What GPU time is counted

Queued time is free. Metering starts when the container is up and the job holds cards. The
figure is `gpu_seconds` — wall time × card count — and it appears on the console's Usage page.

## Next

[Resources](https://docs.tuneplane.ai/en/concepts/resources.md) for the quota and window rules that hold a job in `QUEUED`.
[Pipelines](https://docs.tuneplane.ai/en/guides/pipelines.md) for chaining steps.
[Troubleshooting](https://docs.tuneplane.ai/en/troubleshooting.md) for a job that will not start.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.