> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# How you ask for GPUs

> What --profile expands into, and the quota and schedule rules that decide when the job runs.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile h200:8
```

`--profile` is the only resource flag. Node count, parallelism and the NCCL environment come
from the server's hardware registry, not from your config.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp status                             # the profiles you may use, and cluster headroom
tp submit my-grpo --profile h200      # the registry's default shape
tp submit my-grpo --profile h200:4    # 4 GPUs
tp submit my-grpo --profile h200:16   # 16 GPUs, two full nodes
```

## Profiles and series

A **series** is one class of accelerator, as your deployment defines it. A **profile** is a
named preset over a series: a default shape, plus process environment and framework overrides
for that shape.

| | Series | Profile |
| - | - | - |
| Example | `h200`, `h100` | `h200` (1×8), `h200-2g` (1×2), `h100` (1×1) |
| Defines | Per-card memory, compute weight, the scheduling pin, the Slurm constraint, how many cards exist | A default node count and GPUs per node, plus env and framework overrides |
| Used for | Quota and capacity accounting | What you write after `--profile` |

Overrides travel with the profile, not with your experiment: NCCL settings, the PyTorch
allocator, all-ones parallelism, a vLLM memory ratio. Switching GPU type needs no experiment
change.

`h200:16` is a total, not a per-node figure. The registry turns it into nodes × GPUs-per-node.

## Splitting a job across two GPU types

Repeat `--profile` with a role prefix to put training and rollout on different cards:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile train=h200:8 --profile rollout=h100:2
```

Roles that share one pool are comma-separated:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-distill --profile actor,rollout=h200:8 --profile teacher=h200:8
```

Every pool needs the same `gpus_per_node`. Multiple pools work on `kubernetes` (one worker group
per pool) and `slurm` (a hetjob, one component per pool, `--constraint` from the series). They
do not work on `node`, which supports one homogeneous pool across at most 8 machines. Slurm refuses a series that has
no constraint configured.

## Quota

An administrator sets quota per user. It is checked at submission and again at dequeue.

| Dimension | Default | Meaning |
| - | - | - |
| `max_concurrent_gpus` | 8 | GPUs running plus reserved in the queue |
| `max_concurrent_jobs` | 4 | Simultaneously active jobs |
| `daily_gpu_hours` | 0 | Cumulative GPU-hours per day. 0 is unlimited |
| `max_storage_gb` | 0 | What your run directories may hold, in GiB. 0 is unlimited |
| `allowed_profiles` | empty | Which profiles you may use. Empty means all of them |
| `priority` | 0 | Queue-ordering weight |

Per-series limits can be set on top of the scalar ones, which is how a mixed cluster counts an
H100 job and an H200 job separately.

Going over quota does not refuse the job. It waits in `QUEUED`. `tp status` and the console's
Usage page show where you stand. Storage is measured rather than reserved, so that gate lags
slightly behind reality while the GPU gate is exact.

## Schedule windows

An administrator can restrict when a series may run — inference during the day, training
overnight. Windows are per weekday and may cross midnight.

When a window is closed:

* New jobs are not dequeued. They are also held back when too little of the window is left.
* Running jobs get a grace period, then pause. **Checkpoints are kept.**
* When the window reopens, paused jobs resume from their checkpoint.

`tp status` and the job page name the window as the reason a job is waiting.

## What gets counted

Every job records `gpu_seconds`: wall time holding cards, times the card count. Queued time is
not counted. Metering starts when the container is up and the job is actually occupying GPUs.

For a job with several pools, the split is by card count, and it is exact rather than
approximate: a job's pools are placed, started and released together.

## Next

[Job states](https://docs.tuneplane.ai/en/concepts/job-lifecycle.md) for what happens between `QUEUED` and `RUNNING`.
[Submitting a job](https://docs.tuneplane.ai/en/guides/submit.md) for the rest of the flags.
[Roles and quotas](https://docs.tuneplane.ai/en/ops/roles-and-quotas.md) if you are the one setting them.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.