Skip to main content
--profile is the only resource flag. Node count, parallelism and the NCCL environment come from the server’s hardware registry, not from your config.

​
Profiles and series

A series is one class of accelerator, as your deployment defines it. A profile is a named preset over a series: a default shape, plus process environment and framework overrides for that shape. Overrides travel with the profile, not with your experiment: NCCL settings, the PyTorch allocator, all-ones parallelism, a vLLM memory ratio. Switching GPU type needs no experiment change. h200:16 is a total, not a per-node figure. The registry turns it into nodes × GPUs-per-node.

​
Splitting a job across two GPU types

Repeat --profile with a role prefix to put training and rollout on different cards:
Roles that share one pool are comma-separated:
Every pool needs the same gpus_per_node. Multiple pools work on kubernetes (one worker group per pool) and slurm (a hetjob, one component per pool, --constraint from the series). They do not work on node, which supports one homogeneous pool across at most 8 machines. Slurm refuses a series that has no constraint configured.

​
Quota

An administrator sets quota per user. It is checked at submission and again at dequeue. Per-series limits can be set on top of the scalar ones, which is how a mixed cluster counts an H100 job and an H200 job separately. Going over quota does not refuse the job. It waits in QUEUED. tp status and the console’s Usage page show where you stand. Storage is measured rather than reserved, so that gate lags slightly behind reality while the GPU gate is exact.

​
Schedule windows

An administrator can restrict when a series may run — inference during the day, training overnight. Windows are per weekday and may cross midnight. When a window is closed:
  • New jobs are not dequeued. They are also held back when too little of the window is left.
  • Running jobs get a grace period, then pause. Checkpoints are kept.
  • When the window reopens, paused jobs resume from their checkpoint.
tp status and the job page name the window as the reason a job is waiting.

​
What gets counted

Every job records gpu_seconds: wall time holding cards, times the card count. Queued time is not counted. Metering starts when the container is up and the job is actually occupying GPUs. For a job with several pools, the split is by card count, and it is exact rather than approximate: a job’s pools are placed, started and released together.

​
Next

Job states for what happens between QUEUED and RUNNING. Submitting a job for the rest of the flags. Roles and quotas if you are the one setting them.