> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run jobs on Slurm

> A slurm Fleet: an allocation through slurmrestd, then srun with Ray.

For sites that already run Slurm. The control plane talks to **slurmrestd** (REST + JWT) only — no sbatch or SSH side channel. Training runs inside Enroot/Pyxis or Apptainer.

## Prerequisites

1. Slurm 26.05 with slurmrestd API `v0.0.45` and JWT authentication;
2. Container runtime installed on compute nodes: Enroot + Pyxis (recommended) or Apptainer;
3. A shared filesystem (for job staging directories and artifacts);
4. First-class framework execution artifacts published to the runtime registry as **SQSH / SIF** (custom + slurm does not allow implicit OCI image materialization).

## Configuration

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
TUNEPLANE_SLURM_REST_URL=https://slurmrestd.internal:6820
TUNEPLANE_SLURM_REST_API_VERSION=v0.0.45
TUNEPLANE_SLURM_REST_USER=tuneplane
TUNEPLANE_SLURM_JWT_FILE=/run/secrets/slurm-jwt
TUNEPLANE_SLURM_PARTITION=gpu
TUNEPLANE_SLURM_RUNTIME_PROFILE=enroot-pyxis   # or apptainer
TUNEPLANE_STORAGE_ROOT=/lustre/tuneplane        # the one storage root, on the parallel filesystem
# TUNEPLANE_SLURM_STORAGE_ROOT=                 # only if the cluster mounts it at another path
TUNEPLANE_SLURM_TLS_VERIFY=1
```

## Key semantics

> **Batch script rendering**
>
> LaunchRequest renders a batch script with explicit image, mounts and topology. Each attempt keeps its own immutable environment and scheduler logs. A lost submission response retains those files and blocks another submit until the original allocation is found. Lifecycle operations use the native SLUID; numeric JobIDs are used only to retrieve the initial receipt. Automatic Slurm requeue is disabled because TunePlane owns retries.

> **Heterogeneous multi-pool = hetjob**
>
> Multi-pool jobs are rendered as Slurm **heterogeneous jobs**: one component per pool, with `--constraint` pinning nodes via the GPU series' `slurm_constraint`. If a series declares no constraint, the assembly layer rejects outright (fail-closed — components never land on arbitrary GPU nodes).

> **Status and logs**
>
> Status is polled via REST and mapped onto the platform state machine; logs are read from the shared directory with a 64 KiB limit per file and at most 2,000 lines. Epilog and staging flags keep resources occupied until cleanup finishes.

## What this backend cannot do

**No Playground.** A compute node usually has no inbound path from the console, so there is
nowhere to reach a session. Same for a Model Deployment.

**No Volume mount.** The platform does not control the bind list of the SIF or SQSH image, so it
refuses a `spec.data.volumes` reference here rather than starting a job whose mount would be
empty.

**The [smoke test](https://docs.tuneplane.ai/en/ops/smoke-test.md) does not cover it.** That would need a containerised
slurmrestd plus Apptainer. Batch script rendering, the REST client, hetjob layout and status
mapping are covered by unit tests instead, so what stays unproven here is the round trip against
a real controller.

## CPU-only custom jobs

Use `custom/custom` with one pool, `nodes: 1`, `gpus_per_node: 0`, and positive
`cpus` and `memory_gb`. Select a CPU-capable Slurm partition. The generated batch
script requests `--cpus-per-task` and `--mem`, with no GPU allocation or Ray.

Publish a SIF/SQSH artifact under an explicit `custom-...` runtime id in
`TUNEPLANE_RUNTIME_REGISTRY_FILE`. Set `spec.framework.runtime_id` to that id and
omit `spec.framework.image`. Artifact hashes are verified before submission and
on the compute node. The shared storage and container runtime prerequisites
still apply. Scratch and multi-machine CPU requests are not supported.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.