> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a training image

> What goes in the image, what must stay out, the registry allowlist, and tags against digests.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
docker build -t registry.example.com/my-train:v1 .
docker push registry.example.com/my-train:v1
tp submit my-custom --profile h200:8 \
  --image registry.example.com/my-train:v1 \
  --observability-url https://wandb.example/my-proj
```

A training image holds CUDA, Python, and whatever your trainer imports. Nothing else.

> [!WARNING]
>
> Do not copy the platform's source tree into the image, and do not install `tuneplane-server`.
> The runtime is injected at launch as a content-addressed PEX, so the image needs nothing from
> the platform for logs, lifecycle or artifacts to work.
>
> The one exception is `import tuneplane` in your training code under the published
> `custom/custom` recipe, which declares `external` observability and therefore does not patch
> `PYTHONPATH`. For that, install the `tuneplane` wheel. See
> [custom training](https://docs.tuneplane.ai/en/guides/custom-training.md).

## What belongs in it

| In the image | Not in the image |
| - | - |
| CUDA and cuDNN matching your cluster's drivers | `tuneplane-server`, or the platform source tree |
| Python 3.10 to 3.13 — the runner requires `>=3.10,<3.14` | A baked-in `HF_TOKEN`, or any credential as a layer |
| Your trainer's dependencies: transformers, deepspeed, vLLM | Datasets. Pull them at runtime, or use a platform dataset |
| The `tuneplane` wheel, if your code calls `tuneplane.report` | A baked-in console URL |

Logs need nothing: `print`, `logging` and framework stdout all reach the Logs tab. Charts are
not parsed from stdout. Use `tuneplane.report` or declare TensorBoard/JSONL outputs for
[native collection](https://docs.tuneplane.ai/en/ops/telemetry.md); the independent collector needs no Python SDK in the training image.

## A Dockerfile to start from

```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# syntax=docker/dockerfile:1
ARG BASE_IMAGE=nvidia/cuda:12.8.0-devel-ubuntu24.04
FROM ${BASE_IMAGE}

ARG PYTHON_VERSION=3.12
ENV DEBIAN_FRONTEND=noninteractive \
    HF_HUB_ENABLE_HF_TRANSFER=1

RUN apt-get update && apt-get install -y --no-install-recommends \
        git curl ca-certificates \
    && rm -rf /var/lib/apt/lists/*

COPY --from=ghcr.io/astral-sh/uv:0.8 /uv /usr/local/bin/uv

RUN uv python install ${PYTHON_VERSION} \
 && uv venv --python ${PYTHON_VERSION} /opt/train-venv
ENV VIRTUAL_ENV=/opt/train-venv PATH=/opt/train-venv/bin:$PATH

# Pin these in production.
RUN uv pip install --no-cache \
        torch transformers accelerate trl hf-transfer tuneplane

CMD ["python", "-c", "import torch; print(torch.__version__)"]
```

Two things about that venv. The Job Capsule uses the image's own Python, so keep the venv first
on `PATH` — otherwise `python` inside `train.sh` is `/usr/bin/python` with no torch. And use a
`devel` base if you need nvcc, which DeepSpeed's JIT compile does.

## Push, then pin

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
docker push registry.example.com/my-train:v1
docker buildx imagetools inspect registry.example.com/my-train:v1   # read the digest
```

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-custom --profile h200:8 \
  --image registry.example.com/my-train@sha256:abcd… \
  --observability-url https://wandb.example/my-proj
```

Tags work. A digest is what stops a moving tag from changing the job after admission.

## The registry allowlist

`--image` is parsed for its registry host: `ghcr.io`, `localhost:5000`, or `docker.io` when the
name carries no host. That host has to be in `TUNEPLANE_ALLOWED_IMAGE_REGISTRIES`, a
comma-separated list the deployment sets.

| Allowlist | A custom `--image` | Overriding a catalogue framework's image |
| - | - | - |
| Empty | **Refused**: "the allowlist is empty, which forbids user images entirely" | Allowed — an empty list is no limit here |
| Set, host missing | Refused, printing the allowed hosts | Refused the same way |
| Host present | Allowed | Allowed |

An administrator sets something like
`TUNEPLANE_ALLOWED_IMAGE_REGISTRIES=registry.example.com,ghcr.io`, and the nodes have to be able
to pull: `imagePullSecrets` on KubeRay, a Docker login on each registered machine, or an
already-converted SIF on Slurm.

## What each backend wants

| Fleet kind | Image form |
| - | - |
| `node` | OCI. Registered machines need the same pull credentials |
| `kubernetes` | OCI plus a pull secret on the RayJob. A slow pull looks like `PENDING` |
| `slurm` | Usually a SIF or SQSH in the runtime registry. A raw Docker tag is often not enough; an administrator converts it and registers the `runtime_id` |

A slow pull leaves the job in `PENDING` until the pre-running deadline, then `FAILED`. That is
not a training bug.

## Check it worked

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
docker run --rm registry.example.com/my-train:v1 \
  python -c "import torch, tuneplane.report; print(torch.__version__)"
```

Then submit. `tp job logs` reaching your own launch line means the image pulled and the venv is
on `PATH`.

## Next

[Custom training](https://docs.tuneplane.ai/en/guides/custom-training.md) · [Job environment](https://docs.tuneplane.ai/en/reference/job-env.md) ·
[Fleets](https://docs.tuneplane.ai/en/ops/fleets.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.