Build a training image

docker build -t registry.example.com/my-train:v1 .
docker push registry.example.com/my-train:v1
tp submit my-custom --profile h200:8 \
  --image registry.example.com/my-train:v1 \
  --observability-url https://wandb.example/my-proj

A training image holds CUDA, Python, and whatever your trainer imports. Nothing else.

Do not copy the platform’s source tree into the image, and do not install tuneplane-server. The runtime is injected at launch as a content-addressed PEX, so the image needs nothing from the platform for logs, lifecycle or artifacts to work.The one exception is import tuneplane in your training code under the published custom/custom recipe, which declares external observability and therefore does not patch PYTHONPATH. For that, install the tuneplane wheel. See custom training.

What belongs in it

In the image	Not in the image
CUDA and cuDNN matching your cluster’s drivers	`tuneplane-server`, or the platform source tree
Python 3.10 to 3.13 — the runner requires `>=3.10,<3.14`	A baked-in `HF_TOKEN`, or any credential as a layer
Your trainer’s dependencies: transformers, deepspeed, vLLM	Datasets. Pull them at runtime, or use a platform dataset
The `tuneplane` wheel, if your code calls `tuneplane.report`	A baked-in console URL

Logs need nothing: print, logging and framework stdout all reach the Logs tab. Charts are not parsed from stdout. Use tuneplane.report or declare TensorBoard/JSONL outputs for native collection; the independent collector needs no Python SDK in the training image.

A Dockerfile to start from

# syntax=docker/dockerfile:1
ARG BASE_IMAGE=nvidia/cuda:12.8.0-devel-ubuntu24.04
FROM ${BASE_IMAGE}

ARG PYTHON_VERSION=3.12
ENV DEBIAN_FRONTEND=noninteractive \
    HF_HUB_ENABLE_HF_TRANSFER=1

RUN apt-get update && apt-get install -y --no-install-recommends \
        git curl ca-certificates \
    && rm -rf /var/lib/apt/lists/*

COPY --from=ghcr.io/astral-sh/uv:0.8 /uv /usr/local/bin/uv

RUN uv python install ${PYTHON_VERSION} \
 && uv venv --python ${PYTHON_VERSION} /opt/train-venv
ENV VIRTUAL_ENV=/opt/train-venv PATH=/opt/train-venv/bin:$PATH

# Pin these in production.
RUN uv pip install --no-cache \
        torch transformers accelerate trl hf-transfer tuneplane

CMD ["python", "-c", "import torch; print(torch.__version__)"]

Two things about that venv. The Job Capsule uses the image’s own Python, so keep the venv first on PATH — otherwise python inside train.sh is /usr/bin/python with no torch. And use a devel base if you need nvcc, which DeepSpeed’s JIT compile does.

Push, then pin

docker push registry.example.com/my-train:v1
docker buildx imagetools inspect registry.example.com/my-train:v1   # read the digest

tp submit my-custom --profile h200:8 \
  --image registry.example.com/my-train@sha256:abcd… \
  --observability-url https://wandb.example/my-proj

Tags work. A digest is what stops a moving tag from changing the job after admission.

The registry allowlist

--image is parsed for its registry host: ghcr.io, localhost:5000, or docker.io when the name carries no host. That host has to be in TUNEPLANE_ALLOWED_IMAGE_REGISTRIES, a comma-separated list the deployment sets.

Allowlist	A custom `--image`	Overriding a catalogue framework’s image
Empty	Refused: “the allowlist is empty, which forbids user images entirely”	Allowed — an empty list is no limit here
Set, host missing	Refused, printing the allowed hosts	Refused the same way
Host present	Allowed	Allowed

An administrator sets something like TUNEPLANE_ALLOWED_IMAGE_REGISTRIES=registry.example.com,ghcr.io, and the nodes have to be able to pull: imagePullSecrets on KubeRay, a Docker login on each registered machine, or an already-converted SIF on Slurm.

What each backend wants

Fleet kind	Image form
`node`	OCI. Registered machines need the same pull credentials
`kubernetes`	OCI plus a pull secret on the RayJob. A slow pull looks like `PENDING`
`slurm`	Usually a SIF or SQSH in the runtime registry. A raw Docker tag is often not enough; an administrator converts it and registers the `runtime_id`

A slow pull leaves the job in PENDING until the pre-running deadline, then FAILED. That is not a training bug.

Check it worked

docker run --rm registry.example.com/my-train:v1 \
  python -c "import torch, tuneplane.report; print(torch.__version__)"

Then submit. tp job logs reaching your own launch line means the image pulled and the venv is on PATH.

Custom training · Job environment · Fleets

Get started

Core concepts

Train a model

Evaluate and serve

Agent environments

Extend the platform

Collaborate

Integrations

Reference

Help

Build a training image

What belongs in it

A Dockerfile to start from

Push, then pin

The registry allowlist

What each backend wants

Check it worked

Next

​What belongs in it

​A Dockerfile to start from

​Push, then pin

​The registry allowlist

​What each backend wants

​Check it worked

​Next

What belongs in it

A Dockerfile to start from

Push, then pin

The registry allowlist

What each backend wants

Check it worked

Next