> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Bring your own trainer

> Run your own train.sh on the cluster: the environment contract, reporting metrics, and where checkpoints have to go.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp new my-custom --method custom/custom
# write experiments/my-custom/train.sh
tp submit my-custom --profile h200:8 \
  --image registry.example.com/my-trainer:v1 \
  --observability-url https://wandb.example/my-proj
```

`custom/custom` runs exactly one file: `experiments/<name>/train.sh`. The platform does not read
a `FRAMEWORK` variable, does not look for a `run.py`, and never falls back to custom because
another adapter failed. What that file does is yours.

> [!WARNING]
>
> Do not go custom to change a learning rate. The catalogue methods already wire metrics,
> checkpoints and images, and custom hands all three back to you.
>
> For a supervised PyTorch loop over labelled data,
> [`pytorch/image-classification`](https://docs.tuneplane.ai/en/guides/pytorch) gives you the same freedom inside
> `train.py` plus multi-node torchrun, checked hyperparameters and a platform image.

## Scaffold

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp new my-custom --method custom/custom
```

You get `train.sh`, plus `config.yaml`, `recipe.lock.json` and `README.md`. The recipe's
entrypoint is `train.sh` by name, so renaming it fails submission with `custom entrypoint is
missing or escapes the experiment directory`.

The adapter runs `bash <absolute path to train.sh>`. The working directory is the **job package
root**, not the experiment folder, so point Python at `"${TUNEPLANE_EXP_DIR}/train.py"` rather
than a relative path.

The custom adapter only compiles the training operation. `tp export` and `tp eval` against a
custom experiment are refused.

## The environment contract

The scaffold dies with `:?` if any of these is missing. Do not rename them.

| Variable | What it is |
| - | - |
| `TUNEPLANE_WORK_DIR` | The unpacked job package root |
| `TUNEPLANE_EXP_DIR` | This experiment's directory |
| `TUNEPLANE_OUT_DIR` | The output directory, already created and isolated per run. The control plane resolves it; never compose it yourself |
| `TUNEPLANE_FRAMEWORK`, `TUNEPLANE_RECIPE` | Both `custom` |
| `TUNEPLANE_CLUSTER_NUM_NODES` | Nodes you actually have |
| `TUNEPLANE_CLUSTER_GPUS_PER_NODE` | GPUs per node you actually have |
| `TUNEPLANE_JOB_ENDPOINT`, `_RUN_ID`, `_TOKEN` | Ingest credentials. Absent outside a job |
| `TUNEPLANE_JOB_ENABLED` | `1` when ingest is bound, else `0` |

`HF_HOME`, `HF_TOKEN` and the `*_DATA_DIR` variables arrive too, when the deployment has them.
The full list is in the [job environment reference](https://docs.tuneplane.ai/en/reference/job-env.md).

Quota and the watchdog are measured against `TUNEPLANE_CLUSTER_*`. Occupying more GPUs than
that gets the job warned and can get it stopped, so pass the same numbers into
`accelerate launch --num_processes` or `torchrun --nproc_per_node`.

## Three things your script must do

1. Write everything you want to keep under `$TUNEPLANE_OUT_DIR`. The scratch tree disappears
   when the container exits.
2. Report scalars with `tuneplane.report`, or declare native TensorBoard/JSONL outputs
   for the [independent collector](https://docs.tuneplane.ai/en/ops/telemetry.md). Stdout alone does not produce metric points.
3. Respect `TUNEPLANE_CLUSTER_*`.

The platform looks for these paths under the output directory. They are realpath-checked and
must stay inside it:

| Path | For |
| - | - |
| `checkpoints/*` | Weights |
| `logs` | Text logs you write yourself. Optional — stdout already streams |
| `hf_export` | A HuggingFace export directory |
| `eval/report.json` | An evaluation report you write by hand |

## Write train.sh

`set -euo pipefail` is already there. Replace the scaffold's `exit 1` with a launch.

```bash One process theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
exec python "${TUNEPLANE_EXP_DIR}/train.py" \
  --output-dir "${TUNEPLANE_OUT_DIR}" \
  --config "${TUNEPLANE_EXP_DIR}/config.yaml"
```

```bash Accelerate, one process per GPU theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
exec accelerate launch \
  --num_processes "${TUNEPLANE_CLUSTER_GPUS_PER_NODE}" \
  "${TUNEPLANE_EXP_DIR}/train.py" \
  --output_dir "${TUNEPLANE_OUT_DIR}" \
  --config "${TUNEPLANE_EXP_DIR}/config.yaml"
```

```bash torchrun theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
exec torchrun \
  --nproc_per_node "${TUNEPLANE_CLUSTER_GPUS_PER_NODE}" \
  "${TUNEPLANE_EXP_DIR}/train.py" \
  --output-dir "${TUNEPLANE_OUT_DIR}"
```

Do not `cd` somewhere and write `./checkpoints`. Use the variables.

## Report metrics

The PyPI name is `tuneplane` and the import is `tuneplane`. The reporting module pulls in
neither transformers nor Ray, and it never raises into your training loop. With no
`TUNEPLANE_JOB_TOKEN` it does nothing, which is what you want outside a job.

```python A hand-written loop theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
from tuneplane.report import init, log, finish

init(hparams={"lr": 1e-5, "batch_size": 8})

for step, batch in enumerate(loader):
    loss = train_one(batch)
    log({"train/loss": float(loss)}, step=step)

finish()
```

```python A HuggingFace trainer theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
from tuneplane.report import TunePlaneCallback
from trl import SFTTrainer

trainer = SFTTrainer(..., callbacks=[TunePlaneCallback()])
trainer.train()
```

```python From a reward function theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
from tuneplane.report import log

log({"env/tool_success_rate": 0.83, "env/avg_turns": 3.2}, step=step)
```

`init()` is idempotent. Nested dicts are flattened, and a non-scalar that cannot be reduced to a
mean is dropped. `prefix=` prepends a namespace when the key does not already carry one.

`TunePlaneCallback` is duck-typed rather than a `TrainerCallback` subclass. It calls `init` on
train begin, `log` on `on_log`, `log(..., prefix="validation")` on evaluate, and `finish` at the
end.

Hardware sampling starts with `init(monitor_hardware=True)`, on rank zero only — one sampler per
job, not one per GPU. The interval is `TUNEPLANE_JOB_MONITOR_INTERVAL`, in seconds, default
`10`.

Two things not to do: do not POST to the ingest API yourself, because stdout is already
forwarded and a second path duplicates lines; and do not hard-code the console URL, because the
container already has `TUNEPLANE_JOB_ENDPOINT`.

## Why --observability-url is required

The published `custom/custom` recipe declares `observability: external`. Two consequences:

* Submission **requires** `--observability-url`. Without it, compile fails with `custom external
  observability requires spec.framework.observability_url`. Any URL your team uses works; the
  platform does not start wandb for you. It reaches the job as
  `TUNEPLANE_EXTERNAL_OBSERVABILITY_URL`.
* The adapter does **not** rewrite `PYTHONPATH`. `import tuneplane` works only if the wheel is
  in your image, or you put it on `PYTHONPATH` yourself.

A recipe declaring `observability: platform` behaves the other way round: `--observability-url`
is refused, and the runner prepends the capsule roots to `PYTHONPATH` so
`from tuneplane.report import log` works with no wheel in the image. You cannot flip that from
your experiment directory — it is a property of the published recipe.

For SDK-based charts with this recipe, install `tuneplane` in your image and still pass
`--observability-url`. Native collection is a separate output declaration and deployment path.

`TUNEPLANE_JOB_ENABLED=1` is separate. That is the ingest binding the server always sets when
the job has an ingest token.

## Submit

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
git add -A && git commit -m "custom trainer"
tp submit my-custom --profile h200:8 \
  --image registry.example.com/my-trainer:v1 \
  --observability-url https://wandb.example/my-proj
```

`--image` is the image: custom has no catalogue pin, since its `runtime.default_version` is
`user-managed`. Tags work; pin `@sha256:…` in production so a moving tag cannot change the job
after admission.

An empty `TUNEPLANE_ALLOWED_IMAGE_REGISTRIES` **refuses** custom user images, while catalogue
frameworks can still use the deployment's defaults. Ask an administrator to add your registry
hostname. See [custom images](https://docs.tuneplane.ai/en/guides/custom-images.md).

## Check it worked

`tp job logs` shows your own launch line, then your trainer's output. The Charts tab gets its
first point after your first `log()` call.

Inside the container:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
python -c "import tuneplane.report; print('ok')"
echo "$TUNEPLANE_OUT_DIR" "$TUNEPLANE_CLUSTER_GPUS_PER_NODE"
env | grep TUNEPLANE_JOB
```

## When it does not work

| Symptom | Cause |
| - | - |
| Fails in seconds, log says `no training command yet` | The scaffold `train.sh` was never edited |
| Logs flow, charts stay empty | No `log()` or callback, or `import tuneplane` failed, or weights are still downloading |
| `ModuleNotFoundError: tuneplane` | No wheel in the image, and the recipe is `external` so `PYTHONPATH` was not patched |
| Imports work locally, not in the job | `train.sh` execs a different Python than you tested. Align `PATH` with the image's venv |
| Charts and logs both empty | The container never started. Check the state and the Events tab |
| Checkpoints vanished | Written to the working directory or `/tmp`. Use `$TUNEPLANE_OUT_DIR/checkpoints` |
| An allowlist error | The registry host of `--image` is not in `TUNEPLANE_ALLOWED_IMAGE_REGISTRIES` |
| `custom adapter` refuses `export` | Custom has no export or eval compile path |

## config.yaml is yours

Nothing in the custom adapter reads it. The custom recipe declares no parameters, so
`tp validate` has nothing to check and `--set` values land in `spec.hyperparams` without
reaching your argv. Parse the file in `train.py`.

`--model` and `--train-data` are the verl and TRL bindings. Custom ignores them: use the
environment and the files you packed.

## Next

[Custom images](https://docs.tuneplane.ai/en/guides/custom-images.md) · [Job environment](https://docs.tuneplane.ai/en/reference/job-env.md) ·
[Reporting API](https://docs.tuneplane.ai/en/api-reference/python-sdk.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.