> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Train with TRL

> The TRL trainers as platform methods: required model and data, the OPSD teacher, and where train.py fits.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp new my-sft --method trl/sft
tp submit my-sft --profile h200:4 \
  --model Qwen/Qwen2.5-7B --train-data data/train.parquet
```

Hugging Face TRL at catalogue version `1.10.0`, as its own runtime rather than routed through
`custom`. Methods: `trl/sft`, `dpo`, `grpo`, `grpo-judge`, `rm`, `kto`, `rloo`, `opsd`.

Observability is `platform`. TRL's own metric names — `loss`, `eval_loss`, `learning_rate` —
are aliased onto the console's series, so the charts fill in without you renaming anything.

## Create

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp new my-sft  --method trl/sft
tp new my-kto  --method trl/kto
tp new my-opsd --method trl/opsd
```

The entrypoint is `train.py` in the experiment directory. Hyperparameters belong in
`config.yaml` or `--set`; edit `train.py` only when you actually need a fork of the loop.

The integration adds `tensorboard` to `report_to` and sets the granted logging directory.
An isolated collector reads native metrics; explicitly named trackers such as `wandb` remain. Validation
samples come from the callback attached in the template's `train.py`: keep it when you fork the
loop, or the samples tab stays empty while curves still arrive. DPO and GRPO use
`TunePlaneValidationCallback`. SFT and KTO use a recipe-owned `CompletionSamplesCallback` based on
TRL's official distributed completion callback. It generates at most eight prompts with 128
new tokens at evaluation; edit that policy in the recipe. Set
`TUNEPLANE_JOB_VAL_UPLOAD_SAMPLES=0` to disable sample collection and its extra generation.

Public callbacks enqueue bounded local records; the separate collector owns network delivery.
Sample generation and extraction still cost evaluation time. Local queues and disk budgets are
bounded, loss is reported, and shutdown drains only until its deadline.

## Submit with a model and data

`--model` and `--train-data` are required unless the config already fills the bound fields.
`--validation-data` usually is too.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-sft --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-data data/train.parquet \
  --validation-data data/val.parquet
```

For a gated Hub repository, link your Hugging Face account on the console first so submission
preflight can check access. See [the Hugging Face integration](https://docs.tuneplane.ai/en/integrations/huggingface.md).

The adapter launches through Accelerate. GPU count still comes from `--profile`, by way of
`TUNEPLANE_CLUSTER_GPUS_PER_NODE`.

## Distillation needs a same-vocabulary teacher

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-opsd --profile h200:8 \
  --model Qwen/Qwen3.5-4B \
  --train-data data/train.parquet \
  -s teacher_model=Qwen/Qwen3.5-9B \
  -s teacher_model_revision=main \
  -s beta=0.0
```

`teacher_model` is required and the template's `train.py` checks it. The teacher has to share
the student's vocabulary.

`beta` here is the generalised JSD interpolation, not GRPO's KL-penalty coefficient: `0.0` is
forward KL and the on-policy distillation paper's setting, `1.0` is reverse KL, `0.5` is
standard JSD.

`use_liger_kernel` computes JSD with the fused kernel for less memory and more throughput. It
is refused for models with `logit_scale` or `final_logit_softcapping`, such as Gemma.

## Score against a rubric instead of a rule

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp new my-judge --method trl/grpo-judge
```

`trl/grpo-judge` takes its reward from the platform judge scoring each completion against a
versioned [rubric](https://docs.tuneplane.ai/en/guides/rubrics.md). Use it when no rule decides correctness — writing
quality, tone, domain conventions, whether an explanation explains. The same rubric can score a
benchmark, so training and evaluation cannot drift apart.

## Check it worked

`tp job logs` shows Accelerate starting the processes, then the first optimizer step. The
Charts tab fills with `train/loss`, `validation/loss`, `train/token_accuracy`,
`train/grad_norm` and `train/lr`.

## After training

Checkpoints follow the Hugging Face layout — `checkpoint-*` and `final_model`.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp export my-sft --checkpoint final_model --checkpoint-format huggingface
```

Export and evaluation are declared on the recipe, so `--then export --then eval` works on
submit. Evaluation runs the experiment's own `eval.py`.

## Next

[Method catalogue](https://docs.tuneplane.ai/en/guides/methods.md) · [Rubrics](https://docs.tuneplane.ai/en/guides/rubrics.md) ·
[Pipelines](https://docs.tuneplane.ai/en/guides/pipelines.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.