1.10.0, as its own runtime rather than routed through
custom. Methods: trl/sft, dpo, grpo, grpo-judge, rm, kto, rloo, opsd.
Observability is platform. TRL’s own metric names — loss, eval_loss, learning_rate —
are aliased onto the console’s series, so the charts fill in without you renaming anything.
Create
train.py in the experiment directory. Hyperparameters belong in
config.yaml or --set; edit train.py only when you actually need a fork of the loop.
The integration adds tensorboard to report_to and sets the granted logging directory.
An isolated collector reads native metrics; explicitly named trackers such as wandb remain. Validation
samples come from the callback attached in the template’s train.py: keep it when you fork the
loop, or the samples tab stays empty while curves still arrive. DPO and GRPO use
TunePlaneValidationCallback. SFT and KTO use a recipe-owned CompletionSamplesCallback based on
TRL’s official distributed completion callback. It generates at most eight prompts with 128
new tokens at evaluation; edit that policy in the recipe. Set
TUNEPLANE_JOB_VAL_UPLOAD_SAMPLES=0 to disable sample collection and its extra generation.
Public callbacks enqueue bounded local records; the separate collector owns network delivery.
Sample generation and extraction still cost evaluation time. Local queues and disk budgets are
bounded, loss is reported, and shutdown drains only until its deadline.
Submit with a model and data
--model and --train-data are required unless the config already fills the bound fields.
--validation-data usually is too.
--profile, by way of
TUNEPLANE_CLUSTER_GPUS_PER_NODE.
Distillation needs a same-vocabulary teacher
teacher_model is required and the template’s train.py checks it. The teacher has to share
the student’s vocabulary.
beta here is the generalised JSD interpolation, not GRPO’s KL-penalty coefficient: 0.0 is
forward KL and the on-policy distillation paper’s setting, 1.0 is reverse KL, 0.5 is
standard JSD.
use_liger_kernel computes JSD with the fused kernel for less memory and more throughput. It
is refused for models with logit_scale or final_logit_softcapping, such as Gemma.
Score against a rubric instead of a rule
trl/grpo-judge takes its reward from the platform judge scoring each completion against a
versioned rubric. Use it when no rule decides correctness — writing
quality, tone, domain conventions, whether an explanation explains. The same rubric can score a
benchmark, so training and evaluation cannot drift apart.
Check it worked
tp job logs shows Accelerate starting the processes, then the first optimizer step. The
Charts tab fills with train/loss, validation/loss, train/token_accuracy,
train/grad_norm and train/lr.
After training
Checkpoints follow the Hugging Face layout —checkpoint-* and final_model.
--then export --then eval works on
submit. Evaluation runs the experiment’s own eval.py.