Train with NeMo-RL

tp new my-grpo --method nemo-rl/grpo
tp validate my-grpo
tp submit my-grpo --profile h200:8

NeMo-RL is the default catalogue for a full post-training stack, at 0.7.0. Methods: nemo-rl/sft, dpo, grpo, grpo-lora, ppo, rm, distillation, maxrl, opsd. Two things you do not have to do: pass --observability-url, and call tuneplane.report. This framework’s observability is platform: the integration enables native TensorBoard output (logger.tensorboard_enabled). An isolated collector reads the event files without modifying NeMo-RL logger classes. Explicitly configured external trackers remain available.

Create and tune

tp new my-grpo --method nemo-rl/grpo
tp methods nemo-rl/grpo          # every key you can set, with ranges
tp validate my-grpo

Model and datasets come from configs/base/ and a configs/models/ fragment, so a submission usually needs no --model or --train-data. See config.yaml for how the layering resolves.

Submit

git add -A && git commit -m "grpo lr 2e-6"
tp submit my-grpo --profile h200:8

The adapter overwrites four keys from the environment the server injects. Do not fight them in your config:

Key	Comes from
`cluster.num_nodes`	`TUNEPLANE_CLUSTER_NUM_NODES`
`cluster.gpus_per_node`	`TUNEPLANE_CLUSTER_GPUS_PER_NODE`
`checkpointing.checkpoint_dir`	`TUNEPLANE_OUT_DIR`
`logger.log_dir`	`TUNEPLANE_OUT_DIR`

Put rollout on different cards

tp submit my-grpo --profile train=h200:8 --profile rollout=h100:2

With a separate rollout pool the adapter sets policy.generation.colocated.enabled=false and fills the colocated resource counts from TUNEPLANE_POOL_TOPOLOGY. You do not maintain a non-colocated overlay YAML for this. Multiple pools need a kubernetes or slurm Fleet. Node Fleets support one homogeneous pool across at most 8 machines.

Bring your own run.py

The recipe’s entrypoint is the versioned NeMo-RL file. If the recipe declares experiment_override and that file exists in your experiment directory, yours wins — which is how an experiment with a custom dataset or a tool environment ships its own run.py. spec.source.entrypoint in the JobSpec is refused. You cannot override the recipe from submission JSON. The container needs NEMO_RL_DIR, which the deployment sets as TUNEPLANE_NEMO_RL_DIR and is typically /opt/nemo-rl in the NGC image. Without it the submission fails naming the missing configuration.

Check it worked

tp job logs shows Ray starting and then the first training step. The Charts tab fills with train/reward, validation/accuracy, train/entropy, train/kl_penalty and train/token_mult_prob_error. Watch that last one. Above one or two percent, weight sync between the trainer and the rollout engine is broken, and a reward curve that looks fine cannot be trusted.

After training

tp export my-grpo --checkpoint checkpoints/step_500 --checkpoint-format nemo-dcp
tp eval my-grpo --model /shared/hf_export/my-model --eval-config configs/eval/gsm8k.yaml
tp bench run bench-my-grpo -m run:run-4f2a91 --suites gsm8k,mmlu

Formats here are nemo-dcp and nemo-megatron. --then export --then eval on submit runs the first two automatically once the job succeeds.

Pipelines · Benchmarks · Agent environments for GRPO and PPO against tool environments

Get started

Core concepts

Train a model

Evaluate and serve

Agent environments

Extend the platform

Collaborate

Integrations

Reference

Help

Train with NeMo-RL

Create and tune

Submit

Put rollout on different cards

Bring your own run.py

Check it worked

After training

Next

​Create and tune

​Submit

​Put rollout on different cards

​Bring your own run.py

​Check it worked

​After training

​Next

Create and tune

Submit

Put rollout on different cards

Bring your own run.py

Check it worked

After training

Next