Train with verl

tp new my-verl --method verl/grpo --framework-version 0.9.0
tp submit my-verl --profile h200:8 \
  --model Qwen/Qwen2.5-7B --train-data data/train.parquet

ByteDance verl at catalogue versions 0.9.0 (the default) and 0.9.1. Methods: verl/grpo, dapo, ppo, rloo, reinforce-baseline, remax, distillation, sft. A new upstream version is opt-in before it is the default: pass --framework-version 0.9.1 to try it, and see Adopt a new upstream version for why. Unlike NeMo-RL, verl needs --model and --train-data at submission. Observability is platform: the integration adds tensorboard to trainer.logger and declares its output directory. An isolated collector reads those files; other explicitly configured trackers remain. No Tracking constructor or import hook is patched.

verl images are deployment artifacts. An administrator sets TUNEPLANE_IMAGE_VERL or registers the runtime id before anybody can submit. Without one, submission fails naming the missing runtime rather than the method.

Create with an explicit version

tp new my-verl --method verl/grpo --framework-version 0.9.0

0.9 uses the unified verl.trainer.main_ppo. Upstream removed 0.8’s main_ppo_sync, so the catalogue no longer carries the older runtime. If experiments/my-verl/main.py exists, that file becomes the entrypoint. That is the verl-recipe style: subclass TaskRunner or RayPPOTrainer. A reward-only change does not need it — set custom_reward_function.path in the config instead.

Model and data are required

The recipe binds three flags onto Hydra keys:

Flag	Hydra key
`--model`	`actor_rollout_ref.model.path`
`--train-data`	`data.train_files`
`--validation-data`	`data.val_files`

tp submit my-verl --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-data data/train.parquet \
  --validation-data data/val.parquet

With a platform dataset, --train-data is a path inside it:

tp submit my-verl --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-dataset alice/gsm8k-zh@v2 \
  --train-data train.parquet

Prefer data.train.dataset in the config; the flag is a temporary override. See datasets. tp validate fails early when a required declaration is missing.

The method is the estimator

Every RL method here runs the same verl.trainer.main_ppo. The recipe pins algorithm.adv_estimator, and that is what the method is.

Method	Estimator	Baseline from	Notes
`verl/grpo`	`grpo`	in-group mean ÷ in-group std	needs `rollout.n ≥ 2`
`verl/rloo`	`rloo`	the mean of the other samples	unbiased, no std division
`verl/reinforce-baseline`	`reinforce_plus_plus_baseline`	group mean, then global-batch whitening	same estimator as `openrlhf/reinforce-baseline`
`verl/remax`	`remax`	the model’s own greedy answer	`rollout.n = 1`, plus one greedy pass per step
`verl/ppo`	`gae`	a trained critic	a second full-size model; watch `train/value_loss`
`verl/dapo`	`grpo`	in-group mean on filtered groups	the four DAPO techniques, on by default

Writing algorithm.adv_estimator into config.yaml is refused at submission. To change estimator, change method. Loss variants are knobs, not methods: set policy_loss_mode on any of these recipes for GSPO, CISPO, GMPO, clip-cov, kl-cov or GPG.

DAPO

verl/dapo is configuration, not a vendored trainer. 0.9 carries all four published techniques in mainline main_ppo, so the recipe pins algorithm.filter_groups.enable and the dapo reward manager, and ships the paper’s values as defaults: clip 0.2/0.28, token-mean aggregation, a 4096-token overlong buffer at factor 1.0, 16 samples per prompt, and no KL on either path. Two things to know before submitting one:

Dynamic sampling judges a group on filter_metric during rollout, so the reward has to exist then. Use a rule-based reward function, or a reward model in its own pool. A colocated reward model scores after sampling, and upstream asserts on that.
Step time varies by construction: the trainer keeps regenerating until enough groups qualify.

For the “DAPO without dynamic sampling” configuration, use verl/grpo with clip_ratio_high=0.28 and loss_agg_mode=token-mean.

Rollout correction

The rollout engine (vLLM, bf16) and the trainer (FSDP, fp32) are not the same policy even with identical weights. Stale or refilled rollouts widen the gap. Every verl RL recipe exposes the correction:

tp submit my-verl --profile h200:8 \
  -s rollout_calculate_log_probs=true \
  -s rollout_is=sequence \
  -s rollout_is_threshold=2.0

rollout_is_threshold=0.5_5.0 gives the IcePop band, which zeroes rather than clamps outside it. rollout_calculate_log_probs is not optional. Without the rollout engine’s own log-probs there is no behaviour policy to compare against, and the platform refuses the combination at compile time rather than doing nothing quietly. Leaving rollout_is unset reports the mismatch metrics without correcting anything, which is a reasonable first step on a new setup.

Distillation needs two pools

verl/distillation runs the teacher inference servers in their own Ray placement group on top of the trainer’s, so the job declares two pools and maps the teacher role to the second:

pools:
  - {name: train, series: h100, nodes: 1, gpus_per_node: 8}
  - {name: teach, series: h100, nodes: 1, gpus_per_node: 8}
roles: {actor: train, rollout: train, teacher: teach}

The platform fills trainer.nnodes from the student pool and distillation.nnodes from the teacher pool. A single-pool submission is refused at compile time rather than hanging on placement. Multiple pools need a kubernetes or slurm Fleet.

Topology

The adapter writes trainer.nnodes and trainer.n_gpus_per_node from TUNEPLANE_CLUSTER_*, taking the primary pool when the job has a teacher pool.

Check it worked

tp job logs shows the Ray cluster forming, then the first rollout batch. The Charts tab fills with reward and loss series. On a PPO run, train/value_loss that never moves means the critic is not learning, and the advantage is noise.

After training

tp export my-verl --checkpoint checkpoints/global_step_500 --checkpoint-format verl-fsdp

Formats are verl-fsdp and verl-megatron. An SFT evaluation can attach its samples back onto the training run with tp eval --run-id. See pipelines.

Get started

Core concepts

Train a model

Evaluate and serve

Agent environments

Extend the platform

Collaborate

Integrations

Reference

Help

Create with an explicit version

Model and data are required

The method is the estimator

DAPO

Rollout correction

Distillation needs two pools

Topology

Check it worked

After training

​Create with an explicit version

​Model and data are required

​The method is the estimator

​DAPO

​Rollout correction

​Distillation needs two pools

​Topology

​Check it worked

​After training

Create with an explicit version

Model and data are required

The method is the estimator

DAPO

Rollout correction

Distillation needs two pools

Topology

Check it worked

After training