> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Train with verl

> Required model and data, how the method pins its advantage estimator, DAPO, rollout correction, and distillation's second pool.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp new my-verl --method verl/grpo --framework-version 0.9.0
tp submit my-verl --profile h200:8 \
  --model Qwen/Qwen2.5-7B --train-data data/train.parquet
```

ByteDance verl at catalogue versions `0.9.0` (the default) and `0.9.1`. Methods: `verl/grpo`,
`dapo`, `ppo`, `rloo`, `reinforce-baseline`, `remax`, `distillation`, `sft`.

A new upstream version is opt-in before it is the default: pass `--framework-version 0.9.1`
to try it, and see [Adopt a new upstream version](https://docs.tuneplane.ai/en/ops/upgrades.md) for why.

Unlike NeMo-RL, verl needs `--model` and `--train-data` at submission. Observability is
`platform`: the integration adds `tensorboard` to `trainer.logger` and declares its output
directory. An isolated collector reads those files; other explicitly configured trackers remain.
No Tracking constructor or import hook is patched.

> [!NOTE]
>
> verl images are deployment artifacts. An administrator sets `TUNEPLANE_IMAGE_VERL` or registers
> the runtime id before anybody can submit. Without one, submission fails naming the missing
> runtime rather than the method.

## Create with an explicit version

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp new my-verl --method verl/grpo --framework-version 0.9.0
```

0.9 uses the unified `verl.trainer.main_ppo`. Upstream removed 0.8's `main_ppo_sync`, so the
catalogue no longer carries the older runtime.

If `experiments/my-verl/main.py` exists, that file becomes the entrypoint. That is the
verl-recipe style: subclass `TaskRunner` or `RayPPOTrainer`. A reward-only change does not need
it — set `custom_reward_function.path` in the config instead.

## Model and data are required

The recipe binds three flags onto Hydra keys:

| Flag | Hydra key |
| - | - |
| `--model` | `actor_rollout_ref.model.path` |
| `--train-data` | `data.train_files` |
| `--validation-data` | `data.val_files` |

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-verl --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-data data/train.parquet \
  --validation-data data/val.parquet
```

With a platform dataset, `--train-data` is a path inside it:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-verl --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-dataset alice/gsm8k-zh@v2 \
  --train-data train.parquet
```

Prefer `data.train.dataset` in the config; the flag is a temporary override. See
[datasets](https://docs.tuneplane.ai/en/guides/datasets.md). `tp validate` fails early when a required declaration is
missing.

## The method is the estimator

Every RL method here runs the same `verl.trainer.main_ppo`. The recipe pins
`algorithm.adv_estimator`, and that is what the method is.

| Method | Estimator | Baseline from | Notes |
| - | - | - | - |
| `verl/grpo` | `grpo` | in-group mean ÷ in-group std | needs `rollout.n ≥ 2` |
| `verl/rloo` | `rloo` | the mean of the *other* samples | unbiased, no std division |
| `verl/reinforce-baseline` | `reinforce_plus_plus_baseline` | group mean, then global-batch whitening | same estimator as `openrlhf/reinforce-baseline` |
| `verl/remax` | `remax` | the model's own greedy answer | `rollout.n = 1`, plus one greedy pass per step |
| `verl/ppo` | `gae` | a trained critic | a second full-size model; watch `train/value_loss` |
| `verl/dapo` | `grpo` | in-group mean on filtered groups | the four DAPO techniques, on by default |

Writing `algorithm.adv_estimator` into `config.yaml` is refused at submission. To change
estimator, change method.

Loss variants are knobs, not methods: set `policy_loss_mode` on any of these recipes for GSPO,
CISPO, GMPO, clip-cov, kl-cov or GPG.

## DAPO

`verl/dapo` is configuration, not a vendored trainer. 0.9 carries all four published techniques
in mainline `main_ppo`, so the recipe pins `algorithm.filter_groups.enable` and the `dapo`
reward manager, and ships the paper's values as defaults: clip 0.2/0.28, `token-mean`
aggregation, a 4096-token overlong buffer at factor 1.0, 16 samples per prompt, and no KL on
either path.

Two things to know before submitting one:

* Dynamic sampling judges a group on `filter_metric` **during rollout**, so the reward has to
  exist then. Use a rule-based reward function, or a reward model in its own pool. A colocated
  reward model scores after sampling, and upstream asserts on that.
* Step time varies by construction: the trainer keeps regenerating until enough groups qualify.

For the "DAPO without dynamic sampling" configuration, use `verl/grpo` with
`clip_ratio_high=0.28` and `loss_agg_mode=token-mean`.

## Rollout correction

The rollout engine (vLLM, bf16) and the trainer (FSDP, fp32) are not the same policy even with
identical weights. Stale or refilled rollouts widen the gap. Every verl RL recipe exposes the
correction:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-verl --profile h200:8 \
  -s rollout_calculate_log_probs=true \
  -s rollout_is=sequence \
  -s rollout_is_threshold=2.0
```

`rollout_is_threshold=0.5_5.0` gives the IcePop band, which zeroes rather than clamps outside
it.

`rollout_calculate_log_probs` is not optional. Without the rollout engine's own log-probs there
is no behaviour policy to compare against, and the platform refuses the combination at compile
time rather than doing nothing quietly.

Leaving `rollout_is` unset reports the mismatch metrics without correcting anything, which is a
reasonable first step on a new setup.

## Distillation needs two pools

`verl/distillation` runs the teacher inference servers in their own Ray placement group on top
of the trainer's, so the job declares two pools and maps the `teacher` role to the second:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
pools:
  - {name: train, series: h100, nodes: 1, gpus_per_node: 8}
  - {name: teach, series: h100, nodes: 1, gpus_per_node: 8}
roles: {actor: train, rollout: train, teacher: teach}
```

The platform fills `trainer.nnodes` from the student pool and `distillation.nnodes` from the
teacher pool. A single-pool submission is refused at compile time rather than hanging on
placement. Multiple pools need a `kubernetes` or `slurm` Fleet.

## Topology

The adapter writes `trainer.nnodes` and `trainer.n_gpus_per_node` from `TUNEPLANE_CLUSTER_*`,
taking the primary pool when the job has a teacher pool.

## Check it worked

`tp job logs` shows the Ray cluster forming, then the first rollout batch. The Charts tab fills
with reward and loss series. On a PPO run, `train/value_loss` that never moves means the critic
is not learning, and the advantage is noise.

## After training

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp export my-verl --checkpoint checkpoints/global_step_500 --checkpoint-format verl-fsdp
```

Formats are `verl-fsdp` and `verl-megatron`. An SFT evaluation can attach its samples back onto
the training run with `tp eval --run-id`. See [pipelines](https://docs.tuneplane.ai/en/guides/pipelines.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.