0.11.0 (the default) and 0.11.2. Methods: openrlhf/sft,
dpo, rm, grpo, ppo, reinforce-baseline. Configuration is CLI flags rather than Hydra,
and the adapter launches it as a Python module.
0.11.2 replaced the off-policy correction switches.
is_correction and
is_correction_type exist only on 0.11.0; 0.11.2 has is_correction_level,
is_correction_mode and is_correction_gating instead, which take the same three schemes
apart into the axes they were made of (tis = token + clip, icepop = token + mask,
seq-mask-tis = seq + mask). A parameter the selected version does not have is refused at
validation rather than at container start.What the flags bind to
Everything shares the GPUs
The GRPO recipe pinsalgo.advantage.estimator=group_norm and train.colocate_all=true, plus
the vLLM and DeepSpeed sleep flags. Actor, critic, reference and reward all sit on the same
cards through the hybrid engine, and the topology expands into each role’s num_nodes and
num_gpus_per_node.
Picking an estimator
no_std_norm centres advantages on the mean without dividing by the standard deviation. That
is the Dr.GRPO convention, and it avoids difficulty bias.
openrlhf/reinforce-baseline is REINFORCE++-baseline: global batch normalisation over a group
mean. verl/reinforce-baseline is the same estimator, so a run moves between the two stacks
unchanged.
Where the reward comes from is a choice of one field or the other: reward_pretrain for a
reward model, or remote_reward_url for an external scorer.
Check it worked
tp job logs shows the vLLM engine waking, then the first rollout. The Charts tab fills with
the reward and loss series. If the logs stop after the Ray head starts and no worker joins, the
image’s Ray version does not match what the recipe expects.