> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Train with OpenRLHF

> A DeepSpeed plus colocated vLLM stack: the self-built image, the bindings, and GRPO against REINFORCE++-baseline.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp new my-orlhf --method openrlhf/grpo
tp submit my-orlhf --profile h200:8 \
  --model Qwen/Qwen3.5-9B --train-data data/prompts.jsonl
```

OpenRLHF at catalogue versions `0.11.0` (the default) and `0.11.2`. Methods: `openrlhf/sft`,
`dpo`, `rm`, `grpo`, `ppo`, `reinforce-baseline`. Configuration is CLI flags rather than Hydra,
and the adapter launches it as a Python module.

> [!NOTE]
>
> 0.11.2 replaced the off-policy correction switches. `is_correction` and
> `is_correction_type` exist only on 0.11.0; 0.11.2 has `is_correction_level`,
> `is_correction_mode` and `is_correction_gating` instead, which take the same three schemes
> apart into the axes they were made of (`tis` = token + clip, `icepop` = token + mask,
> `seq-mask-tis` = seq + mask). A parameter the selected version does not have is refused at
> validation rather than at container start.

> [!WARNING]
>
> Upstream's Dockerfile installs the dependency stack but **not** the `openrlhf` package, and
> publishes no ready training image. An administrator has to build one from
> `deploy/docker/Dockerfile.openrlhf` and set `TUNEPLANE_IMAGE_OPENRLHF`, or register the
> `openrlhf-0.11.0` runtime id (or `openrlhf-0.11.2`), before anyone can submit.
>
> The Ray version is pinned by that image. 0.11.0 pins `ray==2.55.0` upstream, so do not copy
> another framework's pin — a mismatch presents as "the cluster came up but workers never
> registered".

## What the flags bind to

| Flag | OpenRLHF field |
| - | - |
| `--model` | `actor.model_name_or_path` |
| `--train-data` | `data.prompt_dataset` |
| `--validation-data` | `eval.dataset` |

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-orlhf --profile h200:8 \
  --model Qwen/Qwen3.5-9B \
  --train-data data/prompts.jsonl
```

## Everything shares the GPUs

The GRPO recipe pins `algo.advantage.estimator=group_norm` and `train.colocate_all=true`, plus
the vLLM and DeepSpeed sleep flags. Actor, critic, reference and reward all sit on the same
cards through the hybrid engine, and the topology expands into each role's `num_nodes` and
`num_gpus_per_node`.

## Picking an estimator

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-orlhf --profile h200:8 \
  --model Qwen/Qwen3.5-9B --train-data data/prompts.jsonl \
  -s no_std_norm=true \
  -s n_samples_per_prompt=8
```

`no_std_norm` centres advantages on the mean without dividing by the standard deviation. That
is the Dr.GRPO convention, and it avoids difficulty bias.

`openrlhf/reinforce-baseline` is REINFORCE++-baseline: global batch normalisation over a group
mean. `verl/reinforce-baseline` is the same estimator, so a run moves between the two stacks
unchanged.

Where the reward comes from is a choice of one field or the other: `reward_pretrain` for a
reward model, or `remote_reward_url` for an external scorer.

## Check it worked

`tp job logs` shows the vLLM engine waking, then the first rollout. The Charts tab fills with
the reward and loss series. If the logs stop after the Ray head starts and no worker joins, the
image's Ray version does not match what the recipe expects.

## Next

[Method catalogue](https://docs.tuneplane.ai/en/guides/methods.md) · [Custom images](https://docs.tuneplane.ai/en/guides/custom-images.md) for how a
deployment publishes one · [Pipelines](https://docs.tuneplane.ai/en/guides/pipelines.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.