> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Choose a method

> Every method the catalogue publishes, how a method id is pinned, and how to move to a newer version.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp methods                  # everything this deployment offers
tp methods nemo-rl/grpo     # one method's tunable parameters, with ranges
```

`tp methods` is authoritative for what your server has published. The tables below are a map,
not a version pin. The console's [Methods page](https://docs.tuneplane.ai/en/console/methods.md) shows the same catalogue
with the parameters laid out, which is easier to read before a first submission.

A method id is `<framework>/<method>`.

## What a recipe pins

`tp new` writes `recipe.lock.json` beside your config:

| Field | Meaning |
| - | - |
| `recipe.name`, `recipe.version` | The method and the recipe version |
| `recipe.digest` | A digest of the manifest and its templates |
| `framework.kind`, `framework.version` | The exact framework build, such as `verl@0.9.0` |
| `framework.runtime_id` | Which deployment artifact runs it: an OCI image, a SIF, a SQSH |
| `requires.core` | The `tuneplane` versions this recipe works with |

At submit the CLI handshakes with the server's catalogue, and the lock has to match what the
server published. When the platform publishes a newer recipe, an old lock is refused rather
than silently upgraded:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp recipe status my-grpo                        # what changed
tp recipe upgrade my-grpo                       # move to the current catalogue
tp recipe upgrade my-grpo --framework-version 0.9.0
tp submit my-grpo --profile h200:8 --upgrade-recipe   # both at once
```

One recipe can publish several framework versions. Entrypoint changes and moved parameter paths
are declared on the version, not branched in code, so adopting a new upstream release is
usually a manifest and an image:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp new my-verl --method verl/grpo --framework-version 0.9.0
```

## NeMo-RL

The default framework for a full post-training pipeline. Catalogue release `nemo-rl@0.7.0`.

| Method | What it is | When |
| - | - | - |
| `nemo-rl/sft` | Supervised instruction fine-tuning | First stage; its output feeds DPO or GRPO |
| `nemo-rl/dpo` | Direct preference optimisation | Pairwise labels; cheaper than GRPO |
| `nemo-rl/grpo` | Group relative policy optimisation | Math and reasoning RL, no critic |
| `nemo-rl/grpo-lora` | GRPO with LoRA | Tight GPU memory, or a larger base on one node. LoRA learning rate runs one to two orders of magnitude higher than full fine-tuning |
| `nemo-rl/ppo` | PPO with a critic and GAE | Dense rewards, or a distorted in-group baseline |
| `nemo-rl/rm` | Bradley-Terry reward model | An RL reward source, or calibrating an LLM judge |
| `nemo-rl/distillation` | On-policy distillation | The student samples, the teacher supplies full-distribution KL |
| `nemo-rl/maxrl` | Maximum-likelihood RL | Puts weight on low-pass-rate hard problems |
| `nemo-rl/opsd` | Self-distillation | Teacher is the same model plus a reference solution |

## verl

ByteDance verl, synchronous. Catalogue releases `verl@0.9.0` (the default) and `verl@0.9.1`.

Every RL method here runs the same upstream entry point, `verl.trainer.main_ppo`. What separates
them is the advantage estimator the recipe pins. The estimator is not a hyperparameter: setting
it in `config.yaml` is refused at submission, because a config that disagrees with the method it
claims to be is worse than an error.

| Method | What it is |
| - | - |
| `verl/grpo` | Synchronous GRPO |
| `verl/dapo` | GRPO plus the four DAPO techniques, on by default: clip-higher, dynamic sampling, token-level loss, overlong reward shaping |
| `verl/ppo` | GAE PPO with a critic. For dense or process rewards |
| `verl/rloo` | Leave-one-out baseline: unbiased, and no division by the in-group standard deviation |
| `verl/reinforce-baseline` | REINFORCE++-baseline: group mean, then global-batch whitening. The same estimator as `openrlhf/reinforce-baseline` |
| `verl/remax` | Baseline from the model's own greedy answer. One sample per prompt, at the cost of an extra greedy pass |
| `verl/distillation` | On-policy distillation. Needs a second Resource Pool for the teacher |
| `verl/sft` | The upstream SFT trainer, launched with explicit torchrun argv |

Loss variants are parameters, not methods: GSPO, CISPO, GMPO, clip-cov, kl-cov and GPG are all
`policy_loss_mode` on any RL recipe above. Rollout correction is the same — every verl RL method
carries the `rollout_is*` parameters, because the rollout engine and the trainer are never quite
the same policy.

## TRL

Hugging Face TRL with Accelerate. Catalogue release `trl@1.10.0`.

| Method | What it is |
| - | - |
| `trl/sft` | `SFTTrainer` |
| `trl/dpo` | `DPOTrainer` |
| `trl/grpo` | `GRPOTrainer`, online RL |
| `trl/grpo-judge` | GRPO whose reward is the platform judge scoring completions against a versioned [rubric](https://docs.tuneplane.ai/en/guides/rubrics.md). For tasks with no rule that decides correctness |
| `trl/rm` | `RewardTrainer` on a sequence-classification head |
| `trl/kto` | Binary good/bad labels, no pairs |
| `trl/rloo` | Leave-one-out in-group baseline; same family as GRPO |
| `trl/opsd` | `DistillationTrainer`, on-policy. Needs a same-vocabulary teacher |

`trl/grpo-judge` is the one place where training and evaluation share an asset: the same rubric
scores a benchmark, so what you optimise and what you report cannot drift apart.

## OpenRLHF

DeepSpeed ZeRO with a colocated vLLM hybrid engine. Catalogue releases `openrlhf@0.11.0` (the
default) and `openrlhf@0.11.2`.

Images are built by the deployment from `deploy/docker/Dockerfile.openrlhf`, because upstream
publishes no image with the package in it. An administrator sets `TUNEPLANE_IMAGE_OPENRLHF` or
adds it to the runtime registry.

| Method | What it is |
| - | - |
| `openrlhf/sft` | Upstream SFT: packing, multi-turn loss, continued pre-training |
| `openrlhf/dpo` | Upstream DPO, with IPO, cDPO and an NLL regulariser |
| `openrlhf/rm` | Bradley-Terry reward model, usable as `reward_pretrain` for its RL methods |
| `openrlhf/grpo` | Group-norm GRPO, no critic. `no_std_norm` is the Dr.GRPO setting |
| `openrlhf/ppo` | GAE PPO with a critic; actor, critic, reference and reward colocated |
| `openrlhf/reinforce-baseline` | REINFORCE++-baseline. The same estimator as `verl/reinforce-baseline`, so a run moves between stacks unchanged |

## PyTorch

A native PyTorch loop under torchrun. Catalogue release `pytorch@2.14.0`. Images are built by
the deployment from `deploy/docker/Dockerfile.pytorch`.

| Method | What it is |
| - | - |
| `pytorch/image-classification` | The only `supervised` method: labelled pairs trained from initialisation, with no base model. The model and input pipeline live in the experiment's `train.py` |

## Evaluation and custom

| Method | What it is | Where |
| - | - | - |
| `evalkit/benchmark` | lm-eval or evalscope; scores land on the dashboard | [Benchmarks](https://docs.tuneplane.ai/en/guides/benchmarks.md) |
| `custom/custom` | Runs `train.sh`. You bring the image and call `tuneplane.report` | [Custom training](https://docs.tuneplane.ai/en/guides/custom-training.md) |

`custom/custom` requires `--image`. The platform does not guess an entrypoint, and it never
falls back to custom after another adapter fails.

## Pick a starting point from the data you have

```mermaid theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
flowchart TD
    A["What data do you have?"] --> B["Instruction-response"]
    A --> C["Pairwise preferences"]
    A --> D["Binary good/bad"]
    A --> E["Verifiable tasks (math/code/tools)"]
    A --> G["No rule decides correctness"]
    A --> F["A stronger teacher"]
    B --> SFT["nemo-rl/sft, trl/sft, or openrlhf/sft"]
    C --> DPO["dpo on nemo-rl / trl / openrlhf<br/>or rm then RL"]
    D --> KTO["trl/kto"]
    E --> GRPO["nemo-rl/grpo (default)<br/>tight memory: grpo-lora<br/>process rewards: ppo<br/>hard problems: maxrl<br/>verl: grpo, rloo, reinforce-baseline, remax, ppo"]
    G --> JUDGE["trl/grpo-judge with a rubric"]
    F --> DIST["nemo-rl/distillation, verl/distillation or trl/opsd<br/>no external teacher: nemo-rl/opsd"]
```

## Next

Per-framework walkthroughs: [NeMo-RL](https://docs.tuneplane.ai/en/guides/nemo-rl.md) · [verl](https://docs.tuneplane.ai/en/guides/verl.md) ·
[TRL](https://docs.tuneplane.ai/en/guides/trl.md) · [OpenRLHF](https://docs.tuneplane.ai/en/guides/openrlhf.md) · [PyTorch](https://docs.tuneplane.ai/en/guides/pytorch.md) ·
[your own trainer](https://docs.tuneplane.ai/en/guides/custom-training.md)

A common chain is SFT → DPO or GRPO → benchmark. See [pipelines](https://docs.tuneplane.ai/en/guides/pipelines.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.