> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Collect preference annotations

> Pair two runs' answers, have people judge them in Argilla, and pull the result back as DPO training data.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp annotate status                                       # is the integration configured
tp annotate push run-4f2a91 run-8b1c02 --limit 200       # pair and send out
# people annotate in Argilla
tp annotate pull tuneplane-run-4f2a91-vs-run-8b1c02 --into qa-preference
```

The last command lands the preferences as a new version of the `qa-preference` dataset, through
the same path as `tp dataset push`: immutable versions, a fingerprint, a quality report.

An LLM judge scores automatically. Some judgements only a person can make, and this is the chain
that turns those back into training data.

## Before you start

* Two finished runs with validation samples, on overlapping prompts.
* An Argilla deployment your administrator has configured. `tp annotate status` says so.

## Pair two runs

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp annotate push run-4f2a91 run-8b1c02 --limit 200
tp annotate push run-4f2a91 run-8b1c02 -d q3-writing-quality
```

Two runs are required: preference annotation asks which of A and B is better, and a single run's
samples can only be rated, not compared.

Pairs are aligned **by prompt**, not by sample index. Two validation passes do not guarantee the
same sampling order, so index alignment would pair A's third question with B's third and the
preferences would mean nothing.

Pairs whose answers are identical are dropped before they reach anybody.

## Pull the result back

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp annotate pull tuneplane-run-4f2a91-vs-run-8b1c02              # just the statistics
tp annotate pull tuneplane-run-4f2a91-vs-run-8b1c02 --into qa-preference
tp annotate pull tuneplane-run-4f2a91-vs-run-8b1c02 --into qa-preference -v 20260912
```

You get `prompt / chosen / rejected` rows. Without `--version` the version is today's date.

Then point the next DPO run at it:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
data:
  train:
    dataset: alice/qa-preference@20260912
```

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-dpo --profile h200:8
```

## Check it worked

`tp annotate pull` prints how many pairs it read, how many produced a preference, and how many
were skipped. `tp dataset ls alice/qa-preference` shows the new version.

A "can't tell" answer produces no training data: DPO needs an ordering, and forcing ties into it
teaches the model a signal that was never there. The skipped count is printed so it does not look
like lost data.

When several people annotate one pair, they are resolved by majority into **one** row. A pair
they split evenly has no ordering anybody agreed on, so it produces no row and is counted
separately. Pulling needs the same access as pushing: you can read a dataset back only if you can
reach both runs its answers came from, and a dataset this platform did not push is not readable
through it.

## Next

[Datasets](https://docs.tuneplane.ai/en/guides/datasets.md) · [Rubrics](https://docs.tuneplane.ai/en/guides/rubrics.md) for the automated version of the
same judgement · [Pipelines](https://docs.tuneplane.ai/en/guides/pipelines.md)

> **Which part is Argilla's, and why the ordering is randomised**
>
> The annotation interface, permissions, progress tracking and inter-annotator agreement are
> [Argilla](https://argilla.io)'s. The platform does the two ends Argilla cannot: pairing, which
> needs both runs' validation samples, and landing the result as a versioned dataset.
>
> A/B order is derived from the prompt rather than from which run is which. It is stable across
> re-opens, so an annotator returning to a record sees the same layout, but an annotator cannot
> learn that "A is always the new model" and develop a systematic bias.

> **Configuration (an administrator does this)**
>
> ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
> TUNEPLANE_ARGILLA_URL=http://argilla.internal:6900
> TUNEPLANE_ARGILLA_API_KEY=<argilla API key>
> TUNEPLANE_ARGILLA_WORKSPACE=argilla        # must already exist in Argilla
> ```
>
> This address is deployment configuration, at the same trust level as the database and
> object-storage addresses, so the "reject private addresses" rule that applies to
> user-supplied webhook URLs is not applied here. On an internal deployment Argilla is on a
> private network by definition.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.