> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Verifiers

> The three ways a task's completion is decided, and the exact contract of each.

```json manifest.json theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
{ "verifier": { "kind": "rubric", "ref": "alice/arithmetic-correctness" } }
```

A verifier decides whether a task was completed. It is always a **reference** to something that
already owns that answer — never something the platform implements.

## The three kinds

| Kind | `ref` is | Who owns the judgement |
| - | - | - |
| `rubric` | `<owner>/<name>` of a [Rubric](https://docs.tuneplane.ai/en/extend/rubrics.md) | Your team's written standard |
| `plugin` | `module:callable` in a `capability: environment` plugin | Your code |
| `endpoint` | An HTTP URL | A service you run |

There is deliberately no fourth kind, and in particular no built-in.

> **Why the platform is never a verifier**
>
> The last time this repository grew a built-in scorer, it wrote its own simulation pipeline with its
> own weights, arrived at a different judgement from the evaluation side, and had to be reverted
> whole. A verifier living in the control plane would be a second definition of "correct" no matter
> how small it started, and the two definitions would disagree exactly when it mattered.

## Contracts

Whatever the kind, the resolved verifier looks the same from the platform's side:

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
VerifyFn = Callable[[Mapping[str, Any], Any], float]
```

Two arguments: the task, and whatever the harness submitted as the trajectory. One return: a reward.

**rubric**

```json theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
{ "verifier": { "kind": "rubric", "ref": "alice/arithmetic-correctness" } }
```

The platform judge scores the trajectory against the named rubric version. It receives the task's
`prompt`, the flattened trajectory text, and the task's `reference` if there is one.

The trajectory is flattened for you — a string stays a string, a message object contributes its
`content` or `text`, and a list is joined. Harnesses differ in shape and the verifier contract is
about what the model produced, so the folding happens once here rather than in every author's code.

**plugin**

```json theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
{ "verifier": { "kind": "plugin", "ref": "verify:score_arithmetic" } }
```

```python verify.py theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
from typing import Any, Mapping

def score_arithmetic(task: Mapping[str, Any], trajectory: Any) -> float:
    expected = str(task.get("reference", "")).strip()
    answer = str(trajectory).strip().split()[-1] if trajectory else ""
    return 1.0 if answer == expected else 0.0
```

Imported from a `capability: environment` plugin the launcher has already vetted through the digest
gates. Imported lazily and cached: an import error surfaces on the first task, where it reads as a
deployment problem, rather than at startup where it would look like the environment itself is
broken.

Must return something `float()` accepts. Raising is fine and correct — see below.

**endpoint**

```json theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
{ "verifier": { "kind": "endpoint", "ref": "https://verifier.corp/score" } }
```

The platform POSTs JSON and expects a reward back:

```json Request theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
{ "task": { "prompt": "...", "reference": "..." }, "trajectory": "..." }
```

```json Response theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
{ "reward": 0.83 }
```

A response without a `reward` field, or with a non-numeric one, is an error rather than a zero.
The default timeout is 60 seconds.

## A failed verification is an error, never a zero

> [!WARNING]
>
> If your verifier cannot answer — the endpoint is down, the module will not import, the rubric is
> gone — the platform raises. It does not report a reward of `0.0`.

This is the single most important rule for anyone writing a verifier, and it is worth being blunt
about why: a silently zeroed reward is **indistinguishable from an answer that was actually wrong**.
A training run whose verifier quietly broke at step 400 would keep producing a reward curve, keep
looking plausible, and teach the model from noise for the remaining hours of the job.

So in your own verifier code:

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
def score(task, trajectory):
    if not task.get("reference"):
        raise ValueError("this task has no reference; it cannot be scored")   # correct
        # return 0.0                                                          # wrong
    ...
```

Errors are wrapped as `VerificationError` with the reference that failed, so the job log names which
verifier stopped and why.

## Picking one

| Use | When |
| - | - |
| `rubric` | Correctness is a matter of judgement: quality, tone, whether an explanation is sound. Also whenever you want one standard to score both training rewards and benchmarks |
| `plugin` | Correctness is computable, cheap and local: an exact match, a parser, a unit test |
| `endpoint` | Something already decides this — a compiler service, a simulator, an internal grading system. Do not reimplement it in a plugin |

## Check it worked

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp env show alice/calculator-tasks@1.0.0
```

Prints the verifier the manifest declares. Then submit a short run and watch the reward
distribution on the Validation tab: a reward flat at exactly one value for every rollout is a
verifier returning a constant, which looks identical to a task the model cannot do.

## Next

[Author an environment](https://docs.tuneplane.ai/en/extend/environments.md) · [Writing rubrics](https://docs.tuneplane.ai/en/extend/rubrics.md) ·
[Using an environment](https://docs.tuneplane.ai/en/guides/agent-envs.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.