> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Get results back from a job

> How metrics, logs and artifacts reach the console, and what to watch when they do not.

This page describes the SDK/runner HTTP ingest path. Declared TensorBoard and JSONL outputs can instead be collected by the independent Rust collector over the separate telemetry gRPC channel. See [native telemetry deployment and troubleshooting](https://docs.tuneplane.ai/en/ops/telemetry.md). HTTP ingest configuration alone does not configure that channel.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
TUNEPLANE_INGEST_URL=https://tuneplane.your-company.com
```

That address has to be reachable **from a GPU node**, not from the console host. Getting it wrong
is the single most common cause of a job whose logs move and whose charts stay empty.

A training container receives `TUNEPLANE_JOB_ENDPOINT`, `TUNEPLANE_JOB_RUN_ID` and
`TUNEPLANE_JOB_TOKEN` derived from it. Catalogue methods are instrumented by the runner; a custom
trainer calls [`tuneplane.report`](https://docs.tuneplane.ai/en/api-reference/python-sdk). Stdout and stderr are forwarded
by the runner either way, so a script should never post logs itself.

## Endpoints

These are for the runner and the SDK. You usually should not call them by hand.

| Endpoint | Data |
| - | - |
| `/api/ingest/metrics` | Training scalars, as key, step and value, batched |
| `/api/ingest/hparams` | The hyperparameter snapshot the Config tab shows |
| `/api/ingest/logs` | Log lines, as chunks plus an eof marker |
| `/api/ingest/validation` | Validation samples: conversations and rewards |
| `/api/ingest/hardware` | GPU utilisation and memory over time |
| `/api/ingest/lifecycle` | `starting`, `running`, `succeeded`, `failed` |
| `/api/ingest/artifact` | Registers a checkpoint, an export or an evaluation report |
| `/api/ingest/benchmark` | Benchmark scores. Idempotent |
| `/api/ingest/environment` | The run environment snapshot: packages, CUDA, GPU models |
| `/api/ingest/environment/nodes` | The same, per node, on a multi-node job |
| `/api/ingest/artifact/upload-url` | A signed URL to upload an artifact the job produced |

Ingest tokens are issued per run, scoped and time-limited (`TUNEPLANE_INGEST_TOKEN_DAYS` must cover the longest training job). The web guard rejects ingest tokens on regular APIs.

## What to watch

> **TUNEPLANE\_INGEST\_URL must be reachable from training nodes**
>
> Empty charts are most often `127.0.0.1` or an address outside the cluster. Containers / Pods / compute nodes need a direct path to that URL.

> **SSE and reverse proxies**
>
> Live logs are SSE: disable `proxy_buffering` for `/api` in nginx and relax read timeouts. Clients resume from a cursor after a disconnect.

> **Metric failures never stop training**
>
> SDK reporting is a side channel: failures log, they do not raise. Lifecycle is strict and is retried.

> **AI diagnostics**
>
> Failed jobs trigger diagnostics (LLM optional). Inputs are the log tail, metric trends, validation samples, and a config snapshot. A diagnostics failure does not change job status.

## Custom training

Do not hand-roll an HTTP client for this.

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
from tuneplane.report import init, log, finish

init(hparams={"lr": 1e-5})
log({"loss": 0.42}, step=1)
finish()
```

Install the `tuneplane` wheel in the image, and import `tuneplane`. Image layout, `train.sh` and
the empty-chart checklist are in [custom training](https://docs.tuneplane.ai/en/guides/custom-training.md).

## Check it worked

Submit a short job and watch the job page. Logs appearing means log ingest works; a first point
on the Charts tab means metric ingest works. Those are two different paths and can fail
independently.

## Method coverage and Agent observations

The Charts tab separates observation coverage from training health. A reported candidate is
not proof of correctness. The run snapshots its method requirements and enabled capabilities;
unknown metrics remain numeric and neutral. Source-specific validation keys retain their
dataset, core/aux group and sampling count. Old collapsed history cannot be reconstructed.

Enable the capabilities the experiment actually uses. This is observation configuration;
it does not enable async training, tools or sandbox execution in the framework.

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
from tuneplane.report import configure_observability, log, observe

configure_observability(
    capabilities=("agent", "async", "environment", "mixed"),
    descriptors={
        "agent/success_rate": {
            "direction": "higher", "display": "ratio", "unit": "fraction",
            "population": "completed held-out trajectories",
            "reduction": "sum(successes) / sum(completed)",
            "formula": "task verifier v3",
            "comparison_id": "suite-revision-42:heldout:verifier-v3:N8",
        },
    },
)
```

A comparison identity is an explicit assertion by the experiment owner that the formula,
evaluation population and sampling settings agree. Different or absent identities on new
observation contracts do not establish an automatic ranking. Never reuse an identity merely
because two metrics have the same name. `percent` display is already 0–100; `ratio` is 0–1.

### Events and distributions

The environment or trainer owns these measurements. Call these functions from its supported
hooks, after it has computed the values; the SDK performs no rollout, scoring or distributed
reduction. Use one designated reporting process for each aggregate.

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
def report_terminal(event_id, learner_step, policy_version, reason, turns):
    return observe(
        "agent/terminal", event_id=event_id, step=learner_step,
        policy_version=policy_version,
        values={"reason": reason, "turns": turns},
    )


def report_staleness(learner_step, bucket_counts):
    return observe(
        "rollout/policy_age", step=learner_step,
        bounds=[0, 1, 2, 4, 8], counts=bucket_counts,
        values={"unit": "policy_versions"},
    )


def report_quality(learner_step, success_rate):
    log({"agent/success_rate": success_rate}, step=learner_step)
```

Histogram counts are non-cumulative, one per interval plus an overflow bucket. With the bounds
above, supply six counts for `(-inf,0]`, `(0,1]`, `(1,2]`, `(2,4]`, `(4,8]`, `(8,+inf)`.
Do not estimate a distribution from its mean. Reusing an event ID retries the same immutable
event; different events at the same learner step remain distinct, including across execution
attempts. The panel shows the latest 100 events and renders histogram buckets separately.
Prompts and trace bodies belong in governed sample/artifact storage, not metric names or event values.

### Delivery and capacity limits

Metrics and observations use bounded background admission. Scalar batches contain up to 2,048
points by default; queues hold up to 100,000 scalar points and 128 observation events. An event
is at most 16 KiB with 32 bounded scalar/string fields; histograms allow up to 64 finite bounds.
A run admits at most 4,096 scalar series. Keep source/bucket dimensions bounded; never put a
trajectory ID in a scalar key. Over-limit batches are rejected explicitly.

This is best-effort delivery, not a durable local journal. Network outages and process exit can
lose pending observations. Transport health reports loss/failure counters and queue occupancy;
an absent/stale heartbeat is unknown health, not proof that training stopped. Numerical invalids
are recorded separately instead of becoming zeros. Health snapshots remain discoverable even
when later events move them out of the recent-event window.

Summaries are maintained during ingest; corrected steps repair their extrema. Chart queries
reduce data in PostgreSQL before Python allocation, and CSV exports stream original scalars.
Large deployments still need capacity testing with their own metric cadence, concurrent runs,
PostgreSQL resources and GPU training workload. A schema migration backfills summaries and adds
an index; schedule it according to the size of existing telemetry rather than assuming zero downtime.

## Evaluation outcome coverage

The signed-in job's **Validation** tab separates model successes and model failures from
execution errors, completed-but-unscored trials and trials not yet reported. These counts
come from explicitly classified events across the full execution, not the sampled
conversations or the latest 100 events. Old runs without these events show an unknown state.

An experiment supplies the meaning of success. Do not turn an arbitrary reward of zero into
an execution failure. Report each held-out completion as a separate trial; repeated samples
of one task need distinct trial IDs. A task retry increments `trial_attempt` and the highest
reported attempt wins, even when older events arrive later. A job restart is isolated by its
authenticated execution attempt.

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
from tuneplane.report import validation_population, validation_outcome, finish

# Declare all trials, including ones that may fail before scoring.
validation_population("heldout-step-100", expected=100)
validation_outcome(
    "heldout-step-100", "task-42:sample-0",
    status="infra_error", training_eligible=False,
)
# A real retry of that same trial; a transport retry reuses its original attempt.
validation_outcome(
    "heldout-step-100", "task-42:sample-0", trial_attempt=1,
    status="success", training_eligible=False,
)
finish()
```

`status` is `success`, `failure`, `infra_error` or `unscored`. Infrastructure failures and
unscored results cannot be eligible for training. Held-out evaluation should remain
ineligible even when correct. This flag reports the producer's decision; it does not promote
a record into a dataset or enforce the trainer's sample filtering.

The success rate uses `success / (success + failure)`; scoring coverage uses
`(success + failure) / expected`. Unknown population, an empty denominator or excess distinct
trials never produces a fabricated percentage. The population declaration and each terminal
trial attempt are immutable: the first accepted event wins. Use a new round ID for a changed
evaluation population. Results may arrive before the declaration.

These convenience functions use the bounded background observation queue. `True` means
queued, not durably delivered. Retain original results when reliable replay matters. A bulk
evaluation can use `Reporter.observations` with validated events in batches of at most 32;
that synchronous API retries transport failures and raises on final failure. The examples
repository's `common/eval/math_eval.py` demonstrates this path with the existing binary math
verifier. It marks held-out trials ineligible and keeps generation/verifier exceptions separate
from incorrect model answers. Actual GPU execution still depends on that example's runtime.

## Readiness before a derived run

Use **Check readiness** in **Re-run** or a pending diagnosis submit plan. The preview checks
recipe compatibility and parameters, source availability, supported effective-configuration
rules, fleet fit and runtime-reference resolution. It copies into a temporary directory and
cleans it up; it neither submits a job nor allocates a GPU. Changing parameters clears the
visible report. Transport errors show a failed check request, never a passing report.

Admission policies are checked on actual submission. Worker-side model downloads, mounts,
environment connectivity and a real generation/scoring/training smoke run remain explicitly
**Not checked**. Source availability does not certify that a fallback repository reproduces
the original archived package. Unsupported framework-specific static rules remain unchecked.

New training submissions save their own check record in **Config** with a timestamp and an
input fingerprint. It is evidence of that submission decision, not a continuing permission
or a worker health certificate. The fingerprint covers the specification and configuration
inputs; it is not a checksum of every source file. Submission always runs the authoritative
gates again, regardless of a previous preview.

### Inspect complete result metadata

In a job's **Validation** tab, choose an evaluation round and select **Inspect
trial results**. Filter successes, model failures, execution errors or unscored
trials. Select a trial ID for its retry history (up to 100 attempts). Counts and
result pages include all reported outcomes; they do not imply that all transcript
bodies were retained. Live results can move between filters as retries finish.

Authenticated **Review** sessions also provide **Matched evaluation evidence**.
Select a round explicitly for each run. A success-rate difference is shown only
when dataset, cohort, verifier and settings digests match and both rounds have
complete scoring coverage. It is an observed difference, not statistical proof.
The existing recipe/metric comparison does not establish protocol equivalence.

An experiment can declare this evidence before publishing outcomes:

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
report.validation_protocol(
    round_id,
    dataset_digest=dataset_sha256,
    cohort_digest=ordered_trial_ids_sha256,
    verifier_digest=verifier_sha256,
    settings_digest=evaluation_settings_sha256,
    split="heldout-v1",
    checkpoint=immutable_checkpoint_reference,
)
```

Use lowercase SHA-256 digests. Settings must include all relevant generation,
context, repetition, runtime and tool/network settings. These are experiment
producer declarations; the platform does not certify their contents. A round's
protocol is immutable: use a new round ID if any field changes. Old evaluations
without this evidence remain readable but cannot use matched comparison.

## Inspect trajectories and compare trial outcomes

The job's **Validation** tab separates full result metadata from retained text.
Open a trial's trajectory to page through messages, inspect tool calls and timing,
or copy a link to that execution. Missing text does not mean the trial failed.
Sampling and truncation are shown separately. Training rollout messages use a
separate scope and may report the policy versions used for collection and consumption.

Trajectory bodies require object storage. They expire after seven days, with periodic
physical cleanup, and can be deleted earlier without deleting result counts. A job
owner also needs the applicable governed material content permissions. These bodies
are not exposed through public job shares.

Custom producers can call `Reporter.from_env().trajectory(...)` when platform
reporting is enabled. Supply `scope`, `round_id`, `trial_id`, `trial_attempt`,
`messages`, `total_messages`, `complete` and `capture_policy` (`all` or `sampled`).
Each message has a `role` and `content`; tool names, call IDs, token counts and
milliseconds are optional. Remove sensitive content before reporting. One document
is limited to 1 MiB and 1,024 messages; declare truncation explicitly. A retained
identity is immutable, so use a new trial attempt for a different execution.

In **Review**, select matching evaluation rounds to inspect failure-to-success,
success-to-failure and missing-result counts. These are observed trial transitions,
not statistical proof that a model improved. Comparing headline success rates also
requires complete scored coverage and matching declared protocols.

With two runs selected, **Start matched benchmark evaluations** lets you choose
registered HF exports, one benchmark and an optional sample limit. Preview the
configuration, then start each side. Both jobs pass normal admission and scheduling;
starting one does not automatically start the other. Open the resulting jobs and
compare their reported evaluation rounds when they finish.

The lm-eval integration sends all native result records. Evaluation cohorts of up
to 128 trials retain all available text by default; larger cohorts retain 32.
`TUNEPLANE_JOB_VAL_UPLOAD_SAMPLES` overrides the cap; zero disables text upload. The effective population reflects
any evaluation limit. Protocol evidence requires native task/version and content
hashes; missing evidence remains visible. Other evaluators can continue reporting
aggregate scores and opt into the outcome/trajectory contracts independently.

## Training rollout diagnostics

In the trajectory browser, choose **Training rollout**, a diagnostic category and
optionally a training step. Filters search the server's full index before pagination.
Low reward, invalid/missing reward and ordinary examples are separate categories.
A low reward is not automatically a failed answer, and these biased samples must not
be used to calculate an overall training success rate.

NeMo-RL 0.7 conversation recipes automatically read completed training JSONL logs
when the certified TunePlane logger is enabled. Each step retains up to 32 stratified
samples; `TUNEPLANE_JOB_ROLLOUT_UPLOAD_SAMPLES` can reduce that cap or disable it with
zero. Network uploads happen in the background. Queue pressure may drop diagnostic
text and is logged. The adapter bounds message size and explicitly marks truncation.

Other frameworks can report `scope="training"` through `Reporter.trajectory`.
Optional fields include `diagnostic` (`failure`, `anomaly`, `low_reward`,
`representative`), a finite `reward`, `step`, `collection_policy` and
`consumption_policy`. Report a failure only when the producer has explicitly
classified it. Missing policy versions stay unknown; a logging step does not prove
which policy generated an asynchronous rollout.

## Import native evaluation artifacts

The examples project's `common/eval/native_import.py` imports existing Lego-RL/Harbor
`result.json` files and optional ATIF `agent/trajectory.json` files. Run it from the
`tuneplane-examples` checkout with the current `tuneplane` SDK installed. It does not
start an agent, a sandbox or training. The supported source-format revision is
`ea644fecbc24abf404483d804df4e28caeb44d33`; other revisions require adapter validation.
This format pin does not certify a GPU runtime.

Create `import-plan.json` from the **planned evaluation cohort**, including trials
that never produced a directory. Supply actual lowercase SHA-256 digests in place
of the four `REPLACE_...` placeholders below. Hash an immutable dataset manifest,
verifier implementation/configuration, runtime lock or image manifest, and harness
implementation/configuration respectively. Keep each fingerprinting convention
identical across compared runs; the importer cannot attest these declarations.

```json theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
{
  "schema": "tuneplane/native-evaluation-import/v1",
  "source_revision": "ea644fecbc24abf404483d804df4e28caeb44d33",
  "round_id": "heldout-1",
  "checkpoint": "model@immutable-revision",
  "dataset_digest": "REPLACE_DATASET_SHA256",
  "verifier_digest": "REPLACE_VERIFIER_SHA256",
  "runtime_digest": "REPLACE_RUNTIME_SHA256",
  "harness_digest": "REPLACE_HARNESS_SHA256",
  "split": "eval",
  "settings": {
    "sampling": {"temperature": 0, "seed": 42},
    "budgets": {"agent_seconds": 300, "verifier_seconds": 60},
    "tool_policy": {"network": false},
    "retry_policy": "latest explicit retry; never best reward"
  },
  "trials": [
    {"task_id": "task-a", "trial_id": "repeat-0", "attempts": [
      {"retry": 0, "directory": "session-a"},
      {"retry": 1, "directory": "session-a-retry"}
    ]},
    {"task_id": "task-b", "trial_id": "repeat-0", "attempts": []}
  ]
}
```

Use the actual sampling parameters, limits and tool policy used by the evaluator.
Task and repeat IDs identify a planned trial; `retry` identifies executions of that
trial and the greatest retry number supplies its current outcome. Directory names
are explicit relative paths beneath `--root`, never inferred task IDs. Traversal,
symlinks and reused directories are rejected. An empty `attempts` list remains
missing coverage. An explicit attempt without a result is unscored. Only numeric
binary rewards map to success/failure; missing or nonbinary rewards remain unscored.
Verifier timeouts are infrastructure errors. Agent timeouts retain a valid binary
verdict; unknown exceptions remain unscored even if a reward is present.

Prepare and inspect a local snapshot first:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
python -m common.eval.native_import --root ./native-results \
  --plan import-plan.json --output native-evaluation.json
```

The snapshot contains population, protocol and every explicit attempt's outcome.
Its `native:` round identity includes a content hash. Repeating the same snapshot
within the same platform run does not add trials; changed inputs create a new round.
The protocol fingerprint covers runtime, harness and evaluation settings, while
checkpoint identity is recorded separately for comparisons.

Text is off by default. Add `--trace-limit 32 --redactor common.redact:clean` only
after implementing that project function. It must accept and return text, removing
sensitive content from messages, tool arguments, observations, names and call IDs.
The function is called before truncation. Inspect the resulting local snapshot.
At most 128 trajectories are retained in deterministic task/repeat/retry order,
each capped at 1 MiB and 1,024 messages. Exported text is always marked sampled and
incomplete: it does not establish the original full history, token counts or
training consumption. Legacy `litellm-trajectory.jsonl` is not supported.

For delivery, add `--upload` to the command **inside a platform job** with its
normal ingest credentials. Results appear in that import job's existing evaluation
results and trajectory browser; they do not modify a previous native job. All
inputs are validated and the snapshot is written before any upload. A delivery
failure stops the command. To retry the saved snapshot without rereading source
artifacts, run this in the same platform job:

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import json
from pathlib import Path
from common.eval.native_import import upload
from tuneplane.reporter import Reporter

reporter = Reporter.from_env()
if reporter is None:
    raise RuntimeError("Upload requires platform job credentials")
bundle = json.loads(Path("native-evaluation.json").read_text(encoding="utf-8"))
upload(bundle, reporter)
```

A snapshot accepts up to 10,000 planned trials and 10,000 attempt records, with
at most 32 retries per trial. Each result is limited to 1 MiB and each input ATIF
file to 4 MiB. Invalid JSON, non-finite numbers and unsupported structures fail
preparation before network delivery. Freeze the source directory while preparing
a snapshot; the importer does not coordinate with a running native writer.

## Execute a fixed-checkpoint native evaluation

The examples project's `common/eval/native_eval.py` runs a small explicit cohort
through Harbor, then creates and optionally uploads the same import snapshot.
Use an image with Harbor, the selected agent implementation and sandbox backend
already installed. The model service must serve the declared immutable checkpoint;
`hosted_vllm/...` is a routing name, not proof of which weights were loaded.
GPU capacity can belong to that model service rather than to this coordinator.

Starting from the import plan above, replace task paths, agent and model endpoint
with your runtime's actual values:

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
import json
from pathlib import Path

plan = json.loads(Path("import-plan.json").read_text(encoding="utf-8"))
for trial in plan["trials"]:
    trial["attempts"] = []
config = {
    "schema": "tuneplane/native-evaluation-execution/v1",
    "plan": plan,
    "task_paths": {"task-a": "/tasks/task-a", "task-b": "/tasks/task-b"},
    "timeout_seconds": 600,
    "trial": {
        "agent": {
            "name": "openhands",
            "model_name": "hosted_vllm/frozen-model",
            "override_timeout_sec": 300,
            "kwargs": {"api_base": "http://model-service:8000/v1", "temperature": 0}
        },
        "environment": {"type": "docker", "delete": True},
        "verifier": {"disable": False, "override_timeout_sec": 60}
    }
}
Path("native-eval.json").write_text(json.dumps(config, indent=2), encoding="utf-8")
```

The task directory basename must match its task ID and contain `task.toml`.
Docker here means a Harbor backend available to the runtime; merely running the
coordinator in a training container does not supply a Docker service or isolation
for nested agent actions. A remote backend needs its own configuration and
credentials. Keep provider secrets out of the plan and inject them into the
runtime through the normal credential mechanism.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
python -m common.eval.native_eval --config native-eval.json \
  --output ./native-eval-run-1 --check-plan
python -m common.eval.native_eval --config native-eval.json \
  --output ./native-eval-run-1 --upload
```

`--check-plan` validates local paths and declarations without importing Harbor or
starting a sandbox. Execution validates every native TrialConfig before starting
work. Up to 128 trials run sequentially with no automatic retries. Settings evidence
is derived from the actual supplied trial configuration. The output directory must
be new: uncertain prior executions are never silently resumed. Replay a saved
snapshot with the importer's `upload` function if only delivery failed.

Harbor owns sandbox creation and cleanup. The coordinator's deadline cancels the
trial and waits for native cleanup; it is cooperative, not a guaranteed hard
termination deadline. Configure the provider's resource lifetime limits too.
An exception or timeout stops further trials and preserves the declared denominator;
SIGTERM cancellation saves partial local evidence. Unscored/infrastructure outcomes
or incomplete execution return nonzero; a fully scored cohort with model failures
is still a completed evaluation. Metadata is saved before optional text conversion
or delivery, with coordinator error types in `execution-errors.json`.

This entry point has CPU tests with a substituted runtime. The native Harbor,
model service and sandbox lifecycle still require an actual deployment smoke test;
no execution backend is certified by these tests. It is a user-project entry point,
not a new built-in recipe or a replacement for the platform's OpenEnv path.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.