TUNEPLANE_JOB_ENDPOINT, TUNEPLANE_JOB_RUN_ID and
TUNEPLANE_JOB_TOKEN derived from it. Catalogue methods are instrumented by the runner; a custom
trainer calls tuneplane.report. Stdout and stderr are forwarded
by the runner either way, so a script should never post logs itself.
Endpoints
These are for the runner and the SDK. You usually should not call them by hand.
Ingest tokens are issued per run, scoped and time-limited (
TUNEPLANE_INGEST_TOKEN_DAYS must cover the longest training job). The web guard rejects ingest tokens on regular APIs.
What to watch
TUNEPLANE_INGEST_URL must be reachable from training nodes
TUNEPLANE_INGEST_URL must be reachable from training nodes
Empty charts are most often
127.0.0.1 or an address outside the cluster. Containers / Pods / compute nodes need a direct path to that URL.SSE and reverse proxies
SSE and reverse proxies
Live logs are SSE: disable
proxy_buffering for /api in nginx and relax read timeouts. Clients resume from a cursor after a disconnect.Metric failures never stop training
Metric failures never stop training
SDK reporting is a side channel: failures log, they do not raise. Lifecycle is strict and is retried.
AI diagnostics
AI diagnostics
Failed jobs trigger diagnostics (LLM optional). Inputs are the log tail, metric trends, validation samples, and a config snapshot. A diagnostics failure does not change job status.
Custom training
Do not hand-roll an HTTP client for this.tuneplane wheel in the image, and import tuneplane. Image layout, train.sh and
the empty-chart checklist are in custom training.
Check it worked
Submit a short job and watch the job page. Logs appearing means log ingest works; a first point on the Charts tab means metric ingest works. Those are two different paths and can fail independently.Method coverage and Agent observations
The Charts tab separates observation coverage from training health. A reported candidate is not proof of correctness. The run snapshots its method requirements and enabled capabilities; unknown metrics remain numeric and neutral. Source-specific validation keys retain their dataset, core/aux group and sampling count. Old collapsed history cannot be reconstructed. Enable the capabilities the experiment actually uses. This is observation configuration; it does not enable async training, tools or sandbox execution in the framework.percent display is already 0–100; ratio is 0–1.
Events and distributions
The environment or trainer owns these measurements. Call these functions from its supported hooks, after it has computed the values; the SDK performs no rollout, scoring or distributed reduction. Use one designated reporting process for each aggregate.(-inf,0], (0,1], (1,2], (2,4], (4,8], (8,+inf).
Do not estimate a distribution from its mean. Reusing an event ID retries the same immutable
event; different events at the same learner step remain distinct, including across execution
attempts. The panel shows the latest 100 events and renders histogram buckets separately.
Prompts and trace bodies belong in governed sample/artifact storage, not metric names or event values.
Delivery and capacity limits
Metrics and observations use bounded background admission. Scalar batches contain up to 2,048 points by default; queues hold up to 100,000 scalar points and 128 observation events. An event is at most 16 KiB with 32 bounded scalar/string fields; histograms allow up to 64 finite bounds. A run admits at most 4,096 scalar series. Keep source/bucket dimensions bounded; never put a trajectory ID in a scalar key. Over-limit batches are rejected explicitly. This is best-effort delivery, not a durable local journal. Network outages and process exit can lose pending observations. Transport health reports loss/failure counters and queue occupancy; an absent/stale heartbeat is unknown health, not proof that training stopped. Numerical invalids are recorded separately instead of becoming zeros. Health snapshots remain discoverable even when later events move them out of the recent-event window. Summaries are maintained during ingest; corrected steps repair their extrema. Chart queries reduce data in PostgreSQL before Python allocation, and CSV exports stream original scalars. Large deployments still need capacity testing with their own metric cadence, concurrent runs, PostgreSQL resources and GPU training workload. A schema migration backfills summaries and adds an index; schedule it according to the size of existing telemetry rather than assuming zero downtime.Evaluation outcome coverage
The signed-in job’s Validation tab separates model successes and model failures from execution errors, completed-but-unscored trials and trials not yet reported. These counts come from explicitly classified events across the full execution, not the sampled conversations or the latest 100 events. Old runs without these events show an unknown state. An experiment supplies the meaning of success. Do not turn an arbitrary reward of zero into an execution failure. Report each held-out completion as a separate trial; repeated samples of one task need distinct trial IDs. A task retry incrementstrial_attempt and the highest
reported attempt wins, even when older events arrive later. A job restart is isolated by its
authenticated execution attempt.
status is success, failure, infra_error or unscored. Infrastructure failures and
unscored results cannot be eligible for training. Held-out evaluation should remain
ineligible even when correct. This flag reports the producer’s decision; it does not promote
a record into a dataset or enforce the trainer’s sample filtering.
The success rate uses success / (success + failure); scoring coverage uses
(success + failure) / expected. Unknown population, an empty denominator or excess distinct
trials never produces a fabricated percentage. The population declaration and each terminal
trial attempt are immutable: the first accepted event wins. Use a new round ID for a changed
evaluation population. Results may arrive before the declaration.
These convenience functions use the bounded background observation queue. True means
queued, not durably delivered. Retain original results when reliable replay matters. A bulk
evaluation can use Reporter.observations with validated events in batches of at most 32;
that synchronous API retries transport failures and raises on final failure. The examples
repository’s common/eval/math_eval.py demonstrates this path with the existing binary math
verifier. It marks held-out trials ineligible and keeps generation/verifier exceptions separate
from incorrect model answers. Actual GPU execution still depends on that example’s runtime.
Readiness before a derived run
Use Check readiness in Re-run or a pending diagnosis submit plan. The preview checks recipe compatibility and parameters, source availability, supported effective-configuration rules, fleet fit and runtime-reference resolution. It copies into a temporary directory and cleans it up; it neither submits a job nor allocates a GPU. Changing parameters clears the visible report. Transport errors show a failed check request, never a passing report. Admission policies are checked on actual submission. Worker-side model downloads, mounts, environment connectivity and a real generation/scoring/training smoke run remain explicitly Not checked. Source availability does not certify that a fallback repository reproduces the original archived package. Unsupported framework-specific static rules remain unchecked. New training submissions save their own check record in Config with a timestamp and an input fingerprint. It is evidence of that submission decision, not a continuing permission or a worker health certificate. The fingerprint covers the specification and configuration inputs; it is not a checksum of every source file. Submission always runs the authoritative gates again, regardless of a previous preview.Inspect complete result metadata
In a job’s Validation tab, choose an evaluation round and select Inspect trial results. Filter successes, model failures, execution errors or unscored trials. Select a trial ID for its retry history (up to 100 attempts). Counts and result pages include all reported outcomes; they do not imply that all transcript bodies were retained. Live results can move between filters as retries finish. Authenticated Review sessions also provide Matched evaluation evidence. Select a round explicitly for each run. A success-rate difference is shown only when dataset, cohort, verifier and settings digests match and both rounds have complete scoring coverage. It is an observed difference, not statistical proof. The existing recipe/metric comparison does not establish protocol equivalence. An experiment can declare this evidence before publishing outcomes:Inspect trajectories and compare trial outcomes
The job’s Validation tab separates full result metadata from retained text. Open a trial’s trajectory to page through messages, inspect tool calls and timing, or copy a link to that execution. Missing text does not mean the trial failed. Sampling and truncation are shown separately. Training rollout messages use a separate scope and may report the policy versions used for collection and consumption. Trajectory bodies require object storage. They expire after seven days, with periodic physical cleanup, and can be deleted earlier without deleting result counts. A job owner also needs the applicable governed material content permissions. These bodies are not exposed through public job shares. Custom producers can callReporter.from_env().trajectory(...) when platform
reporting is enabled. Supply scope, round_id, trial_id, trial_attempt,
messages, total_messages, complete and capture_policy (all or sampled).
Each message has a role and content; tool names, call IDs, token counts and
milliseconds are optional. Remove sensitive content before reporting. One document
is limited to 1 MiB and 1,024 messages; declare truncation explicitly. A retained
identity is immutable, so use a new trial attempt for a different execution.
In Review, select matching evaluation rounds to inspect failure-to-success,
success-to-failure and missing-result counts. These are observed trial transitions,
not statistical proof that a model improved. Comparing headline success rates also
requires complete scored coverage and matching declared protocols.
With two runs selected, Start matched benchmark evaluations lets you choose
registered HF exports, one benchmark and an optional sample limit. Preview the
configuration, then start each side. Both jobs pass normal admission and scheduling;
starting one does not automatically start the other. Open the resulting jobs and
compare their reported evaluation rounds when they finish.
The lm-eval integration sends all native result records. Evaluation cohorts of up
to 128 trials retain all available text by default; larger cohorts retain 32.
TUNEPLANE_JOB_VAL_UPLOAD_SAMPLES overrides the cap; zero disables text upload. The effective population reflects
any evaluation limit. Protocol evidence requires native task/version and content
hashes; missing evidence remains visible. Other evaluators can continue reporting
aggregate scores and opt into the outcome/trajectory contracts independently.
Training rollout diagnostics
In the trajectory browser, choose Training rollout, a diagnostic category and optionally a training step. Filters search the server’s full index before pagination. Low reward, invalid/missing reward and ordinary examples are separate categories. A low reward is not automatically a failed answer, and these biased samples must not be used to calculate an overall training success rate. NeMo-RL 0.7 conversation recipes automatically read completed training JSONL logs when the certified TunePlane logger is enabled. Each step retains up to 32 stratified samples;TUNEPLANE_JOB_ROLLOUT_UPLOAD_SAMPLES can reduce that cap or disable it with
zero. Network uploads happen in the background. Queue pressure may drop diagnostic
text and is logged. The adapter bounds message size and explicitly marks truncation.
Other frameworks can report scope="training" through Reporter.trajectory.
Optional fields include diagnostic (failure, anomaly, low_reward,
representative), a finite reward, step, collection_policy and
consumption_policy. Report a failure only when the producer has explicitly
classified it. Missing policy versions stay unknown; a logging step does not prove
which policy generated an asynchronous rollout.
Import native evaluation artifacts
The examples project’scommon/eval/native_import.py imports existing Lego-RL/Harbor
result.json files and optional ATIF agent/trajectory.json files. Run it from the
tuneplane-examples checkout with the current tuneplane SDK installed. It does not
start an agent, a sandbox or training. The supported source-format revision is
ea644fecbc24abf404483d804df4e28caeb44d33; other revisions require adapter validation.
This format pin does not certify a GPU runtime.
Create import-plan.json from the planned evaluation cohort, including trials
that never produced a directory. Supply actual lowercase SHA-256 digests in place
of the four REPLACE_... placeholders below. Hash an immutable dataset manifest,
verifier implementation/configuration, runtime lock or image manifest, and harness
implementation/configuration respectively. Keep each fingerprinting convention
identical across compared runs; the importer cannot attest these declarations.
retry identifies executions of that
trial and the greatest retry number supplies its current outcome. Directory names
are explicit relative paths beneath --root, never inferred task IDs. Traversal,
symlinks and reused directories are rejected. An empty attempts list remains
missing coverage. An explicit attempt without a result is unscored. Only numeric
binary rewards map to success/failure; missing or nonbinary rewards remain unscored.
Verifier timeouts are infrastructure errors. Agent timeouts retain a valid binary
verdict; unknown exceptions remain unscored even if a reward is present.
Prepare and inspect a local snapshot first:
native: round identity includes a content hash. Repeating the same snapshot
within the same platform run does not add trials; changed inputs create a new round.
The protocol fingerprint covers runtime, harness and evaluation settings, while
checkpoint identity is recorded separately for comparisons.
Text is off by default. Add --trace-limit 32 --redactor common.redact:clean only
after implementing that project function. It must accept and return text, removing
sensitive content from messages, tool arguments, observations, names and call IDs.
The function is called before truncation. Inspect the resulting local snapshot.
At most 128 trajectories are retained in deterministic task/repeat/retry order,
each capped at 1 MiB and 1,024 messages. Exported text is always marked sampled and
incomplete: it does not establish the original full history, token counts or
training consumption. Legacy litellm-trajectory.jsonl is not supported.
For delivery, add --upload to the command inside a platform job with its
normal ingest credentials. Results appear in that import job’s existing evaluation
results and trajectory browser; they do not modify a previous native job. All
inputs are validated and the snapshot is written before any upload. A delivery
failure stops the command. To retry the saved snapshot without rereading source
artifacts, run this in the same platform job:
Execute a fixed-checkpoint native evaluation
The examples project’scommon/eval/native_eval.py runs a small explicit cohort
through Harbor, then creates and optionally uploads the same import snapshot.
Use an image with Harbor, the selected agent implementation and sandbox backend
already installed. The model service must serve the declared immutable checkpoint;
hosted_vllm/... is a routing name, not proof of which weights were loaded.
GPU capacity can belong to that model service rather than to this coordinator.
Starting from the import plan above, replace task paths, agent and model endpoint
with your runtime’s actual values:
task.toml.
Docker here means a Harbor backend available to the runtime; merely running the
coordinator in a training container does not supply a Docker service or isolation
for nested agent actions. A remote backend needs its own configuration and
credentials. Keep provider secrets out of the plan and inject them into the
runtime through the normal credential mechanism.
--check-plan validates local paths and declarations without importing Harbor or
starting a sandbox. Execution validates every native TrialConfig before starting
work. Up to 128 trials run sequentially with no automatic retries. Settings evidence
is derived from the actual supplied trial configuration. The output directory must
be new: uncertain prior executions are never silently resumed. Replay a saved
snapshot with the importer’s upload function if only delivery failed.
Harbor owns sandbox creation and cleanup. The coordinator’s deadline cancels the
trial and waits for native cleanup; it is cooperative, not a guaranteed hard
termination deadline. Configure the provider’s resource lifetime limits too.
An exception or timeout stops further trials and preserves the declared denominator;
SIGTERM cancellation saves partial local evidence. Unscored/infrastructure outcomes
or incomplete execution return nonzero; a fully scored cohort with model failures
is still a completed evaluation. Metadata is saved before optional text conversion
or delivery, with coordinator error types in execution-errors.json.
This entry point has CPU tests with a substituted runtime. The native Harbor,
model service and sandbox lifecycle still require an actual deployment smoke test;
no execution backend is certified by these tests. It is a user-project entry point,
not a new built-in recipe or a replacement for the platform’s OpenEnv path.