Skip to main content
Tasksets live under Assets. Agent evaluations live under Evaluation. A taskset defines the work and its train/eval split. An environment defines where an agent acts. Harbor coordinates the agent, sandbox and verifier for each trial. You can reuse a taskset across models and execution environments.

​
Publish an immutable taskset

Place Harbor task directories beneath a common root. Each directory must contain task.toml; task IDs equal directory names and must be unique.
build records file hashes in taskset.json. To mix training and evaluation tasks, edit individual split fields before publishing. Modified task content requires updated hashes and a new version; existing manifests are never overwritten by build. Symlinks, overlapping task directories and path traversal are rejected. The console displays sealed versions, split counts and the content digest. It exposes no task text or reference-answer download. Private versions are visible only to authorized users. Jobs reference owner/name@version; floating versions are rejected. Authorized local-container jobs receive a read-only taskset mount through TUNEPLANE_JOB_TASKSET_DIR. Other delivery backends are not supported yet. Split labels select the evaluation cohort; they do not hide files from arbitrary code inside an authorized job. Publish held-out tasks as a separate taskset when you need separate access permissions.

​
Prepare a Harbor evaluation

Open Agent evaluations, or follow the link from a taskset version. Choose:
  • A sealed taskset version; only its evaluation split runs.
  • A hosted model endpoint and immutable model revision.
  • Docker or E2B as the sandbox backend available in your runtime.
  • Runtime and harness SHA-256 identities, repetitions, concurrency and trial timeout.
The page validates the request and prepares harbor-evaluation.json. Preparation alone does not start a job.

​
Launch from the console

For the first run, use the harbor-eval experiment in tuneplane-examples, put the prepared file at its project root, and submit it with tp submit --taskset, a configured local profile and a pinned runtime image. This archives the user-owned adapter and execution configuration. --check-plan validates local task content without importing Harbor or creating sandboxes. For subsequent runs, choose that job as the Execution source. The console reuses its archived code, image, resources and profile; it never falls back to a server checkout when the archive is unavailable. Runtime and harness identities come from the source. Change the model endpoint, revision, taskset, sandbox pool, repetitions, concurrency or timeout, prepare the request, then Start evaluation. The job passes the same access, quota and queue admission as a CLI submission. Repeated submission of the same request returns the same job. Editing inputs creates a new request identity. The source list contains custom jobs with a taskset; selecting one verifies that its archive contains the compatible Harbor adapter and configuration. Missing archives or incompatible sources are refused before submission. The console does not build a runtime image or install Harbor for the user.

​
History and comparison

Evaluation history lists evaluations launched from this page, including queued and failed jobs, with their model revision, taskset and submission time. Records and the exact prepared configuration are stored with the job ledger; refreshing the browser does not lose them. Open a job for logs and cancellation, or View results for coverage, classified outcomes, trial attempts and traces. A job with no reported results stays empty rather than appearing successful. Select two runs for comparison; the first is the baseline. Select a reported round for each. The comparison checks the declared task cohort, verifier and execution settings and identifies incompatible inputs before showing score differences. Missing and unscored results remain visible. It uses the existing job outcome and comparison services, including their access controls.

​
Capacity and results

A request supports up to 10,000 planned trials and 256 concurrent trials. Without a selected sandbox pool, this is only a per-job limit. Actual capacity depends on the provider and runtime. Docker access or E2B credentials must be provisioned in the runtime; selecting a backend in the console does not provision it. Infrastructure failures stop new trials. Cooperative cancellation waits for started trials to clean up and saves partial results. Force-kill cleanup requires provider TTLs and reconciliation. Every planned outcome contributes to coverage; missing and unscored results do not become successful evaluations. Text capture is disabled by default. Taskset content, verifier content, runtime, harness and actual execution settings accompany results for comparison. Runtime and harness hashes are declarations, not certification of the installed backend. Shared pool capacity and leases are available through the Sandbox pools page. Native provider certification and automatic orphan reconciliation remain separate work.