Start from an engineering solution

tp solution list                    # what this installation has
tp solution show rtl-repair         # what it references, and what counts as better
tp solution init rtl-repair         # copy its starting point into your project

Every piece a team needs to improve a model on one task already exists on this platform, and that is the problem. A recipe, an environment, two benchmark packs, a rubric, a toolchain and a data format are seven objects in six places, each correct on its own, and assembling them into “here is how we do RTL repair” is work every team does again from scratch. What they cannot copy from each other is the part that took the longest: the acceptance protocol — which baseline, which held-out split, how many tasks, how much better counts as better. A Solution is a declaration that names the seven and writes down the eighth.

A Solution runs nothing

It contains no reward function, no harness and no training code. Every field in it is a name, resolved on the deployment that installs it, and the platform never executes anything inside the pack. That boundary is the same one the rest of the platform keeps: the control plane parses, validates, stores and schedules, and what runs inside a job belongs to your project or to a plugin. A Solution that embedded a rubric’s text would be a second copy of it, drifting away from the one the training run actually scores against. A name that does not resolve is reported as missing, which is a fact an operator can act on; an embedded copy that has drifted is not detectable at all.

solution.yaml

schema: tuneplane/solution/v1
name: rtl-repair
title: RTL repair and generation
summary: >
  Train a model to write and fix Verilog against a specification, and judge it by compiling the
  result and running the official testbench.
recipe: trl/sft
benchmarks: [verilogeval-v2, rtllm-v2, ifeval]
toolchain: eda-oss
data_format: spec_rtl
keywords: [rtl, verilog, hardware, repair]
acceptance:
  baseline: prompt
  metrics:
    - key: pass@1
      min_delta: 0.05
    - key: syntax_pass
      min_delta: 0.0
    - key: inst_level_strict_acc
      min_delta: 0.0
  sample_min: 120
  split_by: design
  human_review: true

Field	What it names
`schema`	`tuneplane/solution/v1`. Anything else is refused with the schema this build reads
`name`	The leaf name, and it must equal the directory the manifest sits in
`title`	Required. What a list row says
`summary`	Optional prose
`recipe`	`<owner>/<name>` of the recipe a run starts from
`environment`	`<owner>/<name>@<version>` of the Environment, when the task needs one
`benchmarks`	Benchmark pack ids. Several, because one number is never the whole answer
`rubric`	`<owner>/<name>` of a Rubric, for the part no tool can check
`toolchain`	The Toolchain Profile the verification stages need
`data_format`	One of the nine row shapes
`template`	The subdirectory `tp solution init` copies. Defaults to `template`
`keywords`	Free tags

A manifest that names neither a recipe nor an environment is refused: there would be nothing for a user to start from. A data_format that is not one of the nine is refused with the list.

Every first-party solution compares against a prompt baseline

acceptance.baseline takes a model reference, or the string prompt — a prompt-only baseline on the same base model. All four shipped solutions use prompt, deliberately.

If careful prompting on the base model already meets the need, the right answer is to ship that and measure it, not to run a training job. A protocol with no prompt baseline never surfaces that case, and the training run gets built anyway.

The rest of the protocol is decided the same way: cheap to make honestly in advance, impossible to make honestly afterwards.

Field	What it decides
`baseline`	A model reference, or `prompt`. A baseline chosen after seeing the candidate’s scores is not a baseline
`metrics[].key`	The metric the candidate has to move
`metrics[].min_delta`	How much better, as a change rather than a floor. A floor is a claim about the task’s difficulty nobody can make before measuring. `0.0` is legitimate and means “must not regress”
`metrics[].direction`	`higher` (default) or `lower`. Latency and area are the cases that are not higher
`sample_min`	The fewest evaluation tasks the comparison may rest on. A benchmark of forty problems moves 2.5% when one flips, so a delta below the resolution of the sample is noise with a decimal point
`split_by`	The metadata field that decides the train/eval split, the same thing `spec.data.quality.group_key` names
`human_review`	Whether a person has to look before release. A field rather than an assumption: a functional pass on a testbench is evidence, an EDA script that “looks right” is not

min_delta cannot be negative. A metric where less is better says so with direction: lower rather than by writing a negative delta. Each solution declares at least one guardrail metric at min_delta: 0.0. A model that gains RTL pass rates while quietly losing instruction following has not improved; it has traded one capability for another without saying so.

The four that ship

tp solution list

Solution	Starts from	Benchmarks	Data format	Toolchain
`rtl-repair`	`trl/sft`	`verilogeval-v2`, `rtllm-v2`, `ifeval`	`spec_rtl`	`eda-oss`
`testbench-generation`	`trl/sft`	`verilogeval-v2-completion`	`testbench`	`eda-oss`
`eda-script-assistant`	`trl/sft`	`humaneval`, `ifeval`	`script`	`eda-oss`
`eda-log-triage`	`trl/sft`	`ifeval`	`log_root_cause`	—

rtl-repair has the shortest path from “the model answered” to “a tool says it works”, which is why it is the first one to try on a new deployment. pass@1 must gain 0.05 over the prompt baseline; syntax_pass and inst_level_strict_acc must not regress; at least 120 tasks; split by design. testbench-generation trains the model to write the bench rather than the design. Its two metrics pull in opposite directions on purpose — a bench has to accept a correct design and reject a broken one, and assert(1) scores perfectly on the first alone. eda-script-assistant is the one with the least public data and the most private data. Its two packs are guardrails, not the measurement: what it is actually judged on is a private taskset of the team’s own scripts, which cannot ship in a first-party solution and has to be declared by the deployment that installs it. human_review is load-bearing here — a script that runs is not a script that is right, since it can silently constrain the wrong clock. eda-log-triage involves no simulator and declares no toolchain, so it is the one workflow here a deployment with no EDA toolchain can still run. It splits by case rather than by design: the same failure reported twice is one fact.

Read the protocol before you train

tp solution show rtl-repair

show prints what the solution references, its data format, and then the protocol in the order somebody has to decide it:

baseline                 prompt
pass@1                   at least +0.05
syntax_pass              must not regress
inst_level_strict_acc    must not regress
minimum tasks            120
split by                 design
human review             required

Freeze that before the first run. The whole value of the field is that it was written down first.

Scaffold it into your project

tp solution init rtl-repair --into ./experiments
tp solution init rtl-repair --force          # overwrite an existing directory

init copies the solution’s template/ directory to <into>/<name> and stops. It does not submit, it does not configure the deployment, and it writes nothing the platform then owns. The files land in your repository and are yours to edit; nothing is uploaded, and the template runs in your container at the trust level of anything else you wrote. An existing target directory is refused rather than merged — a half-overwritten scaffold is worse than either outcome — so --force is how you say you meant it.

All four built-in solutions include starting templates. These are files to adapt in your own project, not complete experiments. The console checks whether a template directory exists on the server; your CLI must have the same solution installed to initialize it.

Whether this deployment can run one

A name is only worth publishing if somebody checks that it resolves, and that check is the one thing the SDK cannot do — it knows what the manifest says and nothing about which of those exist here. Two read-only endpoints do it:

curl -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN" \
  https://tuneplane.your-company.com/api/solutions
curl -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN" \
  https://tuneplane.your-company.com/api/solutions/rtl-repair

The list gives ready and the references that did not resolve, named rather than counted — the next thing somebody does is install exactly those. The detail adds every reference with its kind and a note, plus the acceptance protocol as the solution declared it. ready means every name resolves. A solution one benchmark pack short is not “mostly ready”: the acceptance protocol names that pack, so a run without it is measured against something else. Both routes read. They list the Solutions the server’s SDK ships plus any in the server’s own TUNEPLANE_SOLUTION_PATH; neither creates one.

Toolchain references resolve against registered Toolchain Profiles, including whether they are enabled. Configuring a sandbox image alone does not satisfy a named toolchain dependency.

Use the console

Open Assets → Solutions, then View solution. Each detail view shows the data format, input requirements, referenced assets, acceptance protocol and copyable CLI commands. Built-in summaries are translated; team-authored text keeps its original language. Team solutions use namespaced identities such as our-team/our-rtl-flow, including in detail API requests. Dependencies resolved means only that the declared references resolve on this deployment. It does not certify model or data availability, project permissions, compute capacity or a successful run. Unresolved dependencies can be missing, disabled or temporarily unreadable. The asset catalog links and toolchain guidance explain where to investigate. The log-triage and script-assistant workflows need private business evaluation tasks and scoring criteria in addition to their public guardrail benchmarks. A guardrail passing does not establish that a root cause or script is correct. Use Get started to inspect the protocol and copy the template initialization command. When no template exists, the page says so. For all four built-in solutions, Apply to a project saves an immutable application in your own project namespace. Bind a pinned base model, separate prepared training and validation datasets, a private sealed Harbor taskset (owner/name@version, at least the protocol sample minimum; RTL repair requires 120), and a readable business rubric (owner/name@revision). Model and dataset references require a platform version or a full upstream commit SHA. This checks declared inputs, not semantic data leakage. My project applications reopens these snapshots. Each application retains the protocol, recipe digest, input references, taskset digest and rubric revision digest. Changing inputs requires another application. Applications are creator-owned; project access and current rubric and taskset access are checked again before accepting a bound job.

Submit baseline job uses your evaluator source or image. Supply a custom/custom JobSpec with the bound base model and spec.data.taskset, without training data. Independent taskset delivery currently needs the local container backend. Your evaluator owns inference and rubric scoring; the platform does not supply log-triage evaluation code.
Configure candidate training prefills the frozen recipe, model and dataset references. Set resources and parameters, resolve a preview, then explicitly confirm submission. Candidate training does not receive the evaluation taskset.
All submission paths use the existing immutable submission package and confirmation receipt. The package includes .tuneplane/solution.json; the accepted job and application keep the same provenance. Replaying the same confirmation does not create another job.

Associated jobs link to execution logs and outputs. Candidate evaluation selects a completed hf_export or merged_model from a successful candidate training job (or its export step). Use your own evaluator and the same bound base model/taskset, with spec.model.init_from set to that artifact’s exact reference. A different artifact or execution attempt is refused. EDA applications additionally freeze an enabled toolchain profile and require its digest-pinned image for both evaluation jobs. This does not certify the evaluator or enforce sandbox profile network/resource policy inside a custom job.

Acceptance evidence

Select a baseline and a candidate evaluation, then choose Compare current evidence. Comparison is an on-demand snapshot, scoped to each job’s current execution attempt. It requires:

A business round covering every frozen private evaluation task, with the rubric and taskset digests, split="eval", the actual checkpoint reference, and no unscored, infrastructure-error or training-eligible trials. Business success rate must not regress.
Every metric in snapshot.requirements, including its round, direction and native units. Ratio values and deltas are 0–1, never percentages. A metric matching multiple declared packs is required in each matching round. An unmatched metric is reported in the business round in native units. Required benchmark rounds must have matching dataset, task cohort, verifier and settings digests on both sides, and complete scored coverage of their declared population.
A review of that exact evidence digest. Review identity, findings and the evidence snapshot are retained. Approval cannot override a failed metric or incomplete evidence. New attempts, revised trial outcomes or revoked assets invalidate the old review when the comparison refreshes.

The evaluator reports facts through the existing authenticated observation channel. Add reporting to your evaluator, after real inference and scoring; the following is instrumentation, not a runnable evaluator (task_id, actual_status and actual_score come from your code):

import hashlib
import json
from pathlib import Path
from tuneplane import report

binding = json.loads(Path(".tuneplane/solution.json").read_text(encoding="utf-8"))
snapshot = binding["snapshot"]
checkpoint = (binding["artifact"]["ref"] if binding["role"] == "candidate_evaluation"
              else snapshot["bindings"]["model"])
report.validation_population("business", len(snapshot["business_task_ids"]))
report.validation_protocol(
    "business", dataset_digest=snapshot["taskset_digest"],
    cohort_digest=snapshot["business_cohort_digest"],
    verifier_digest=snapshot["rubric_digest"],
    settings_digest=hashlib.sha256(b"your-exact-evaluator-settings").hexdigest(),
    split="eval", checkpoint=checkpoint,
)
# After your evaluator has actually classified each task:
report.validation_outcome("business", task_id, status=actual_status,
                          training_eligible=False)
# After your external benchmark harness has computed this metric:
report.validation_metric("ifeval", "inst_level_strict_acc", actual_score, unit="ratio")

For each required benchmark round, also report its population, protocol and every classified trial, then its required metrics. The cohort digest is SHA-256 of the sorted task ID list encoded as UTF-8 JSON with ensure_ascii=False and separators=(",", ":"). Both evaluators must use the same settings digest. A round/metric key is immutable within an execution attempt; rerun evaluation to correct it. Trial retries use increasing trial_attempt values. At most 10,000 trials per round are compared; a larger population remains insufficient rather than silently using a subset. Export evidence JSON downloads the current comparison and review history (latest 100 entries). Older applications without frozen requirements need a new application. Results remain declarations from the authenticated producer: these checks establish comparable, complete reported evidence, not independent verification of benchmark execution or semantic train/evaluation separation.

Write your own

A Solution of your own is a directory rather than a published plugin. The platform only parses it, so it needs no registry: put it in a directory named by TUNEPLANE_SOLUTION_PATH, laid out <owner>/<name>/, on every machine and server that should see it — a shared mount or a checkout of your team’s repository.

our-team/our-rtl-flow/
├── solution.yaml     # the manifest above
├── template/         # optional: the starting point `tp solution init` copies
└── README.md

export TUNEPLANE_SOLUTION_PATH=/shared/solutions   # holds our-team/our-rtl-flow/
tp solution list                                  # now in the catalog as our-team/our-rtl-flow

Two rules, checked when the catalog loads:

The directory name must equal name. A directory called rtl-flow holding a manifest that says name: our-rtl-flow does not load.
A built-in cannot be shadowed. Built-ins load first and the first writer wins, so a directory named rtl-repair does not replace the shipped one. Two people running “the same” solution and getting different protocols is worse for an acceptance protocol than for a recipe, because the thing that differs is the definition of the result.

A Solution from TUNEPLANE_SOLUTION_PATH is referenced as <owner>/<name>; built-ins keep the bare name. Several directories are separated by : on Linux and macOS.

What a Solution does not do

Be clear about the boundary before you rely on it.

Acceptance evidence is not a release gate. The console compares reported results and records human review. It does not block promotion or deployment. Enforcement still belongs to spec.evaluation.gates — see running benchmarks.
split_by is a statement of intent. It names the field your split should honour. The platform does not compute the leak for you at submission — see data shapes and splits for what is and is not wired.
The console orchestrates explicit submissions. It does not supply inference/scoring code or automatically run the whole workflow.
Nothing here has been scored end to end on real hardware. The hardware-facing solutions point at a toolchain image that has never been built in CI and evaluators no runner in this project can execute; RTL benchmarks says exactly what stayed unproven.

tp solution reference · RTL benchmarks · Data shapes and splits · Benchmark packs · Plugins

Get started

Core concepts

Train a model

Evaluate and serve

Agent environments

Extend the platform

Collaborate

Integrations

Reference

Help

Start from an engineering solution

A Solution runs nothing

Every first-party solution compares against a prompt baseline

The four that ship

Read the protocol before you train

Scaffold it into your project

Whether this deployment can run one

Use the console

Acceptance evidence

Write your own

What a Solution does not do

Next

​A Solution runs nothing

​Every first-party solution compares against a prompt baseline

​The four that ship

​Read the protocol before you train

​Scaffold it into your project

​Whether this deployment can run one

​Use the console

​Acceptance evidence

​Write your own

​What a Solution does not do

​Next

A Solution runs nothing

Every first-party solution compares against a prompt baseline

The four that ship

Read the protocol before you train

Scaffold it into your project

Whether this deployment can run one

Use the console

Acceptance evidence

Write your own

What a Solution does not do

Next