> ## Documentation Index > Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt > Use this file to discover all available pages before exploring further. # Start from an engineering solution > A recipe, benchmark packs, a toolchain and the acceptance protocol that says what counts as better. ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp solution list # what this installation has tp solution show rtl-repair # what it references, and what counts as better tp solution init rtl-repair # copy its starting point into your project ``` Every piece a team needs to improve a model on one task already exists on this platform, and that is the problem. A recipe, an environment, two benchmark packs, a rubric, a toolchain and a data format are seven objects in six places, each correct on its own, and assembling them into "here is how we do RTL repair" is work every team does again from scratch. What they cannot copy from each other is the part that took the longest: the **acceptance protocol** — which baseline, which held-out split, how many tasks, how much better counts as better. A Solution is a declaration that names the seven and writes down the eighth. ## A Solution runs nothing It contains no reward function, no harness and no training code. Every field in it is a *name*, resolved on the deployment that installs it, and the platform never executes anything inside the pack. That boundary is the same one the rest of the platform keeps: the control plane parses, validates, stores and schedules, and what runs inside a job belongs to your project or to a plugin. A Solution that embedded a rubric's text would be a second copy of it, drifting away from the one the training run actually scores against. A name that does not resolve is reported as missing, which is a fact an operator can act on; an embedded copy that has drifted is not detectable at all. ```yaml solution.yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} schema: tuneplane/solution/v1 name: rtl-repair title: RTL repair and generation summary: > Train a model to write and fix Verilog against a specification, and judge it by compiling the result and running the official testbench. recipe: trl/sft benchmarks: [verilogeval-v2, rtllm-v2, ifeval] toolchain: eda-oss data_format: spec_rtl keywords: [rtl, verilog, hardware, repair] acceptance: baseline: prompt metrics: - key: pass@1 min_delta: 0.05 - key: syntax_pass min_delta: 0.0 - key: inst_level_strict_acc min_delta: 0.0 sample_min: 120 split_by: design human_review: true ``` | Field | What it names | | - | - | | `schema` | `tuneplane/solution/v1`. Anything else is refused with the schema this build reads | | `name` | The leaf name, and it must equal the directory the manifest sits in | | `title` | Required. What a list row says | | `summary` | Optional prose | | `recipe` | `/` of the recipe a run starts from | | `environment` | `/@` of the Environment, when the task needs one | | `benchmarks` | Benchmark pack ids. Several, because one number is never the whole answer | | `rubric` | `/` of a Rubric, for the part no tool can check | | `toolchain` | The Toolchain Profile the verification stages need | | `data_format` | One of the [nine row shapes](https://docs.tuneplane.ai/en/guides/data-formats.md) | | `template` | The subdirectory `tp solution init` copies. Defaults to `template` | | `keywords` | Free tags | A manifest that names **neither** a recipe nor an environment is refused: there would be nothing for a user to start from. A `data_format` that is not one of the nine is refused with the list. ## Every first-party solution compares against a prompt baseline `acceptance.baseline` takes a model reference, or the string `prompt` — a prompt-only baseline on the same base model. All four shipped solutions use `prompt`, deliberately. > [!NOTE] > > **If careful prompting on the base model already meets the need, the right answer is to ship > that and measure it, not to run a training job.** A protocol with no prompt baseline never > surfaces that case, and the training run gets built anyway. The rest of the protocol is decided the same way: cheap to make honestly in advance, impossible to make honestly afterwards. | Field | What it decides | | - | - | | `baseline` | A model reference, or `prompt`. A baseline chosen after seeing the candidate's scores is not a baseline | | `metrics[].key` | The metric the candidate has to move | | `metrics[].min_delta` | How much better, as a **change** rather than a floor. A floor is a claim about the task's difficulty nobody can make before measuring. `0.0` is legitimate and means "must not regress" | | `metrics[].direction` | `higher` (default) or `lower`. Latency and area are the cases that are not higher | | `sample_min` | The fewest evaluation tasks the comparison may rest on. A benchmark of forty problems moves 2.5% when one flips, so a delta below the resolution of the sample is noise with a decimal point | | `split_by` | The metadata field that decides the train/eval split, the same thing `spec.data.quality.group_key` names | | `human_review` | Whether a person has to look before release. A field rather than an assumption: a functional pass on a testbench is evidence, an EDA script that "looks right" is not | `min_delta` cannot be negative. A metric where less is better says so with `direction: lower` rather than by writing a negative delta. Each solution declares at least one guardrail metric at `min_delta: 0.0`. A model that gains RTL pass rates while quietly losing instruction following has not improved; it has traded one capability for another without saying so. ## The four that ship ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp solution list ``` | Solution | Starts from | Benchmarks | Data format | Toolchain | | - | - | - | - | - | | `rtl-repair` | `trl/sft` | `verilogeval-v2`, `rtllm-v2`, `ifeval` | `spec_rtl` | `eda-oss` | | `testbench-generation` | `trl/sft` | `verilogeval-v2-completion` | `testbench` | `eda-oss` | | `eda-script-assistant` | `trl/sft` | `humaneval`, `ifeval` | `script` | `eda-oss` | | `eda-log-triage` | `trl/sft` | `ifeval` | `log_root_cause` | — | **`rtl-repair`** has the shortest path from "the model answered" to "a tool says it works", which is why it is the first one to try on a new deployment. `pass@1` must gain 0.05 over the prompt baseline; `syntax_pass` and `inst_level_strict_acc` must not regress; at least 120 tasks; split by `design`. **`testbench-generation`** trains the model to write the bench rather than the design. Its two metrics pull in opposite directions on purpose — a bench has to accept a correct design *and* reject a broken one, and `assert(1)` scores perfectly on the first alone. **`eda-script-assistant`** is the one with the least public data and the most private data. Its two packs are guardrails, not the measurement: what it is actually judged on is a private taskset of the team's own scripts, which cannot ship in a first-party solution and has to be declared by the deployment that installs it. `human_review` is load-bearing here — a script that runs is not a script that is right, since it can silently constrain the wrong clock. **`eda-log-triage`** involves no simulator and declares no toolchain, so it is the one workflow here a deployment with no EDA toolchain can still run. It splits by `case` rather than by `design`: the same failure reported twice is one fact. ## Read the protocol before you train ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp solution show rtl-repair ``` `show` prints what the solution references, its data format, and then the protocol in the order somebody has to decide it: ```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} baseline prompt pass@1 at least +0.05 syntax_pass must not regress inst_level_strict_acc must not regress minimum tasks 120 split by design human review required ``` Freeze that before the first run. The whole value of the field is that it was written down first. ## Scaffold it into your project ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp solution init rtl-repair --into ./experiments tp solution init rtl-repair --force # overwrite an existing directory ``` `init` copies the solution's `template/` directory to `/` and stops. It does not submit, it does not configure the deployment, and it writes nothing the platform then owns. The files land in your repository and are yours to edit; nothing is uploaded, and the template runs in your container at the trust level of anything else you wrote. An existing target directory is refused rather than merged — a half-overwritten scaffold is worse than either outcome — so `--force` is how you say you meant it. > [!NOTE] > > All four built-in solutions include starting templates. These are files to adapt in your own > project, not complete experiments. The console checks whether a template directory exists on > the server; your CLI must have the same solution installed to initialize it. ## Whether this deployment can run one A name is only worth publishing if somebody checks that it resolves, and that check is the one thing the SDK cannot do — it knows what the manifest says and nothing about which of those exist here. Two read-only endpoints do it: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} curl -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN" \ https://tuneplane.your-company.com/api/solutions curl -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN" \ https://tuneplane.your-company.com/api/solutions/rtl-repair ``` The list gives `ready` and the references that did not resolve, named rather than counted — the next thing somebody does is install exactly those. The detail adds every reference with its kind and a note, plus the acceptance protocol as the solution declared it. `ready` means **every** name resolves. A solution one benchmark pack short is not "mostly ready": the acceptance protocol names that pack, so a run without it is measured against something else. Both routes read. They list the Solutions the server's SDK ships plus any in the server's own `TUNEPLANE_SOLUTION_PATH`; neither creates one. > [!NOTE] > > Toolchain references resolve against registered Toolchain Profiles, including whether they are > enabled. Configuring a sandbox image alone does not satisfy a named toolchain dependency. ## Use the console Open **Assets → Solutions**, then **View solution**. Each detail view shows the data format, input requirements, referenced assets, acceptance protocol and copyable CLI commands. Built-in summaries are translated; team-authored text keeps its original language. Team solutions use namespaced identities such as `our-team/our-rtl-flow`, including in detail API requests. **Dependencies resolved** means only that the declared references resolve on this deployment. It does not certify model or data availability, project permissions, compute capacity or a successful run. Unresolved dependencies can be missing, disabled or temporarily unreadable. The asset catalog links and toolchain guidance explain where to investigate. The log-triage and script-assistant workflows need private business evaluation tasks and scoring criteria in addition to their public guardrail benchmarks. A guardrail passing does not establish that a root cause or script is correct. Use **Get started** to inspect the protocol and copy the template initialization command. When no template exists, the page says so. For all four built-in solutions, **Apply to a project** saves an immutable application in your own project namespace. Bind a pinned base model, separate prepared training and validation datasets, a private sealed Harbor taskset (`owner/name@version`, at least the protocol sample minimum; RTL repair requires 120), and a readable business rubric (`owner/name@revision`). Model and dataset references require a platform version or a full upstream commit SHA. This checks declared inputs, not semantic data leakage. **My project applications** reopens these snapshots. Each application retains the protocol, recipe digest, input references, taskset digest and rubric revision digest. Changing inputs requires another application. Applications are creator-owned; project access and current rubric and taskset access are checked again before accepting a bound job. * **Submit baseline job** uses your evaluator source or image. Supply a `custom/custom` JobSpec with the bound base model and `spec.data.taskset`, without training data. Independent taskset delivery currently needs the local container backend. Your evaluator owns inference and rubric scoring; the platform does not supply log-triage evaluation code. * **Configure candidate training** prefills the frozen recipe, model and dataset references. Set resources and parameters, resolve a preview, then explicitly confirm submission. Candidate training does not receive the evaluation taskset. * All submission paths use the existing immutable submission package and confirmation receipt. The package includes `.tuneplane/solution.json`; the accepted job and application keep the same provenance. Replaying the same confirmation does not create another job. Associated jobs link to execution logs and outputs. **Candidate evaluation** selects a completed `hf_export` or `merged_model` from a successful candidate training job (or its export step). Use your own evaluator and the same bound base model/taskset, with `spec.model.init_from` set to that artifact's exact reference. A different artifact or execution attempt is refused. EDA applications additionally freeze an enabled toolchain profile and require its digest-pinned image for both evaluation jobs. This does not certify the evaluator or enforce sandbox profile network/resource policy inside a custom job. ## Acceptance evidence Select a baseline and a candidate evaluation, then choose **Compare current evidence**. Comparison is an on-demand snapshot, scoped to each job's current execution attempt. It requires: * A `business` round covering every frozen private evaluation task, with the rubric and taskset digests, `split="eval"`, the actual checkpoint reference, and no unscored, infrastructure-error or training-eligible trials. Business success rate must not regress. * Every metric in `snapshot.requirements`, including its round, direction and native units. Ratio values and deltas are 0–1, never percentages. A metric matching multiple declared packs is required in each matching round. An unmatched metric is reported in the business round in native units. Required benchmark rounds must have matching dataset, task cohort, verifier and settings digests on both sides, and complete scored coverage of their declared population. * A review of that exact evidence digest. Review identity, findings and the evidence snapshot are retained. Approval cannot override a failed metric or incomplete evidence. New attempts, revised trial outcomes or revoked assets invalidate the old review when the comparison refreshes. The evaluator reports facts through the existing authenticated observation channel. Add reporting to **your evaluator**, after real inference and scoring; the following is instrumentation, not a runnable evaluator (`task_id`, `actual_status` and `actual_score` come from your code): ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} import hashlib import json from pathlib import Path from tuneplane import report binding = json.loads(Path(".tuneplane/solution.json").read_text(encoding="utf-8")) snapshot = binding["snapshot"] checkpoint = (binding["artifact"]["ref"] if binding["role"] == "candidate_evaluation" else snapshot["bindings"]["model"]) report.validation_population("business", len(snapshot["business_task_ids"])) report.validation_protocol( "business", dataset_digest=snapshot["taskset_digest"], cohort_digest=snapshot["business_cohort_digest"], verifier_digest=snapshot["rubric_digest"], settings_digest=hashlib.sha256(b"your-exact-evaluator-settings").hexdigest(), split="eval", checkpoint=checkpoint, ) # After your evaluator has actually classified each task: report.validation_outcome("business", task_id, status=actual_status, training_eligible=False) # After your external benchmark harness has computed this metric: report.validation_metric("ifeval", "inst_level_strict_acc", actual_score, unit="ratio") ``` For each required benchmark round, also report its population, protocol and every classified trial, then its required metrics. The cohort digest is SHA-256 of the sorted task ID list encoded as UTF-8 JSON with `ensure_ascii=False` and `separators=(",", ":")`. Both evaluators must use the same settings digest. A round/metric key is immutable within an execution attempt; rerun evaluation to correct it. Trial retries use increasing `trial_attempt` values. At most 10,000 trials per round are compared; a larger population remains insufficient rather than silently using a subset. **Export evidence JSON** downloads the current comparison and review history (latest 100 entries). Older applications without frozen requirements need a new application. Results remain declarations from the authenticated producer: these checks establish comparable, complete reported evidence, not independent verification of benchmark execution or semantic train/evaluation separation. ## Write your own A Solution of your own is a directory rather than a published plugin. The platform only parses it, so it needs no registry: put it in a directory named by `TUNEPLANE_SOLUTION_PATH`, laid out `//`, on every machine and server that should see it — a shared mount or a checkout of your team's repository. ```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} our-team/our-rtl-flow/ ├── solution.yaml # the manifest above ├── template/ # optional: the starting point `tp solution init` copies └── README.md ``` ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} export TUNEPLANE_SOLUTION_PATH=/shared/solutions # holds our-team/our-rtl-flow/ tp solution list # now in the catalog as our-team/our-rtl-flow ``` Two rules, checked when the catalog loads: * **The directory name must equal `name`.** A directory called `rtl-flow` holding a manifest that says `name: our-rtl-flow` does not load. * **A built-in cannot be shadowed.** Built-ins load first and the first writer wins, so a directory named `rtl-repair` does not replace the shipped one. Two people running "the same" solution and getting different protocols is worse for an *acceptance* protocol than for a recipe, because the thing that differs is the definition of the result. A Solution from `TUNEPLANE_SOLUTION_PATH` is referenced as `/`; built-ins keep the bare name. Several directories are separated by `:` on Linux and macOS. ## What a Solution does not do Be clear about the boundary before you rely on it. * **Acceptance evidence is not a release gate.** The console compares reported results and records human review. It does not block promotion or deployment. Enforcement still belongs to `spec.evaluation.gates` — see [running benchmarks](https://docs.tuneplane.ai/en/guides/benchmarks.md). * **`split_by` is a statement of intent.** It names the field your split should honour. The platform does not compute the leak for you at submission — see [data shapes and splits](https://docs.tuneplane.ai/en/guides/data-formats.md) for what is and is not wired. * **The console orchestrates explicit submissions.** It does not supply inference/scoring code or automatically run the whole workflow. * **Nothing here has been scored end to end on real hardware.** The hardware-facing solutions point at a toolchain image that has never been built in CI and evaluators no runner in this project can execute; [RTL benchmarks](https://docs.tuneplane.ai/en/guides/rtl-benchmarks.md) says exactly what stayed unproven. ## Next [`tp solution` reference](https://docs.tuneplane.ai/en/cli/solution) · [RTL benchmarks](https://docs.tuneplane.ai/en/guides/rtl-benchmarks.md) · [Data shapes and splits](https://docs.tuneplane.ai/en/guides/data-formats.md) · [Benchmark packs](https://docs.tuneplane.ai/en/extend/benchmark-packs.md) · [Plugins](https://docs.tuneplane.ai/en/extend/plugins.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.