> ## Documentation Index > Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt > Use this file to discover all available pages before exploring further. # Score hardware and RTL work > VerilogEval v2, RTLLM v2, RTL-Repo and CVDP: compile the Verilog and run the testbench. ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp bench run my-bench -m run:run-4f2a91 --suites verilogeval-v2,rtllm-v2 ``` GSM8K judges whether an answer is right. VerilogEval judges whether the generated circuit compiles and passes its testbench. That needs a simulator and pass\@k over repeated sampling, so the platform has a third runner for it, `rtl`, alongside lm-eval and evalscope. Scores land in the same dashboard, so an RTL score and a GSM8K score sit on one row for the same model. ## What you can run | Benchmark | What it measures | Primary metric | | - | - | - | | `verilogeval-v2` | Natural-language spec to Verilog module | `pass@1` | | `verilogeval-v2-completion` | Completing an implementation from the module header | `pass@1` | | `rtllm-v2` | Design-level tasks — ALU, FIFO, pipeline units — synthesisable | `func_pass` | | `rtl-repo` | Completion inside a real repository: long context and project conventions | `exact_match` | | `cvdp` | NVIDIA's comprehensive design problems, non-agentic track | `pass@1` | One evaluation uses one runner, so RTL benchmarks and GSM8K-style ones go in separate jobs. > [!NOTE] > > **`rtl-repo`'s metric is not comparable to the others.** It runs no simulation, because there is > no testbench, and reports text similarity. Ranking its `exact_match` next to VerilogEval's > `pass@1` compares two different things. ## Check it worked ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp job logs ``` The log shows the simulator compiling each candidate. On the Benchmarks page your run gets a row with `pass@1` in the requested columns. A job that fails with `the RTL harness produced no rtl_report.json` means the image the evaluation ran in has no harness where `TUNEPLANE_RTL_HARNESS` points. That is the next section. ## What the image needs (an administrator does this once) The simulators are in `deploy/docker/Dockerfile.evalkit`: `iverilog` and `verilator`. That image is a GPU image — it carries torch, lm-eval and evalscope, and it exists to **generate**. The evaluators, the formal tools and the pinned upstream problem sets live in a second image, `deploy/docker/Dockerfile.eda-oss`, which carries a toolchain and no model runtime and exists to **check** what was generated. Keeping them apart is not tidiness: an offline benchmark step can release its GPUs before the checking starts, and one image would force the cards to be held for the duration of a simulation that cannot use them. It also lets the EDA toolchain be pinned on its own cadence — a simulator version is part of what a result means, and upgrading one because a CUDA wheel moved would silently change every hardware score in the archive. ### What is in the eda-oss image From `ubuntu:24.04`, with the distribution's own versions: | Tool | Why it is there | | - | - | | `iverilog` | Event-driven Verilog simulation. What VerilogEval, RTLLM and CVDP all drive | | `verilator` | Cycle-accurate Verilog/SystemVerilog, and the linter most RTL teams already run. A different question from iverilog's compile: it catches width mismatches and inferred latches that elaborate perfectly well, so the two are separate stages | | `yosys` | Synthesis. A design that simulates and will not synthesise is a common failure of generated RTL and is invisible to a testbench | | `yices2`, SymbiYosys (`sby`) | Bounded model checking with a solver behind it. Formal answers "for all inputs" where a testbench answers "for these" | | `cocotb` 1.9.2, `pytest` 8.3.4 | Driving a simulator from Python, which is how a testbench written by a verification engineer usually reaches a design now | Plus VerilogEval at `/opt/verilog-eval` and RTLLM at `/opt/rtllm`, cloned at pinned commits and never vendored into this repository — a copy of somebody's problem set becomes a different benchmark the moment they edit a problem. Three things it deliberately does not carry: * **No vendor tools.** Synopsys, Cadence and Siemens flows need licences this image cannot hold and a licence server it must not reach on its own. A deployment that has them builds its own image `FROM` this one and registers it as its own Toolchain Profile. What this image is, is the floor: the open-source flow that makes the public benchmarks reproducible without anybody's entitlement. * **No model.** Generation happens in the evalkit image or on a Model Deployment; this image receives RTL and returns verdicts. * **No `tuneplane` install.** The Job Capsule provides the SDK at run time, the same arrangement evalkit uses, so the contract and the adapter that speaks it cannot drift. Build it with the three commits pinned: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} docker build -f deploy/docker/Dockerfile.eda-oss \ --build-arg SBY_COMMIT= \ --build-arg VERILOG_EVAL_COMMIT= \ --build-arg RTLLM_COMMIT= \ -t your-registry/tuneplane-eda-oss:1 . ``` > [!WARNING] > > **None of the three build arguments has a default, and the build fails at that line without > them.** That is deliberate — an unpinned build is a benchmark that changes under whoever runs it > next — but it means there is no `docker build` of this file that works out of the box, and this > repository records no commit values for you to copy. Choose them, and record them beside the > image digest. CVDP and RTL-Repo are **not** cloned by this Dockerfile. Their suites read `TUNEPLANE_CVDP_DIR` and `TUNEPLANE_RTL_REPO_DIR`, defaulting to `/opt/cvdp` and `/opt/rtl-repo`; an image without those directories refuses those suites rather than scoring them. ### The harness is an adapter that scores nothing The platform still defines only the contract: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} $TUNEPLANE_RTL_HARNESS --benchmark --model --output-dir \ [--samples N] [--limit N] ``` What changed is that `TUNEPLANE_RTL_HARNESS` now has a default, `tuneplane-rtl-eval`, a console script the SDK ships. It is an **adapter**, not a benchmark. It loads a suite's problems from the upstream evaluator's own JSONL files, asks the model for samples, hands each sample to that evaluator, and turns what comes back into the report contract. Every judgement about whether a circuit is correct is made by the upstream tool, in the upstream's own way. | Pack | Suite it selects | Where the evaluator must be | | - | - | - | | `verilogeval-v2` | `verilogeval-v2-spec` | `TUNEPLANE_VERILOG_EVAL_DIR`, default `/opt/verilog-eval` | | `verilogeval-v2-completion` | `verilogeval-v2-completion` | `TUNEPLANE_VERILOG_EVAL_DIR`, default `/opt/verilog-eval` | | `rtllm-v2` | `rtllm-v2` | `TUNEPLANE_RTLLM_DIR`, default `/opt/rtllm` | | `cvdp` | `cvdp-nonagentic` | `TUNEPLANE_CVDP_DIR`, default `/opt/cvdp` | | `rtl-repo` | `rtl-repo` | `TUNEPLANE_RTL_REPO_DIR`, default `/opt/rtl-repo` | An evaluator that is not installed at all raises and names the variable to set, rather than being reported as a tool failure: a missing evaluator is a deployment error an operator has to see, where a simulator that crashed on one design is a per-task fact. `TUNEPLANE_RTL_TIMEOUT_S` bounds one simulation — 600 seconds by default, with a floor of 10 — because a testbench's runtime is the customer's business, not the adapter's. > [!NOTE] > > **The adapter reports `pass@1`, `pass@` and `tool_failures` per suite.** The `func_pass` > and `exact_match` metrics that the `rtllm-v2` and `rtl-repo` packs declare are not among them; a > deployment that wants those numbers needs a harness that computes them and points > `TUNEPLANE_RTL_HARNESS` at it. ### What the report carries The adapter writes `/rtl_report.json`. Version 2 adds one verification report per task beside the scores; version 1, `{"results": {…}}` alone, still parses. ```json theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} { "schema": "tuneplane/rtl-report/v1", "results": {"verilogeval-v2-spec": {"pass@1": 0.42, "pass@5": 0.61, "tool_failures": 0.0}}, "verifications": [ { "task_id": "counter", "final": "pass", "complete": true, "tool_failures": 0, "stages": [ {"stage": "simulate", "tool": "iverilog", "tool_version": "Icarus Verilog version 12.0", "tool_ok": true, "verdict": "pass", "log_digest": "3f2a…", "duration_s": 1.4} ] } ] } ``` A stage names one tool that ran and what it concluded. The set is closed — `lint`, `compile`, `simulate`, `formal`, `synthesize`, `coverage`, and `custom` for a deployment's own flow — and ordered as a flow runs them, cheapest first, because a design that does not lint has no simulation result worth waiting for. There is no stage for "the model's answer looked plausible": a judgement with no tool behind it is a [Rubric](https://docs.tuneplane.ai/en/extend/rubrics.md), and rubrics have their own path. | Field | What it is | | - | - | | `tool`, `tool_version` | Required rather than optional. A result whose tool nobody recorded cannot be reproduced six months later, and an evaluation that cannot be reproduced is a number rather than evidence | | `tool_ok` | Did the tool run to completion? `false` means every other field is about the *tool*, not about the design | | `verdict` | `pass`, `fail`, or `unknown` | | `metrics` | Free-form numbers this stage measured: cells, slack, coverage. What a synthesis flow reports is the flow's business | | `log_digest` | A digest of the tool's output, not the output. EDA logs run to megabytes and routinely quote the design | Two shapes are refused when a stage is constructed, because each would be a lie in one direction: a stage that says its tool ran but gives no verdict, and a stage that carries a `pass` or `fail` verdict from a tool that did not run. ### A tool that did not run is not a design that failed Every stage can end three ways rather than two: | Outcome | What it is | | - | - | | The tool ran and the design passed | A result | | The tool ran and the design failed | A result | | **The tool did not run** | Not a result at all | A licence server that was down, a simulator that segfaulted, a container without `iverilog` on its PATH: none of those is evidence about the design. Scoring them as a failure is the single most damaging thing an evaluation of generated engineering work can do. In training it teaches the model to avoid whatever it was doing when the licence queue was busy; in a report it is a claim about a circuit nobody checked. So the rule runs all the way through: * **A report's verdict is derived, never declared.** No stages, or any tool failure anywhere, and the whole report is `unknown` — even when every stage that did run passed. A design that compiles and was never simulated has not been shown to work, and reporting the compile as the answer is how "it builds" becomes "it works". * **A report with any tool failure refuses to become a reward.** `as_reward()` raises and names the broken stages rather than returning `0.0`. A corpus that scores tool failures as zero teaches a policy to avoid whatever it was doing when the tool broke, and the model has no way to tell that apart from the circuit being wrong. * **`tool_failures` travels as its own count.** It is reported beside the pass rate rather than folded into it, the same way `rows_scanned` travels with every data-scan count. A suite with tool failures reads as *incomplete*, not as lower-scoring. * **`pass_rate` is `null` when nothing could be scored**, never `0.0`. An empty denominator is not a pass rate of nought, and that difference is what stops an image with no simulator from publishing a perfect record of failure. The adapter applies the same rule to what it cannot do: a missing simulator becomes one stage with `tool_ok: false`, a simulation that times out becomes a `simulate` stage with `tool_ok: false` (a testbench with no `$finish` hangs on a correct module too), and only a non-zero exit from the evaluator becomes a `fail`. ## What is not proven This section is longer than it would be if the answer were shorter. * **`Dockerfile.eda-oss` has never been built in CI.** The release workflow builds the console image and nothing else. `scripts/image_smoke.py` carries checks for this image — every simulator and formal tool on PATH, `cocotb` importable, the upstream problem sets cloned — and no workflow runs them. * **The build cannot succeed unpinned.** `SBY_COMMIT`, `VERILOG_EVAL_COMMIT` and `RTLLM_COMMIT` have no defaults and the build fails at the line that checks them, so there is no known-good invocation to copy out of this repository. * **The adapter's call into the upstream evaluator has never run against a real checkout.** Its tests stub the subprocess, so loading a suite, building prompts, assembling stages, writing the report and every failure branch are covered, and the actual invocation of VerilogEval's or RTLLM's evaluator is not. * **Generation is a GPU path.** The one function that calls vLLM is isolated on purpose and runs in no CI, because no runner this project uses has a GPU. * **No score in this repository has been produced against a real simulator or a real GPU.** Every number in the tests came from a stub. * **The per-task `verifications` block is written but not yet read.** The SDK can parse it back out of a report; nothing in the control plane or the console consumes it, so the stages do not appear on any page. ## CVDP's agentic track runs outside the scheduler CVDP has two tracks. The non-agentic one is everything above. The agentic track has the model use tools repeatedly while solving: read files, run simulation, inspect waveforms, edit, retry. Two properties put it outside a training job: * **It launches containers.** The official harness pulls a Docker container per problem. An evaluation job here already runs inside a container, and Docker-in-Docker on KubeRay or Slurm needs a privileged container or sysbox, which most internal clusters do not grant. * **It needs a callable endpoint**, not a weights path. The platform supplies the second. Deploy the version, then run the harness on a machine that has Docker, then report the scores back. ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} # 1. On the platform: open an external evaluation and take the run id tp bench external create --model alice/qa@v3 --harness cvdp --train-run run-4f2a91 # 2. On the evaluation host: point the agent at the deployment export OPENAI_BASE_URL=https://tuneplane.your-company.com/inference//v1 export OPENAI_API_KEY= export CVDP_AGENT_ENV=OPENAI_BASE_URL,OPENAI_API_KEY ./run_benchmark.py -f -l -g -p work/ # 3. Report the scores back tp bench external submit run-9c3e10 -f scores.json --harness cvdp ``` `--train-run` is what makes the evaluation gate and the model registry attribute these scores to that training run. Without it they are an isolated number. Two things about that flow: * The ingestion entry point accepts **structured scores only** and does not parse harness report text. CVDP emits `report.txt` and `composite_report.txt`, so keep the conversion in your own script. Then an upstream formatting change is a change to your script rather than a broken ingestion path. * The deployment token carries one permission, reporting scores. It gets pasted into an evaluation host's environment, so it must not also be able to write logs, change job state or upload artifacts. ## Next [Benchmarks](https://docs.tuneplane.ai/en/guides/benchmarks.md) · [Benchmark packs](https://docs.tuneplane.ai/en/extend/benchmark-packs.md) · [Solutions](https://docs.tuneplane.ai/en/guides/solutions.md) · [Model deployments](https://docs.tuneplane.ai/en/guides/deployments.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.