Score hardware and RTL work

tp bench run my-bench -m run:run-4f2a91 --suites verilogeval-v2,rtllm-v2

GSM8K judges whether an answer is right. VerilogEval judges whether the generated circuit compiles and passes its testbench. That needs a simulator and pass@k over repeated sampling, so the platform has a third runner for it, rtl, alongside lm-eval and evalscope. Scores land in the same dashboard, so an RTL score and a GSM8K score sit on one row for the same model.

What you can run

Benchmark	What it measures	Primary metric
`verilogeval-v2`	Natural-language spec to Verilog module	`pass@1`
`verilogeval-v2-completion`	Completing an implementation from the module header	`pass@1`
`rtllm-v2`	Design-level tasks — ALU, FIFO, pipeline units — synthesisable	`func_pass`
`rtl-repo`	Completion inside a real repository: long context and project conventions	`exact_match`
`cvdp`	NVIDIA’s comprehensive design problems, non-agentic track	`pass@1`

One evaluation uses one runner, so RTL benchmarks and GSM8K-style ones go in separate jobs.

rtl-repo’s metric is not comparable to the others. It runs no simulation, because there is no testbench, and reports text similarity. Ranking its exact_match next to VerilogEval’s pass@1 compares two different things.

Check it worked

tp job logs

The log shows the simulator compiling each candidate. On the Benchmarks page your run gets a row with pass@1 in the requested columns. A job that fails with the RTL harness produced no rtl_report.json means the image the evaluation ran in has no harness where TUNEPLANE_RTL_HARNESS points. That is the next section.

What the image needs (an administrator does this once)

The simulators are in deploy/docker/Dockerfile.evalkit: iverilog and verilator. That image is a GPU image — it carries torch, lm-eval and evalscope, and it exists to generate. The evaluators, the formal tools and the pinned upstream problem sets live in a second image, deploy/docker/Dockerfile.eda-oss, which carries a toolchain and no model runtime and exists to check what was generated. Keeping them apart is not tidiness: an offline benchmark step can release its GPUs before the checking starts, and one image would force the cards to be held for the duration of a simulation that cannot use them. It also lets the EDA toolchain be pinned on its own cadence — a simulator version is part of what a result means, and upgrading one because a CUDA wheel moved would silently change every hardware score in the archive.

What is in the eda-oss image

From ubuntu:24.04, with the distribution’s own versions:

Tool	Why it is there
`iverilog`	Event-driven Verilog simulation. What VerilogEval, RTLLM and CVDP all drive
`verilator`	Cycle-accurate Verilog/SystemVerilog, and the linter most RTL teams already run. A different question from iverilog’s compile: it catches width mismatches and inferred latches that elaborate perfectly well, so the two are separate stages
`yosys`	Synthesis. A design that simulates and will not synthesise is a common failure of generated RTL and is invisible to a testbench
`yices2`, SymbiYosys (`sby`)	Bounded model checking with a solver behind it. Formal answers “for all inputs” where a testbench answers “for these”
`cocotb` 1.9.2, `pytest` 8.3.4	Driving a simulator from Python, which is how a testbench written by a verification engineer usually reaches a design now

Plus VerilogEval at /opt/verilog-eval and RTLLM at /opt/rtllm, cloned at pinned commits and never vendored into this repository — a copy of somebody’s problem set becomes a different benchmark the moment they edit a problem. Three things it deliberately does not carry:

No vendor tools. Synopsys, Cadence and Siemens flows need licences this image cannot hold and a licence server it must not reach on its own. A deployment that has them builds its own image FROM this one and registers it as its own Toolchain Profile. What this image is, is the floor: the open-source flow that makes the public benchmarks reproducible without anybody’s entitlement.
No model. Generation happens in the evalkit image or on a Model Deployment; this image receives RTL and returns verdicts.
No tuneplane install. The Job Capsule provides the SDK at run time, the same arrangement evalkit uses, so the contract and the adapter that speaks it cannot drift.

Build it with the three commits pinned:

docker build -f deploy/docker/Dockerfile.eda-oss \
  --build-arg SBY_COMMIT=<commit> \
  --build-arg VERILOG_EVAL_COMMIT=<commit> \
  --build-arg RTLLM_COMMIT=<commit> \
  -t your-registry/tuneplane-eda-oss:1 .

None of the three build arguments has a default, and the build fails at that line without them. That is deliberate — an unpinned build is a benchmark that changes under whoever runs it next — but it means there is no docker build of this file that works out of the box, and this repository records no commit values for you to copy. Choose them, and record them beside the image digest.

CVDP and RTL-Repo are not cloned by this Dockerfile. Their suites read TUNEPLANE_CVDP_DIR and TUNEPLANE_RTL_REPO_DIR, defaulting to /opt/cvdp and /opt/rtl-repo; an image without those directories refuses those suites rather than scoring them.

The harness is an adapter that scores nothing

The platform still defines only the contract:

$TUNEPLANE_RTL_HARNESS --benchmark <name> --model <path> --output-dir <dir> \
                   [--samples N] [--limit N]

What changed is that TUNEPLANE_RTL_HARNESS now has a default, tuneplane-rtl-eval, a console script the SDK ships. It is an adapter, not a benchmark. It loads a suite’s problems from the upstream evaluator’s own JSONL files, asks the model for samples, hands each sample to that evaluator, and turns what comes back into the report contract. Every judgement about whether a circuit is correct is made by the upstream tool, in the upstream’s own way.

Pack	Suite it selects	Where the evaluator must be
`verilogeval-v2`	`verilogeval-v2-spec`	`TUNEPLANE_VERILOG_EVAL_DIR`, default `/opt/verilog-eval`
`verilogeval-v2-completion`	`verilogeval-v2-completion`	`TUNEPLANE_VERILOG_EVAL_DIR`, default `/opt/verilog-eval`
`rtllm-v2`	`rtllm-v2`	`TUNEPLANE_RTLLM_DIR`, default `/opt/rtllm`
`cvdp`	`cvdp-nonagentic`	`TUNEPLANE_CVDP_DIR`, default `/opt/cvdp`
`rtl-repo`	`rtl-repo`	`TUNEPLANE_RTL_REPO_DIR`, default `/opt/rtl-repo`

An evaluator that is not installed at all raises and names the variable to set, rather than being reported as a tool failure: a missing evaluator is a deployment error an operator has to see, where a simulator that crashed on one design is a per-task fact. TUNEPLANE_RTL_TIMEOUT_S bounds one simulation — 600 seconds by default, with a floor of 10 — because a testbench’s runtime is the customer’s business, not the adapter’s.

The adapter reports pass@1, pass@<samples> and tool_failures per suite. The func_pass and exact_match metrics that the rtllm-v2 and rtl-repo packs declare are not among them; a deployment that wants those numbers needs a harness that computes them and points TUNEPLANE_RTL_HARNESS at it.

What the report carries

The adapter writes <output-dir>/rtl_report.json. Version 2 adds one verification report per task beside the scores; version 1, {"results": {…}} alone, still parses.

{
  "schema": "tuneplane/rtl-report/v1",
  "results": {"verilogeval-v2-spec": {"pass@1": 0.42, "pass@5": 0.61, "tool_failures": 0.0}},
  "verifications": [
    {
      "task_id": "counter",
      "final": "pass",
      "complete": true,
      "tool_failures": 0,
      "stages": [
        {"stage": "simulate", "tool": "iverilog", "tool_version": "Icarus Verilog version 12.0",
         "tool_ok": true, "verdict": "pass", "log_digest": "3f2a…", "duration_s": 1.4}
      ]
    }
  ]
}

A stage names one tool that ran and what it concluded. The set is closed — lint, compile, simulate, formal, synthesize, coverage, and custom for a deployment’s own flow — and ordered as a flow runs them, cheapest first, because a design that does not lint has no simulation result worth waiting for. There is no stage for “the model’s answer looked plausible”: a judgement with no tool behind it is a Rubric, and rubrics have their own path.

Field	What it is
`tool`, `tool_version`	Required rather than optional. A result whose tool nobody recorded cannot be reproduced six months later, and an evaluation that cannot be reproduced is a number rather than evidence
`tool_ok`	Did the tool run to completion? `false` means every other field is about the tool, not about the design
`verdict`	`pass`, `fail`, or `unknown`
`metrics`	Free-form numbers this stage measured: cells, slack, coverage. What a synthesis flow reports is the flow’s business
`log_digest`	A digest of the tool’s output, not the output. EDA logs run to megabytes and routinely quote the design

Two shapes are refused when a stage is constructed, because each would be a lie in one direction: a stage that says its tool ran but gives no verdict, and a stage that carries a pass or fail verdict from a tool that did not run.

A tool that did not run is not a design that failed

Every stage can end three ways rather than two:

Outcome	What it is
The tool ran and the design passed	A result
The tool ran and the design failed	A result
The tool did not run	Not a result at all

A licence server that was down, a simulator that segfaulted, a container without iverilog on its PATH: none of those is evidence about the design. Scoring them as a failure is the single most damaging thing an evaluation of generated engineering work can do. In training it teaches the model to avoid whatever it was doing when the licence queue was busy; in a report it is a claim about a circuit nobody checked. So the rule runs all the way through:

A report’s verdict is derived, never declared. No stages, or any tool failure anywhere, and the whole report is unknown — even when every stage that did run passed. A design that compiles and was never simulated has not been shown to work, and reporting the compile as the answer is how “it builds” becomes “it works”.
A report with any tool failure refuses to become a reward. as_reward() raises and names the broken stages rather than returning 0.0. A corpus that scores tool failures as zero teaches a policy to avoid whatever it was doing when the tool broke, and the model has no way to tell that apart from the circuit being wrong.
tool_failures travels as its own count. It is reported beside the pass rate rather than folded into it, the same way rows_scanned travels with every data-scan count. A suite with tool failures reads as incomplete, not as lower-scoring.
pass_rate is null when nothing could be scored, never 0.0. An empty denominator is not a pass rate of nought, and that difference is what stops an image with no simulator from publishing a perfect record of failure.

The adapter applies the same rule to what it cannot do: a missing simulator becomes one stage with tool_ok: false, a simulation that times out becomes a simulate stage with tool_ok: false (a testbench with no $finish hangs on a correct module too), and only a non-zero exit from the evaluator becomes a fail.

What is not proven

This section is longer than it would be if the answer were shorter.

Dockerfile.eda-oss has never been built in CI. The release workflow builds the console image and nothing else. scripts/image_smoke.py carries checks for this image — every simulator and formal tool on PATH, cocotb importable, the upstream problem sets cloned — and no workflow runs them.
The build cannot succeed unpinned. SBY_COMMIT, VERILOG_EVAL_COMMIT and RTLLM_COMMIT have no defaults and the build fails at the line that checks them, so there is no known-good invocation to copy out of this repository.
The adapter’s call into the upstream evaluator has never run against a real checkout. Its tests stub the subprocess, so loading a suite, building prompts, assembling stages, writing the report and every failure branch are covered, and the actual invocation of VerilogEval’s or RTLLM’s evaluator is not.
Generation is a GPU path. The one function that calls vLLM is isolated on purpose and runs in no CI, because no runner this project uses has a GPU.
No score in this repository has been produced against a real simulator or a real GPU. Every number in the tests came from a stub.
The per-task verifications block is written but not yet read. The SDK can parse it back out of a report; nothing in the control plane or the console consumes it, so the stages do not appear on any page.

CVDP’s agentic track runs outside the scheduler

CVDP has two tracks. The non-agentic one is everything above. The agentic track has the model use tools repeatedly while solving: read files, run simulation, inspect waveforms, edit, retry. Two properties put it outside a training job:

It launches containers. The official harness pulls a Docker container per problem. An evaluation job here already runs inside a container, and Docker-in-Docker on KubeRay or Slurm needs a privileged container or sysbox, which most internal clusters do not grant.
It needs a callable endpoint, not a weights path.

The platform supplies the second. Deploy the version, then run the harness on a machine that has Docker, then report the scores back.

# 1. On the platform: open an external evaluation and take the run id
tp bench external create --model alice/qa@v3 --harness cvdp --train-run run-4f2a91

# 2. On the evaluation host: point the agent at the deployment
export OPENAI_BASE_URL=https://tuneplane.your-company.com/inference/<deployment-id>/v1
export OPENAI_API_KEY=<deployment token>
export CVDP_AGENT_ENV=OPENAI_BASE_URL,OPENAI_API_KEY
./run_benchmark.py -f <dataset> -l -g <your agent image> -p work/

# 3. Report the scores back
tp bench external submit run-9c3e10 -f scores.json --harness cvdp

--train-run is what makes the evaluation gate and the model registry attribute these scores to that training run. Without it they are an isolated number. Two things about that flow:

The ingestion entry point accepts structured scores only and does not parse harness report text. CVDP emits report.txt and composite_report.txt, so keep the conversion in your own script. Then an upstream formatting change is a change to your script rather than a broken ingestion path.
The deployment token carries one permission, reporting scores. It gets pasted into an evaluation host’s environment, so it must not also be able to write logs, change job state or upload artifacts.

Benchmarks · Benchmark packs · Solutions · Model deployments

Get started

Core concepts

Train a model

Evaluate and serve

Agent environments

Extend the platform

Collaborate

Integrations

Reference

Help

Score hardware and RTL work

What you can run

Check it worked

What the image needs (an administrator does this once)

What is in the eda-oss image

The harness is an adapter that scores nothing

What the report carries

A tool that did not run is not a design that failed

What is not proven

CVDP’s agentic track runs outside the scheduler

Next

​What you can run

​Check it worked

​What the image needs (an administrator does this once)

​What is in the eda-oss image

​The harness is an adapter that scores nothing

​What the report carries

​A tool that did not run is not a design that failed

​What is not proven

​CVDP’s agentic track runs outside the scheduler

​Next

What you can run

Check it worked

What the image needs (an administrator does this once)

What is in the eda-oss image

The harness is an adapter that scores nothing

What the report carries

A tool that did not run is not a design that failed

What is not proven

CVDP’s agentic track runs outside the scheduler

Next