Skip to main content
A real tp submit, real assembly and scheduling, a container that actually trains, real ingest of metrics, logs and artifacts, and assertions on SUCCEEDED. The Node loop uses real CPU resource grants and two separately enrolled Node daemons. The Kubernetes loop simulates GPU attachment. Neither proves physical GPU execution. Run it before go-live, after an upgrade, and after any change to a backend. It needs a checkout of the platform repository, so this is a step for whoever operates the deployment rather than for somebody using a released console.

​
Run it

Build the Node test image and configure PostgreSQL as described in the repository’s docs/ops/e2e-closed-loop.md. The Node loop retains its logs and cleans up its containers. The Kubernetes harness accepts --keep to preserve its resources. Both use the repository SDK CLI and create an independent temporary project with tp init before submitting.

​
The GPU simulation switches

Scheduling, quotas, capacity, and the ledger all execute with real semantics. Production keeps the default (passthrough).

​
What it asserts

  1. The state machine reaches SUCCEEDED (passing through PENDING / RUNNING);
  2. Metrics closed loop: train/loss can be read back from the API;
  3. Logs closed loop: training log lines are stored and queryable;
  4. Artifacts closed loop: checkpoint registration is visible.

​
CI

.github/workflows/e2e.yml: unit tests → Node closed loop (sim + real SFT matrix) and Kubernetes closed loop (kind + operator), with no cross-repository token.

​
What it does not cover

The slurm backend is not yet in the closed loop (requires containerized slurmrestd + Apptainer): batch script rendering, the REST client, hetjob, and status mapping are covered by unit tests. Details and roadmap in the repo’s docs/ops/e2e-closed-loop.md.