> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Smoke-test a deployment

> Run submit, schedule, train and ingest end to end, with no GPU.

A real `tp submit`, real assembly and scheduling, a container that actually trains, real ingest of
metrics, logs and artifacts, and assertions on `SUCCEEDED`. The Node loop uses real CPU resource grants and two separately enrolled Node daemons.
The Kubernetes loop simulates GPU attachment. Neither proves physical GPU execution.

Run it before go-live, after an upgrade, and after any change to a backend.

It needs a checkout of the platform repository, so this is a step for whoever operates the
deployment rather than for somebody using a released console.

## Run it

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# Node executor: simulated training workload (~10 seconds)
uv run python scripts/e2e/node_loop.py --evidence /tmp/node-sim.json

# Node executor: real TRL SFT (actual training of a tiny model on CPU)
uv run python scripts/e2e/node_loop.py --flavor sft --evidence /tmp/node-sft.json

# kubernetes executor: kind cluster + operator + MinIO distribution (~40 seconds)
bash scripts/e2e/setup-kind.sh
uv run python scripts/e2e/kuberay_loop.py
```

Build the Node test image and configure PostgreSQL as described in the repository's
`docs/ops/e2e-closed-loop.md`. The Node loop retains its logs and cleans up its containers.
The Kubernetes harness accepts `--keep` to preserve its resources. Both use the repository SDK CLI
and create an independent temporary project with `tp init` before submitting.

## The GPU simulation switches

| Setting | Semantics |
| - | - |
| `TUNEPLANE_K8S_GPU_PASSTHROUGH=0` | kubernetes: Pods do not request `nvidia.com/gpu`, Ray logical resources work as usual |

Scheduling, quotas, capacity, and the ledger all execute with real semantics. **Production keeps the default (passthrough).**

## What it asserts

1. The state machine reaches SUCCEEDED (passing through PENDING / RUNNING);
2. Metrics closed loop: `train/loss` can be read back from the API;
3. Logs closed loop: training log lines are stored and queryable;
4. Artifacts closed loop: checkpoint registration is visible.

## CI

`.github/workflows/e2e.yml`: unit tests → Node closed loop (sim + real SFT matrix) and Kubernetes closed loop (kind + operator), with no cross-repository token.

## What it does not cover

The slurm backend is not yet in the closed loop (requires containerized slurmrestd + Apptainer): batch script rendering, the REST client, hetjob, and status mapping are covered by unit tests. Details and roadmap in the repo's `docs/ops/e2e-closed-loop.md`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.