Skip to main content
deploy/k8s/ is a kustomize overlay covering the namespace, config, secrets, RBAC, deployment, service, ingress and the two PVCs.

​
What it creates

Point the image at your registry before applying:
deploy/k8s/kustomization.yaml

​
The storage claim is the part that matters

ReadWriteMany is not optional, and RWO fails quietly. An RWO claim does not error — Kubernetes gives each pod its own volume. A multi-node job then splits its checkpoint shards across pods and the run cannot be resumed, with nothing in the logs to say why.The executor’s health check verifies TUNEPLANE_K8S_STORAGE_SHARED against the claim’s real accessModes. Read it after the first deploy rather than assuming.
The console mounts that claim at exactly the path TUNEPLANE_STORAGE_ROOT names — it stages job directories into runs/<user>/<exp>/<run_id>/work, measures disk pressure, and reclaims. Training pods mount the same claim, so a path the console writes is the path a job opens. Size it for weight cache plus dataset cache plus every run directory. 2Ti is a starting point, not a recommendation; the weight cache alone is tens of gigabytes per base model.
deploy/k8s/configmap.yaml
Four of those are worth understanding rather than copying:
Training pods report back to it. The external ingress address would leave the cluster and come back in; the service DNS name does not. Never 127.0.0.1 — that is the worker’s own loopback.
A mismatch presents as “the cluster came up but workers never registered”, and the only clue is a single version warning easily lost in startup output.
Ray’s object store lives in /dev/shm. At the default, anything but a tiny batch OOMs immediately. It is RAM-backed, so this plus the memory limit must stay under physical memory.
Without it, an unpullable image or a nodeSelector nothing satisfies leaves a job Pending forever while holding its queue slot.

​
Secrets

secret.example.yaml exists to show the shape. Replace it with a real source — kubectl create secret, Sealed Secrets, or the External Secrets Operator — and remove it from kustomization.yaml before production. At minimum: TUNEPLANE_WEB_JWT_SECRET, TUNEPLANE_DB_URL, TUNEPLANE_REDIS_URL, TUNEPLANE_S3_SECRET_KEY, TUNEPLANE_SECRET_ENC_KEY, and the OIDC client secret.

​
Replicas

Use the same PostgreSQL database, Redis 6.2 or later, fixed JWT secret and shared storage across all API and worker pods. Enable worker.enabled=true in Helm; it turns off API background loops and creates one scalable worker Deployment. Every worker runs every coordination loop (scheduling, reconcile, watchdog) under a leader lease, so exactly one replica acts per loop and the others stand by, and each worker adds consumer capacity. Set worker.replicas=3 for three workers. Kustomize includes worker.yaml; there is no separate controller.
Every worker uses the same command and automatically receives tasks. Independent run actions, diagnoses, Hub imports, and imports into different Volumes can run concurrently. GPU allocation stays with whichever worker holds the scheduler lease. Shared maintenance retains its coordination; adding replicas does not divide a single transfer. TUNEPLANE_WORKER_CONCURRENCY controls in-flight deliveries per process (default 4). TUNEPLANE_WORKER_IO_CONCURRENCY limits imports and Volume measurements across the entire pool (default 4). Set the same shared limit on every replica. Increasing replicas alone does not raise that storage budget. Account for database connections and storage bandwidth when raising it. Use storage that supports conditional writes. Browser upload CORS must allow If-None-Match alongside the existing upload headers. Upgrade API, consumers, SDK and console together: uploaded Volume files require publication before becoming visible. Managed reads use immutable generations; old mutable data directories are not supported by this protocol.

​
Check it worked

Then check the executor’s own view — it is what tells you whether the storage claim is really RWX: