> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Install on Kubernetes

> The console in-cluster with kustomize, and the RWX claim the KubeRay executor depends on.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
kubectl apply -k deploy/k8s/
kubectl -n tuneplane rollout status deploy/tuneplane-server
```

`deploy/k8s/` is a kustomize overlay covering the namespace, config, secrets, RBAC, deployment,
service, ingress and the two PVCs.

## What it creates

| Manifest | What it is for |
| - | - |
| `namespace.yaml` | `tuneplane` |
| `configmap.yaml` | Non-secret `TUNEPLANE_*` settings |
| `secret.example.yaml` | Secret settings — **replace the source before production** |
| `rbac.yaml` | The RayJob permissions the `kubernetes` executor needs. Without it, dispatch gets a 403 |
| `deployment.yaml` | One replica, probes on `/healthz`, `250m`–`2` CPU, `512Mi`–`2Gi` memory |
| `service.yaml`, `ingress.yaml` | In-cluster address and the external route |
| `pvc.yaml` | The shared storage-root claim |

Point the image at your registry before applying:

```yaml deploy/k8s/kustomization.yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
images:
  - name: registry.company.com/tuneplane/tuneplane-server
    newTag: "0.3.0"
```

## The storage claim is the part that matters

```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
accessModes: [ReadWriteMany]
resources:
  requests:
    storage: 2Ti
storageClassName: nfs-client     # must actually support RWX
```

> [!WARNING]
>
> **ReadWriteMany is not optional, and RWO fails quietly.** An RWO claim does not error — Kubernetes
> gives each pod its own volume. A multi-node job then splits its checkpoint shards across pods and
> the run cannot be resumed, with nothing in the logs to say why.
>
> The executor's health check verifies `TUNEPLANE_K8S_STORAGE_SHARED` against the claim's real
> `accessModes`. Read it after the first deploy rather than assuming.

The console mounts that claim at exactly the path `TUNEPLANE_STORAGE_ROOT` names — it stages job
directories into `runs/<user>/<exp>/<run_id>/work`, measures disk pressure, and reclaims. Training
pods mount the same claim, so a path the console writes is the path a job opens.

Size it for weight cache plus dataset cache plus every run directory. 2Ti is a starting point, not a
recommendation; the weight cache alone is tens of gigabytes per base model.

## Recommended configuration

```yaml deploy/k8s/configmap.yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
TUNEPLANE_STORAGE_ROOT: "/tuneplane"
TUNEPLANE_K8S_STORAGE_PVC: "tuneplane-storage"
TUNEPLANE_K8S_STORAGE_SHARED: "true"
TUNEPLANE_PUBLIC_URL: "https://tuneplane.company.com"
TUNEPLANE_INGEST_URL: "http://tuneplane-server.tuneplane.svc.cluster.local"
TUNEPLANE_KUBERAY_RAY_VERSION: "2.55.1"
TUNEPLANE_K8S_SHM_SIZE: "64Gi"
TUNEPLANE_KUBERAY_PRERUNNING_DEADLINE_S: "1800"
TUNEPLANE_JOB_RUNNER_MODE: "bundled"
TUNEPLANE_QUOTA_ENFORCE: "1"
```

Four of those are worth understanding rather than copying:

> **TUNEPLANE\_INGEST\_URL uses the in-cluster service name**
>
> Training pods report back to it. The external ingress address would leave the cluster and come
> back in; the service DNS name does not. Never `127.0.0.1` — that is the worker's own loopback.

> **TUNEPLANE\_KUBERAY\_RAY\_VERSION must match the training image**
>
> A mismatch presents as "the cluster came up but workers never registered", and the only clue is a
> single version warning easily lost in startup output.

> **TUNEPLANE\_K8S\_SHM\_SIZE, because 64MB is the container default**
>
> Ray's object store lives in `/dev/shm`. At the default, anything but a tiny batch OOMs
> immediately. It is RAM-backed, so this plus the memory limit must stay under physical memory.

> **TUNEPLANE\_KUBERAY\_PRERUNNING\_DEADLINE\_S bounds a stuck job**
>
> Without it, an unpullable image or a nodeSelector nothing satisfies leaves a job Pending forever
> while holding its queue slot.

## Secrets

`secret.example.yaml` exists to show the shape. Replace it with a real source — `kubectl create
secret`, Sealed Secrets, or the External Secrets Operator — and remove it from
`kustomization.yaml` before production.

At minimum: `TUNEPLANE_WEB_JWT_SECRET`, `TUNEPLANE_DB_URL`, `TUNEPLANE_REDIS_URL`, `TUNEPLANE_S3_SECRET_KEY`,
`TUNEPLANE_SECRET_ENC_KEY`, and the OIDC client secret.

## Replicas

Use the same PostgreSQL database, Redis 6.2 or later, fixed JWT secret and shared
storage across all API and worker pods. Enable `worker.enabled=true` in Helm;
it turns off API background loops and creates one scalable worker Deployment.
Every worker runs every coordination loop (scheduling, reconcile, watchdog)
under a leader lease, so exactly one replica acts per loop and the others stand
by, and each worker adds consumer capacity. Set `worker.replicas=3` for three
workers. Kustomize includes `worker.yaml`; there is no separate controller.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
kubectl -n tuneplane scale deployment tuneplane-worker --replicas=3
```

Every worker uses the same command and automatically receives tasks. Independent
run actions, diagnoses, Hub imports, and imports into different Volumes can run
concurrently. GPU allocation stays with whichever worker holds the scheduler
lease. Shared maintenance retains its coordination; adding replicas does not
divide a single transfer.

`TUNEPLANE_WORKER_CONCURRENCY` controls in-flight deliveries per process (default
4\). `TUNEPLANE_WORKER_IO_CONCURRENCY` limits imports and Volume measurements
across the entire pool (default 4). Set the same shared limit on every replica.
Increasing replicas alone does not raise that storage budget. Account for database
connections and storage bandwidth when raising it.

Use storage that supports conditional writes. Browser upload CORS must allow
`If-None-Match` alongside the existing upload headers. Upgrade API, consumers, SDK and
console together: uploaded Volume files require publication before becoming
visible. Managed reads use immutable generations; old mutable data directories
are not supported by this protocol.

## Check it worked

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
kubectl -n tuneplane get pods
kubectl -n tuneplane port-forward svc/tuneplane-server 8080:80
curl -s localhost:8080/api/version
```

Then check the executor's own view — it is what tells you whether the storage claim is really RWX:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
curl -s localhost:8080/api/cluster/health | jq .storage
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.