Skip to main content
A Node Fleet runs a tuneplane-node daemon on each machine. Each daemon opens an outbound bidirectional gRPC connection to the control gateway. Node control needs no inbound HTTP port or shared bearer credential. Serving workloads have separate inbound network requirements, described below. Each Node has its own revocable credential, rotated automatically over server-authenticated TLS.

​
Requirements

  • Docker or Podman; GPU workloads also need NVIDIA Container Toolkit.
  • A Node Fleet and a shared storage root mounted at the same absolute path on the console and every workload machine.
  • A gateway server TLS certificate trusted by the Nodes. System trust roots are used by default; private CAs can be configured explicitly. No Node client certificates or separate PKI service are required.
  • Explicit CPU, memory and GPU-index budgets on each Node. GPU jobs also declare CPU and memory limits. Runtime images must resolve to immutable digests.

​
Enroll and start

Create a Join Token on the Fleet page, then run:
The address is the workload hostname used for peer communication. It is not a control endpoint. Enrollment saves the Node identity and its private credential under TUNEPLANE_NODE_STATE_DIR. Set this directory before both joining and starting the daemon. Keep it on persistent local disk outside shared job storage. Processes managing the same container runtime must share that state directory.
An empty GPU-index list declares a CPU-only Node. The installer enrolls the Node and installs its systemd unit; configure resource grants in /etc/tuneplane-node.env before starting it. Preserve the state directory across upgrades and restarts.

​
Automatic gateway trust at enrollment

Set the address Nodes can reach on the control plane:
On first startup, the API generates a self-signed gateway certificate valid for 3650 days, encrypts it in PostgreSQL using TUNEPLANE_SECRET_ENC_KEY, and caches it under <repo_root>/.tuneplane-data/gateway-tls/current/. The gateway reuses that identity; restarts never replace it. Nodes receive the address and public certificate during enrollment over verified Console HTTPS, then need only tuneplane-node serve. No per-Node gateway certificate setup is required. TUNEPLANE_GATEWAY_TLS_DIR overrides the private local cache directory, outside workload storage. Private keys have mode 0600. Each API, worker and gateway replica can use its own directory: all restore the same database identity. Concurrent first startup is serialized in PostgreSQL. Back up the database and its encryption key together; a lost local directory is rebuilt without changing Node trust. Existing local certificates are imported on first database publication. A local identity that differs from the database, is incomplete, expired or address-mismatched fails startup rather than silently replacing trust.

​
Multiple control-plane instances

All instances must share PostgreSQL, TUNEPLANE_SECRET_ENC_KEY, the web JWT secret, and the same public gateway target. Use a stable DNS name in front of gateway replicas with TCP/TLS passthrough. Nodes reconnect after a gateway exits; durable command ownership remains in PostgreSQL and the Node journal. Redis and the normal worker/leader configuration still apply to control-plane scaling. A separate Docker named volume on each machine is sufficient for TLS caching; no network filesystem is required for certificates. Compose retains its existing local volume. Helm uses a per-Pod cache when gateway.tlsSecret is empty, and an external Secret when it is set. Workload storage remains a separate deployment requirement. For externally issued gateway certificates, set TUNEPLANE_GATEWAY_TLS_AUTO_GENERATE=false, pass --certificate and --private-key to the gateway, and set TUNEPLANE_NODE_GATEWAY_TRUST_BUNDLE to the public CA file readable by the API. Leave that bundle empty for system-trusted issuers. Explicit trust bundles also suppress automatic generation. Never distribute a private key. Console HTTPS and gateway TLS may use different CAs. If Console HTTPS itself uses a private CA, establish that initial trust with join --trust-bundle or TUNEPLANE_NODE_ENROLLMENT_TRUST_BUNDLE (and curl --cacert when fetching the installer). Enrollment does not disable HTTPS verification. TUNEPLANE_NODE_GATEWAY and TUNEPLANE_NODE_TRUST_BUNDLE remain explicit daemon overrides; do not use the latter for an enrollment-only CA when the gateway has a different issuer. Enrolled trust is a saved snapshot: a new server certificate from the same CA needs no Node changes; changing the CA requires updating Node trust before switching the gateway.

​
Generate a 10-year self-signed certificate

From the repository root, list every DNS name or IP that Nodes use to reach the gateway:
The certificate expires in 3650 days. Repeat --host for additional names or IPv6 addresses; do not include a URL scheme or port. The output directory must be new: the script refuses to overwrite existing certificates or keys.
  • tls.crt and tls.key: mount on the gateway; the key is created with mode 0600 and must be readable by the gateway process owner.
  • ca.crt: the public copy of this self-signed certificate. Place it where the API can read it and set TUNEPLANE_NODE_GATEWAY_TRUST_BUNDLE to that path. Do not distribute tls.key to Nodes.
This script is optional. To use its output, disable automatic generation and pass --certificate <output>/tls.crt --private-key <output>/tls.key to the gateway. Set the API’s public CA path to <output>/ca.crt. The address advertised through TUNEPLANE_NODE_GATEWAY_TARGET must match one of the certificate names.

​
Deploy the gateway

Gateway replicas share the migrated PostgreSQL database and the console’s SecretBox key. They do not mount the container runtime or job storage. For Helm, enable gateway.enabled and configure the public gateway target. Optionally set gateway.tlsSecret to an external TLS Secret containing tls.crt and tls.key. The optional plain Kubernetes manifest is deploy/k8s/gateway.yaml. Compose provides a gateway profile and shares the automatically generated identity from its persistent volume. The Helm chart disables automatic generation when gateway.tlsSecret is configured. The same gateway process also admits collector telemetry on port 8444. Expose port 8443 through a TCP endpoint that preserves TLS to the gateway. The Node verifies the gateway hostname and presents only its own credential. Removing a Node invalidates its credential and its existing control session. Credentials last thirty days and online Nodes rotate them daily. A one-hour overlap and atomic local state recover a lost rotation response. A Node offline past credential expiry must enroll again.

​
Node program updates

The managed installer creates immutable release environments under /opt/tuneplane-node/releases/, with current selecting the daemon and updater selecting the last healthy updater. Node identity, credentials and its command journal stay in the persistent Node state directory. Re-running the installer uses the same idle-only updater. For an existing unmanaged pip/uv installation, first drain workloads and stop its service, preserve the original TUNEPLANE_NODE_STATE_DIR in /etc/tuneplane-node.env, then run the new installer. It refuses to replace an active unmanaged service.
Updates follow the exact release advertised by the enrolled control plane at /api/node-release, over verified HTTPS. They never select the package index’s latest version or automatically downgrade when a load balancer reaches an older control-plane replica. Private Console CAs use TUNEPLANE_NODE_ENROLLMENT_TRUST_BUNDLE, including in /etc/tuneplane-node.env. The configured package mirror must support HTTPS and provide compatible wheels. A different wire protocol requires a planned upgrade, not an automatic switch. Automatic updating is off by default. After testing one canary Node, enable it on selected Nodes:
The installer also accepts TUNEPLANE_NODE_AUTO_UPDATE=true. The timer checks hourly with up to ten minutes of jitter. Operators control batches by enabling the timer on selected hosts; there is no fleet-wide rollout concurrency limit. Training, Playground sessions, deployments, prepared reservations and uncertain ownership all defer the update. New code is installed in a separate environment before stopping the daemon. A local admission fence and a second journal check close the race with a newly accepted job. The updater snapshots the idle journal, switches the executable atomically and waits up to sixty seconds for fresh, healthy inventory and an authenticated gateway connection. Failed startup restores the previous executable and journal; it never re-enrolls the Node or kills job containers. A crashed switch is recovered by the last healthy updater before contacting the control plane. Long-lived deployments can postpone updates indefinitely; relocate or stop them in a planned maintenance window. Inspect /opt/tuneplane-node/update.json and journalctl -u tuneplane-node-update.service. A release that failed health checking is blocked until an operator runs tuneplane-node update --retry or a newer release is selected. Old environments are retained for recovery. Containerized Nodes are upgraded by their deployment system, not by this host/systemd updater. This policy draws on K3s upgrade plans and Kubernetes worker upgrades: pin the target, limit the initial rollout, and protect workload ownership before switching.

​
Execution behavior

A launch first persists its immutable member plans and pending attempt. Separate launch consumers prepare all members, then commit. Retries reuse the same operation identity; a lost response does not authorize a second workload. A disconnected Node produces no terminal evidence. Its capacity remains held until reconciliation confirms termination. Reconnection invalidates old free capacity until fresh inventory arrives. Draining and operator cordons survive reconnection. Stop and bounded log reads have capacity independent of slow launches. Node Fleets support one homogeneous resource pool on at most 8 machines; a larger job is refused when it is submitted and belongs on a Kubernetes or Slurm Fleet. Multi-machine GPU jobs use process-group or Ray orchestration. Ray requires an exact ray_version in the runtime artifact. Multiple resource pools use Kubernetes or Slurm. Peer ports and shared storage must be reachable independently of the outbound control connection. Object delivery and physical distributed runtime certification are not established by enabling the gateway.

​
Completed object inputs

A Node job can reference a completed checkpoint stored in the platform’s S3 artifact registry. Set TUNEPLANE_NODE_STAGING_CAPACITY_GB on each Node to reserve local disk for verified input copies; zero capacity refuses these inputs. Existing output and cache mounts still require the shared storage root. Capsule object delivery is described below. The control gateway needs the same object-store configuration as the console. Only that gateway signs read URLs. The Node obtains fresh, 15-minute read grants over its authenticated gRPC control connection, then downloads directly over HTTPS. Object-store URLs must use certificates trusted by the Node’s Python runtime; for a private object-store CA, configure its system trust or SSL_CERT_FILE in the daemon environment. Redirects and HTTP-only object endpoints are refused for this input path. Downloads reserve disk before starting, resume interrupted files, verify the completed manifest and publish a read-only mount. The workload receives the path through TUNEPLANE_INIT_FROM_PATH, without the download credentials. Retries are bounded by the Node preparation deadline. Preserve the Node state directory and its materialized inputs across daemon restarts. Stopping a launch, disabling its owner or withdrawing the source artifact prevents new read grants. A URL already issued remains usable until its expiry; this is not immediate object-store revocation. Non-exportable Volumes are not eligible for this artifact delivery channel.

​
Object Capsule delivery

Set TUNEPLANE_NODE_CAPSULE_OBJECT_PULL=true on the control plane to deliver Capsules through the same scoped object-read channel. Reserve enough Node staging disk for both the immutable cache and the writable execution copy. The trusted runner remains a read-only mount. Shared output and cache mounts are still required, including one shared output directory for distributed training. Completed artifact inputs also work on Slurm and Kubernetes through verified copies on the working tier; Kubernetes needs its working-storage PVC. For Kubernetes Capsule startup, configure an HTTPS public control endpoint that Pods can reach. Pods obtain fresh short-lived object URLs at startup, so ordinary queue delays do not expire a URL created at submission time. The attempt-specific read credential expires after thirty days and is revoked by an inactive attempt or disabled user. An explicit spec.resources.network can request transport: tcp, isolation: fleet and an exact interface such as bond0. The interface must exist in the workload namespace. RDMA and per-job isolation requests are refused until a backend has an enforceable, certified profile. Validate workload peer routes and object-store access separately from the control connection.

​
Playground and model deployments

Node Fleets support both Playground sessions and Model Deployments. Serving reserves a port on the selected Node before starting its container. The control plane reads the adopted Node address and proxies requests to that address; it needs no GPU or Docker socket. Advertise a Node address reachable from the control plane and allow the configured serving port range through the network. Models, outputs and caches must satisfy the shared-storage mount requirements. An outbound gRPC connection alone does not make the serving endpoint reachable.