> ## Documentation Index > Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt > Use this file to discover all available pages before exploring further. # Fleets > Registering machines: what a Fleet is, how a node joins, drain and liveness Node control uses an outbound TLS-authenticated gRPC connection to the control gateway. Before starting the installed service, configure its private state directory, shared storage, CPU/memory budgets and gateway trust. Enrollment alone does not make a Node ready. The Fleet page shows the control connection separately from scheduling state and resource inventory. See [Node deployment](https://docs.tuneplane.ai/en/ops/executor-node.md) and [native telemetry](https://docs.tuneplane.ai/en/ops/telemetry.md). A **Fleet** is a named set of machines and the backend that runs work on them. It is a record you create, not a setting you export: one console carries several Fleets at once, and every job and deployment records the Fleet it was admitted against. This replaced a single deployment-wide backend variable. See [ADR-0013](https://github.com/tuneplane/tuneplane/blob/main/docs/adr/0013-an-execution-backend-is-a-registered-fleet.md) for why. A new console starts with no Fleets; administrators register each backend explicitly. ## What a Fleet carries | Field | Values | Changeable | | - | - | - | | `kind` | `node` \| `kubernetes` \| `slurm` | **No.** It describes what the machines are | | `delivery` | `shared-mount` \| `object-pull` | **No.** Re-pointing it relocates where every running job writes | | `capabilities` | any of `train`, `eval`, `serve`, `env` | Yes | | `visibility` | `public` \| `project` | Yes | | `state` | `active` \| `draining` \| `disabled` | Yes | | `config` | deployment settings this Fleet overrides | Yes | `config` holds the keys that differ between two Fleets of the same kind — the slurm REST endpoint, the KubeRay namespace. A key no setting reads is logged as a warning rather than ignored: a misspelled override is a setting an operator believes is in effect and is not. > [!NOTE] > > A capability is a promise, not a preference. A Fleet without `serve` is not a Fleet that > serves badly — a deployment naming it is refused. ## Creating one **Fleets → New fleet**. Pick the backend and what the Fleet is for; the delivery follows the backend unless you change it. Or through the API: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} curl -X POST https://tuneplane.corp/api/fleets \ -H "Authorization: Bearer $ADMIN_TOKEN" \ -H 'Content-Type: application/json' \ -d '{"id": "gpu-lab", "kind": "node", "delivery": "shared-mount", "capabilities": ["train", "serve"]}' ``` Administrators only. Registering machines is an operator's job; picking a Fleet to submit to is a user's. Register and configure a Fleet before submitting work. Startup, placement and restart never create Fleets, including after the last Fleet is deleted. Existing Fleets are preserved. ## Registering a machine Only `node` Fleets have machines to register — `kubernetes` and `slurm` get theirs from the cluster manager. The control plane needs no GPU. Mint a join token on the Fleet page — **Add a machine**. It is single-use, expires in 24 hours, and is stored only as a hash. The page shows one line to paste on the machine: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} curl -fsSL https://tuneplane.corp/install.sh | TUNEPLANE_FLEET=gpu-lab TUNEPLANE_JOIN_TOKEN= sh ``` On a machine that already has the package: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tuneplane-node join --server https://tuneplane.corp --fleet gpu-lab --token tuneplane-node serve --target control.tuneplane.corp:8443 ``` > [!WARNING] > > The join token is the whole authorization. You need root on the machine, not an account > on the console — which is what makes the one-liner safe to hand to whoever administers > the box. Treat a token in a chat log as a standing key until it expires. What comes back is the node's identity and a bounded, automatically rotated credential, written under `TUNEPLANE_NODE_STATE_DIR`. **The credential is returned once.** Losing it means asking for another token and joining again as a new node, which is correct: a machine that lost its identity is not the same machine to a control plane reconciling containers against it. The node reports its own inventory at join — card name, count, driver, container runtime, and whether it can see the storage root. The console maps the card name onto a hardware series. A card the registry does not know is named in the join output so an administrator can add it; until then, jobs that pin a series are not placed there. ### An address is not an identity A node is identified by the id the console issued, never by `host:port`. A machine that moves stays the same machine. The consequence is that two records may legitimately answer at one address, so joining where another live node already answers is a warning rather than a refusal — printed by `tuneplane-node join`, because the usual cause is a re-join that left the old record behind, and then the console counts one machine's cards twice. ## Liveness New Node placements require a live control-session lease and healthy inventory observed within 45 seconds. A displayed `active` state alone is insufficient. The capability-specific liveness policy below governs stale-node handling; it does not allow stale inventory to receive work: | Capability | Quiet for | Consequence | | - | - | - | | `train` | 300s | Stops receiving new work. Running work is reconciled separately | | `serve` | 45s | Revisions on it are marked not-Ready and the deployment reports degraded | | `env` | 45s | The Fleet stops being a candidate; in-flight episodes fail on their own timeout | These are three consequences, not three states. A node is declared `gone` on the **most generous** of its Fleet's capabilities; the tighter rules answer per capability. "Do not send traffic there now" and "do not place anything there at all" are different claims, and a Fleet that declares all three would otherwise stop taking training work after 45 seconds of silence. Losing the connection is not evidence that a container exited. Attempt reconciliation and gang deadlines determine recovery or failure; terminal attempts still require cleanup. A reconnecting Node reports durable command state so the control plane can reconcile it. A `shared-mount` node that reports losing the storage root is cordoned, and un-cordoned when the mount returns. An operator's own cordon is never lifted by a heartbeat — that one is a decision, and a heartbeat is an observation. ## Drain, cordon, remove | | Scope | New work | Running work | | - | - | - | - | | Fleet `draining` | every node | stops | continues | | Fleet `disabled` | every node | stops | continues | | Node `draining` | one node | stops | continues | | Node `cordoned` | one node | stops | continues | None of them kills anything; that is the point. `cordoned` is for a machine that is coming back, `draining` for one that is leaving. Removing an active node is refused. Its identity is what makes its containers reconcilable, so drain first and let the work finish. `force=true` exists for a machine that is already gone; it is recorded in the audit log as forced, because what it leaves behind is a container nothing can reconcile. A Fleet is deletable only with no nodes and no live deployment pinned to it. A deployment never changes Fleet, so deleting the Fleet under it would leave a row whose backend resolves to nothing: unobservable, unstoppable, and still holding cards. ## Choosing a Fleet | Workload | How it picks | | - | - | | Training job | Placed automatically among Fleets that can `train`, fit the requested series and card count, and are visible to the submitter | | Lifecycle job | The same placement, on the same gates | | Model deployment | **Named by the submitter.** With one serving Fleet the choice is made for them; with more than one, an unnamed Fleet is refused | | Volume, dataset | Neither. They are storage, and reach a job through its Fleet's `delivery` | | Playground session | Placed among Fleets that can `serve`; `node` and `kubernetes` | | Environment | **Named in its manifest.** The Fleet must declare `env`, and the address is resolved per job from a node answering right now | A deployment is the exception because its address outlives the decision and never moves, and nothing on a Fleet record says which department owns the machines or what data may touch them. The platform declines to answer a question it cannot answer. Placement refuses at admission with a reason — capability, visibility, or inventory fit. A job that no Fleet can hold is told so while the submitter is still looking at the response, not hours later at the top of a queue. > [!NOTE] > > Inventory fit is a check, not a reservation. The allocator remains authoritative for > which cards a job actually gets; this only stops a request for hardware the Fleet has > never had from queueing forever. ## Visibility A `project` Fleet is invisible outside its project, and a job naming one is refused as **not found** rather than forbidden — a private Fleet's name does not leak through an error message. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.