> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Storage layout

> Separate run data, caches and governed inputs, with filesystem capacity counted once.

`TUNEPLANE_STORAGE_ROOT` supplies the default layout. Optional absolute directory
overrides let different purposes use different disks or mounts. The API, workers
and Nodes must agree on these settings. Changes require a restart and do not move
existing files.

## Configure independent directories

| Setting | Default | Contents |
| - | - | - |
| `TUNEPLANE_RUNS_ROOT` | `<storage>/runs` | Complete runs: work, attempt outputs and logs |
| `TUNEPLANE_CACHE_ROOT` | `<storage>/cache` | Default parent for caches |
| `TUNEPLANE_HF_HUB_CACHE_ROOT` | `<cache>/hf/hub` | HF model and raw dataset repositories |
| `TUNEPLANE_HF_DATASETS_CACHE_ROOT` | `<cache>/hf/datasets` | HF Arrow and processed data |
| `TUNEPLANE_MODELSCOPE_CACHE_ROOT` | `<cache>/modelscope` | Native ModelScope SDK cache |
| `TUNEPLANE_HUB_CACHE_ROOT` | `<cache>/hubs` | TunePlane Hub downloads, including ModelScope |
| `TUNEPLANE_DATASET_CACHE_ROOT` | `<cache>/datasets` | Platform Dataset working copies |
| `TUNEPLANE_VOLUME_CACHE_ROOT` | `<cache>/volumes` | Materialized Volume inputs |
| `TUNEPLANE_PACKAGES_ROOT` | `<storage>/packages` | Filesystem job packages |
| `TUNEPLANE_REFLOW_ROOT` | `<storage>/reflow` | Captured traffic buffers |
| `TUNEPLANE_STATE_ROOT` | `<storage>/state` | Executor bookkeeping |

Empty overrides use the defaults. An explicitly configured directory must already
exist and be accessible; startup does not create a missing override. Configure
mount dependencies in the host service manager as well: an existing empty mount
point does not prove the external filesystem is mounted. Overlapping categories
and resolved symlink aliases are rejected.

A run keeps this shape under the resolved runs location:

```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
<runs>/<user>/<experiment>/<run_id>/
├── work/
└── attempts/<attempt>/
    ├── out/       checkpoints, exports, reports
    └── logs/
```

A disk named `checkpoints` can hold `TUNEPLANE_RUNS_ROOT`; checkpoints remain part
of their run. There is no independent checkpoint-only setting. This preserves
resume, quota accounting and whole-run reclamation.

## Framework caches

Training bindings set a container-private `HF_HOME=/tmp/tuneplane-hf`, with
`HF_HUB_CACHE`, `HF_DATASETS_CACHE` and `MODELSCOPE_CACHE` pointing at the resolved
persistent directories. HF tokens must not be stored in shared cache directories.
The HF Hub cache contains both models and raw datasets; it is not a model registry.
TunePlane's own ModelScope prefetch uses `TUNEPLANE_HUB_CACHE`, independently of the
native SDK. Explicit framework `cache_dir` and output arguments still matter.

Shared cache directories are intended for mutually trusted workloads. Moving
credentials out of `HF_HOME` does not make cached private model bytes private.
Use governed read-only inputs for material requiring per-project authorization.
Node-local Torch/Triton compilation and Ray temporary paths should be assigned
by the training image or recipe; this change does not configure them automatically.

## Mount relationships

With Docker Local execution, every configured host path must be visible at the
same absolute path in API and worker containers. The host Docker daemon resolves
bind sources on the host, not in the API container. Training containers receive
only their entitled run slices, framework caches and authorized inputs, never the
entire state or package tree. See [Compose installation](https://docs.tuneplane.ai/en/ops/install-compose.md).

Kubernetes keeps `TUNEPLANE_K8S_STORAGE_PVC` for the default root. For independent
claims, set explicit locations and map their setting names to PVC names:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
TUNEPLANE_RUNS_ROOT=/srv/tuneplane-runs
TUNEPLANE_CACHE_ROOT=/srv/tuneplane-cache
TUNEPLANE_K8S_STORAGE_PVCS={"runs_root":"run-data","cache_root":"download-cache"}
```

Each claim starts at its configured directory. Mount these claims into API and
worker pods too; Helm provides `storageLocations` entries with `name`,
`existingClaim`, and `mountPath`. Job subPath mounts stay scoped. Multi-node shared
writes require RWX claims. Overrides outside the default root keep the same
absolute path when the default root is rebased for Kubernetes or Slurm.

## Governed and external storage

`TUNEPLANE_VOLUME_ROOT` must be separate from the Storage Root and every override.
Register existing model, dataset and shared material through Storage Connections
and [External Volumes](https://docs.tuneplane.ai/en/guides/volumes.md). Jobs receive authorized read-only
subdirectories at `$VOLUMES_DIR`; naming a disk `shared` does not grant access.
Keep PostgreSQL, Node credentials and telemetry spool outside job mounts.

S3 holds canonical dataset versions, packages and published artifacts. Filesystem
caches are working copies. S3 API storage is not implicitly a POSIX mount; use a
verified framework upload/publish workflow for durable outputs. External Volume
contents remain externally owned.

## Space accounting and pressure

The overview distinguishes two numbers:

* **Tracked platform files:** logical file bytes in runs, Reflow, caches,
  packages, state and other files beneath the default root/cache parent. This is not allocated disk blocks or a promised reclaim size.
* **Filesystem capacity:** used/free/total for distinct local mounted filesystems,
  counted once when several configured directories share a filesystem. These
  figures include non-TunePlane files on those filesystems.

Object-store buckets, external read-only Volumes, PostgreSQL, Redis, Node-local
scratch and container image layers are not part of tracked Working Tier files.
Do not interpret this number as all infrastructure consumption. Hard links inside
one scanned directory are counted once; cross-run quota attribution remains per
run, and filesystem capacity is the physical authority.

User quotas count runs plus that user's Reflow. Team totals follow each run's
stamped team, not the current membership of its owner. Shared caches, packages and
state are platform overhead. A missing or unreadable location marks a measurement
incomplete and keeps the last complete quota measurement; it never replaces a
missing mount with a fresh zero. The overview shows unavailable paths and per-disk
free space. `TUNEPLANE_STORAGE_SCAN_INTERVAL_S` controls scan frequency.

`TUNEPLANE_DISK_WATERMARK_PCT` checks the most pressured filesystem, not a weighted
average; a full cache disk cannot be hidden by an empty run disk. An unavailable
configured filesystem also holds new jobs when this gate is enabled. A value of
zero disables the gate. Space quota remains eventually consistent between scans.

## Retention and moving data

Caches are reconstructible only while their sources remain accessible. Reclaim
only with no active readers/writers; no automatic cache deletion is introduced by
these settings. Runs, unpublished exports, Reflow and executor state are not
ordinary disposable caches. Preserve working packages for pause/resume.

Before changing an existing location, drain affected work, back up metadata and
files, copy and verify content, and migrate stored absolute references or retain
the old mount until those references are retired. Update API/worker/Node mounts
and settings together, verify resume and artifact reads, then retire the old
location. Editing a setting alone does not relocate historical paths.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.