> ## Documentation Index > Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt > Use this file to discover all available pages before exploring further. # Roles and quotas > Who may do what, and how much of the cluster they may hold. ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp admin user-add alice tp admin set-role alice operator tp admin set-quota alice --gpus 8 --jobs 4 --daily-gpu-hours 200 ``` ## Roles Three of them. | Role | Can | | - | - | | `operator` | Submit, watch, compare, publish plugins and environments, manage their own datasets and models. The default | | `auditor` | Read the [audit trail](https://docs.tuneplane.ai/en/console/audit.md), and nothing beside it. Scoped like an operator, refused like an operator, offered no premium surface, and never exempt from quota | | `admin` | All of that, plus users, quotas, hardware, settings, scheduled tasks, audit, and disabling published extensions | An admin-only endpoint returns `403` to an `operator` or `auditor` token, not `401`: the token is fine, the permission is not. `auditor` exists so somebody can read the record without being able to act on the platform, which is a separation an MLPS level 3 assessment asks for by name. Every parameterless admin `GET` is swept by a test that refuses an auditor token, so a route added without a gate fails the day it lands. `operator` is the default for a new account, and `TUNEPLANE_OIDC_DEFAULT_ROLE` decides what a first-time single sign-on login gets. ## Quota dimensions A quota is per user, with independent limits: **max\_concurrent\_gpus** `int` (default: 8) The most GPUs this user may hold at once, across all their running jobs. Exact — admission counts it before dispatch. **max\_concurrent\_jobs** `int` (default: 4) How many jobs may run at once, regardless of size. Stops one person filling the queue with small jobs. **daily\_gpu\_hours** `int` (default: 0) GPU-hours per day. 0 is unlimited. Day boundaries are computed in the service timezone. **max\_concurrent\_cpus** `int | null` (default: null) CPU cores reserved across this user's CPU and GPU jobs. Null is unlimited; zero refuses new reservations. Configure it in the user's quota editor. **max\_concurrent\_memory\_gb** `int | null` (default: null) Memory reserved across jobs, in GiB. Null is unlimited; zero refuses new reservations. Each pool must declare positive CPU and memory requests when either host limit applies. Host limits also appear in the team quota editor. Clearing a field removes its limit; entering zero retains a zero allowance. Queued jobs reserve quota, while paused and terminal jobs release host reservations. Unknown historical requests block new work under a host limit until those jobs have been reconciled. These figures describe reservations, not measured CPU utilization or process memory. They are separate from the local executor's physical host capacity budget. Public zero-GPU submission is still disabled while CPU execution and admission integration is completed. **max\_storage\_gb** `int` (default: 0) How much the user's run directories may hold. 0 is unlimited. Measured rather than reserved, so it is eventually consistent. **allowed\_profiles** `comma-separated` (default: all) Which hardware profiles this user may request. Empty means all of them. Plus `priority`, which orders the queue, and a per-series map for a heterogeneous cluster — `{series: {max_concurrent_gpus, daily_gpu_hours}}` — when one global number is too blunt. ## Concurrency and a daily budget are different tools | | Governs | Over it means | | - | - | - | | Concurrent GPUs and jobs | How much somebody holds at once | The job waits in `QUEUED` | | Daily GPU-hours | How much somebody burns in a day | The job waits, and dequeues after the budget resets | With only a concurrency quota, one person running 8 GPUs all day and one running them for ten minutes look equally compliant, while the cost differs by a factor of 140. Set `0` for unlimited. The number the gate reads is the same one the user's [Usage](https://docs.tuneplane.ai/en/console/usage.md) page shows, integrated over the platform's local day and including the elapsed part of running jobs. So "today's budget is used up" is a claim the person can check themselves. ## Dequeue order ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} TUNEPLANE_SCHED_POLICY=fair-share # default: priority-fifo TUNEPLANE_FAIRSHARE_WINDOW_H=24 ``` `priority-fifo`, the default, sorts by priority then submission time. Predictable, and enough for one team. `fair-share` puts recently-lighter users first **within the same priority band**. An explicitly granted priority still wins, because fairness should not override a decision somebody made. It compares *relative* usage — recent GPU-hours divided by quota GPUs — so 100 GPU-hours from somebody with a 64-GPU quota is not the same as 100 from somebody with 8. Switch when more than about three teams share one cluster. Before that, first-come-first-served is easier to predict. After it, one person submitting twenty jobs inside their quota fills the queue, and that becomes an interpersonal problem that gets blamed on the platform. ## Team quotas Personal quotas are not enough once several departments share a cluster: eight people at four cards each are all inside their own limits and are thirty-two cards of one department's capacity. That is what a team quota governs. **A run's budget comes from the team owning its project, never from whoever submitted it.** This is the crux of the model: a person can belong to several teams, and only the project answers whose capacity a run spent. The team is stamped on the job row at submit, so moving a project to another department later does not rewrite work already done -- the same rule `gpu_seconds` follows by freezing on a terminal state. Create teams, set their quota and manage members under **Administration → Teams**; a project's owning team is changed in its settings, by an administrator. Two semantics are the opposite of the per-user ones, and both are worth remembering: * **No team quota means no team limit**, not a limit of zero. The per-user reading is the reverse (no quota, no submitting), which is a deliberate deny-by-default for new accounts; carried up here it would make creating a team a way to block everyone in it instantly. * **Administrators are not exempt from a team quota.** A personal exemption says "this person is trusted with the cluster"; skipping a team's budget spends another department's money, and being an administrator is not a claim on that. A job over its team's limit **queues** rather than being rejected: it is a shared condition that clears as colleagues' jobs finish, and rejecting would mean resubmitting by hand for something the platform can wait out itself. ## Billing The platform has always measured GPU-seconds exactly and attributed them to a project, a person and a team. Billing adds **one thing: a price.** It re-measures nothing. The hours come from the same ledger the usage page shows, and the attribution rule is `cost_breakdown`'s: **a run is counted in the period it ended.** A long run spanning months belongs to the month it finished in; a job still going is counted for the part it has run and labelled separately. An invoice that disagrees with the usage page is worse than no invoice -- it is an argument nobody can settle, and the usage page is the number people already trust. Read the bill under **Administration → Billing**, grouped by team, project or person, and exportable as .xlsx. ### The price book Rates are per **accelerator series** and carry a start date. Two rules: * **The first rate for a card type covers all history** -- nothing was ever billed at another one, so there is no settled period to rewrite. That is what lets a deployment turning billing on price the quarter it already ran, which is the first thing anybody asks for. * **A later rate cannot move backwards.** It would rewrite a bill somebody has already been shown and acted on. A change is a new row with a later start; rates accumulate and are never edited. A card type that produced hours with no rate in force is **named** (`unpriced_series`) and counts as zero. A confident total quietly missing a series nobody remembered to price is much worse than one that says so. Money is **integer micros** (millionths of a currency unit) from the ledger to the report, rounded once at the edge. Floating point is not a thing to add up money with: a fraction of a cent of drift per run becomes a total nobody can reconcile. ### Monthly budgets A team can carry a budget **per calendar month** in the service timezone. Over it, new work **queues** rather than being rejected -- the month rolls over on its own, and somebody told to resubmit in eleven days will not. The reading is **already spent**, not predicted: nobody knows what a job that has not run will cost. So the run that crossed the line always finishes -- the overage is something it produced, not something it was admitted with. The same shape as the daily GPU-hour budget, one level up and denominated in money. ## Borrowing and reclaim (preemption) Static quotas fragment a cluster and then leave it idle -- the complaint every shared cluster eventually produces, and the reason Kueue (`reclaimWithinCohort`), Volcano (`deserved` plus reclaim), YARN (guaranteed vs maximum capacity) and Slurm (QOS preemption) all grew **borrowing**. With `TUNEPLANE_SCHED_PREEMPTION` on, a team's concurrent-GPU quota becomes a **guarantee** rather than a ceiling: * a team may run above it while the cluster has room (`capacity_gate` still bounds that by physics) * that overflow is reclaimed when a team **below its own guarantee** is queued and blocked on capacity **Borrowing and reclaim are one switch.** Lending without reclaim lets one team hold the cluster indefinitely while others sit inside their quotas waiting, which is worse than the hard cap it replaced. ### What is reclaimable, stated as narrowly as possible All five must hold. Each exists because without it the mechanism does something somebody would be right to be angry about: | Condition | Without it | | - | - | | the taker is under its guarantee | this is not reclaim, it is one team outbidding another -- and no team could rely on the number it was given | | the victim's team is over its guarantee | taking from a team inside its guarantee breaks the only promise a quota makes | | a different team | a team preempting itself is a queue reordering itself, at the cost of a checkpoint | | it has run for the minimum | a busy queue would reclaim, admit, reclaim, and the cluster would spend its day writing checkpoints | | it holds the series that is short | reclaiming an H200 job does not help somebody waiting for an H100 | It takes the **youngest** first: the least work thrown away. Deliberately not the lowest priority -- priority orders who runs next, and reusing it here would let a high-priority long job be safe while a high-priority job that started a minute ago is taken, answering two questions with one number. ### Reclaim is a pause The platform already had this path and it already means the right thing: `PAUSED(auto_resume)` is the *system-side* pause that maintenance drain and window enforcement use. Metering stops at the pause, the working directory is kept, and the queue worker resumes from the last checkpoint. **The person whose job was reclaimed does nothing**, and is told why. The cost is honest and bounded: progress since the last checkpoint. NeMo-RL saves every `save_period` steps by default, so it is minutes. ## Splitting a GPU (MIG, time-slicing) **It is one row in the hardware registry.** The NVIDIA GPU Operator advertises a MIG partition as its own extended resource (`nvidia.com/mig-3g.40gb`) and a time-slice as `nvidia.com/gpu.shared`. Register the partition as a card type under **Administration → Settings → Hardware** with that name in `k8s_resource`, and it gets independent capacity, quota, metering and time windows -- because all four are already keyed by series id. **The scheduler does not change at all.** Splitting a card is a registry entry, not a scheduling feature. Only the kubernetes backend reads the field. On `node` the allocator hands out whole cards by index and has no notion of a partition; declaring one there would advertise capacity nothing can place. ## Approvals: an escalation path, not a workflow engine A department that spends its month's budget on the third is queued for twenty-eight days. Queueing is the right **default** -- the condition clears on its own, and rejecting would mean telling somebody to resubmit in three weeks -- but there has to be a way to say "this one genuinely cannot wait", and for that to leave a record of who said so. **It is not wired into admission.** The obvious design is a third verb beside `queue` and `reject`: over budget, so route to approval. It is wrong, and the reason generalises. Team quota often clears within minutes as colleagues' jobs finish; sending every such denial to a human produces a queue of requests that answer themselves before anybody reads them, and an approver who learns to ignore the list. So escalation is **opt-in**: the job queues as it always did, and the submitter asks for an exception from the job page when waiting is not an option. **One approval lets one run past.** Not the team, not the month, and not until somebody remembers to revoke it. A grant that outlives the reason it was given is how a budget stops being a budget, and a narrow grant is one an approver can actually reason about in the ten seconds they will spend on it. The approver is an **owner of the team that holds the budget**, with administrators as a fallback -- somebody has to be able to answer when a team has no owner, or when the only owner is the person asking. **Nobody approves their own request**; rejecting your own is allowed, because withdrawing needs no second person. A quota change is always an administrator's call: a team owner approving their own raise is not a limit. ### Deliberately absent One level, and an expiry. No counter-signing, chains, or delegation. Those belong to whatever system an organisation already runs its approvals in; rebuilding them here produces a worse copy that nobody's compliance department accepts anyway. What the platform owes such a system is **a record and a webhook**, not a re-implementation. Requests nobody answers expire on their own. A pending list that only grows is one nobody reads, and the request that mattered is somewhere in it. ## Syncing teams from single sign-on With `TUNEPLANE_OIDC_GROUP_CLAIM` set, every sign-in reconciles the person's membership against the directory -- **removing as well as adding**: somebody who left a department in the IdP has left it here. Empty (the default) means the platform never touches membership, which a deployment that manages teams by hand must keep, since reconciling against an absent claim would empty everybody out. A group the directory names but the platform does not have is skipped, never created: an IdP group is not permission to invent a budget holder. `TUNEPLANE_OIDC_GROUP_PREFIX` strips the path prefix a directory hands out. ## CPU and memory usage The usage page reports reserved CPU cores and memory alongside metered CPU core-hours and memory GiB-hours. Metering multiplies the declared request by recorded execution time. Queuing and pause gaps add no time; resume and retry add another execution interval. A CPU-only job always contributes zero GPU-hours. Pause, terminal reporting and deletion freeze host usage. Deleting a job keeps its usage attributed to the original user and team. Rows from before host metering, missing requests and missing timestamps make the displayed totals partial. The page names the number of incomplete jobs and separately labels estimated end times. A partial total is not evidence of no resource use. Host usage is available through the user and team usage APIs; it is not yet converted into currency or a daily host-resource budget. ## Soft by default `TUNEPLANE_QUOTA_ENFORCE=false` records usage without refusing anything. That is useful while you work out what the numbers should be, and dangerous to leave on afterwards. Even with enforcement on, bare-metal Ray has no cgroup or GPU hard isolation, so admission is a gate applied *before* the job starts. The [runtime watchdog](https://docs.tuneplane.ai/en/ops/observability.md) is the second layer: it reconciles what the cluster is actually using against what the ledger says, and can stop a job that reliably exceeds its allocation. A hard quota needs infrastructure like Kubernetes or Kueue. ## Reading a refusal | Message names | Meaning | | - | - | | Concurrent GPUs | The user is at their ceiling. Not a cluster problem | | Storage quota | Run directories over budget. `tp clean` an exported experiment | | Daily GPU-hours | Today's budget is spent; it resets at midnight in the service timezone | | Profile not allowed | `allowed_profiles` does not include the requested one | | Cluster capacity | Genuinely no free cards. This one queues rather than refusing | The last row is the important distinction: **waiting in the queue is not a quota failure**. A queued job was admitted. ## Audit Creating a user, setting a role, enabling or disabling an account and setting a quota are all recorded in the [audit log](https://docs.tuneplane.ai/en/console/audit.md) with the administrator who did it. So is every submission that was refused on quota — which is what makes "why did my job not run" answerable after the error message has scrolled away. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.