> ## Documentation Index > Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt > Use this file to discover all available pages before exploring further. # Publish and use a dataset > Preprocess locally, push an immutable version, and reference it from a job by name. ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp dataset prepare gsm8k-zh # preprocess locally tp dataset push gsm8k-zh v2 ./out/gsm8k-zh # upload as v2 tp submit my-grpo --profile h200:8 \ --train-dataset alice/gsm8k-zh@v2 --train-data train.parquet ``` A platform dataset gives you three things a shared filesystem does not: an exact version a finished run can name, distribution into the cluster's cache without anybody copying files to nodes, and permission that travels with the data rather than with a directory. ## Preprocess ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp dataset prepare # the preprocessing scripts this project has tp dataset prepare gsm8k-zh # runs common/data/prepare_gsm8k_zh.py ``` Scripts live at `common/data/prepare_*.py` by convention and write a local directory of parquet or jsonl. ## Push a version ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp dataset push gsm8k-zh v1 ./out/gsm8k-zh # private tp dataset push gsm8k-zh v1 ./out/gsm8k-zh --public # anyone can reference it ``` Upload is chunked and streamed, checksummed with SHA-256, and resumable: re-running after an interruption skips what already landed. **Versions are immutable.** A version of the same name cannot be overwritten. Push `v2`. ## Use it from the console 1. Open **Data → Datasets**. Upload files to publish a version, or open an existing dataset. 2. The **Use in a job** section shows a copyable, versioned reference immediately. Open **Usage guide**, choose a version and **Training** or **Validation**, then choose a file from that version. 3. Copy the generated flags into your existing `tp submit` command. Keep your experiment, model, hardware profile and other required flags. 4. For your own preprocessing script, copy the Python path example. Its environment variable is available inside the job after the dataset is prepared. **Referenced by** lists past jobs. It is separate from the usage guide. Choose a **dataset** when runs need to name a fixed input version. Choose a **volume** for a directory whose contents change, including external data or model files. Both can be read by scripts; publishing a new dataset version does not change a reference that already includes `@version`. ## Import selected Hugging Face shards In **Data → Datasets → Import from a hub**, choose Hugging Face and enable **Select config and split**. Enter the names declared in the repository's data card. The importer fixes the repository commit before copying the selected Parquet or JSONL files; their names, shard boundaries and embedded images remain unchanged. Use **Record file patterns** to narrow a declared split, for example `data/train-*.parquet`. If the data card has no `configs` declaration, explicit patterns are required. This mode records your file selection and labels; it does not infer the repository's split layout. Each pattern must match a file. For records that reference separate images, also enter **Companion image patterns**, such as `images/**/*.png`. These paths are relative to the repository root, which becomes the Dataset root. Embedded image bytes need no extra files. The importer copies files without decoding images or executing dataset scripts. Selected imports currently support public repositories, Parquet/JSONL record files and explicit companion images, with an 8 GiB limit per file. Other formats need preparation before upload. Private Hub credentials, automatic format conversion and physical resharding are not available in this mode. Leave the selection switch off to use the existing whole-repository import. The import list and dataset origin show the selected configuration/split and the resolved commit. Successful import creates a private dataset by default, but does not imply a clean scan or grant training admission; run the required governance checks on the new version. ## CPU OCR for image governance Image governance defaults to `governance_ocr_engine=rapidocr-v6` on CPU. The worker needs the server's `ocr-v6` extra and a verified PP-OCRv6 small model bundle. To build a prepared worker image from this checkout, use `docker build --build-arg BUILD_NATIVE=1 --build-arg BUILD_GOVERNANCE_OCR=1 -t governance-worker .` and configure its registry digest as `governance_worker_image`. The ordinary control-plane build does not include OCR models. Explicit existing settings stay in effect: `rapidocr` selects the legacy engine; an empty value disables OCR. Upgrade the approved worker image before relying on the new default; a missing bundle fails the job instead of silently choosing another engine. Prepare the bundle with `scripts/prepare_governance_ocr.py --output ` on a connected build machine, then include it in the worker image at `/opt/tuneplane/ocr-v6`. Keep the directory read-only and readable by the worker. The worker refuses missing or changed models instead of downloading them. The v6 profile limits ONNX and OpenCV to one thread and checks CPU providers. JPEG, PNG, static WebP and supported TIFF OCR inputs honor EXIF orientation. Redaction regions refer to original image pixels; derived images are oriented upright with metadata removed. Invalid or incomplete OCR output cannot count as a completed inspection. TIFF support is limited to classic single-page, unsigned 8-bit grayscale, RGB or straight-alpha RGBA images, with at most 4096 strips/tiles. Multipage, BigTIFF, pyramids, planar, grayscale-alpha and higher-bit-depth images remain unsupported and cannot count as fully inspected. Image size/decode limits apply. The native decoder shares oriented RGBA pixels with CPU OCR to avoid a second image decode. Validate Chinese recognition and sensitive region coverage against your own documents before enabling production policies; the included Simplified/Traditional Chinese fixtures are regression checks. Parquet governance reuses bounded Arrow batch projections and coalesces output chunks before encoding. Typed untouched values, source row identities and embedded image checks are preserved. Exact-duplicate scratch indexes store compact binary fingerprints; report identities and survivor order stay unchanged. These optimizations do not replace deployment-specific TB-scale acceptance. ## Compare governance distributions When starting an inspection, enable **Collect distribution comparisons**. Length is collected automatically. For language, task and source, optionally declare a metadata field path and up to 64 comma-separated category labels. For example, use `/language` with `en, zh, fr`. Undeclared values are grouped as **Other**; missing/null values and unconfigured dimensions are shown separately. After treatment completes, **Distribution comparison** shows source/output counts, shares and changes in percentage points. Both JSONL and Parquet collect these statistics during processing. Length measures mapped Unicode characters per record, not tokens or OCR text. Empty denominators show a dash. Language/task/source are declared metadata labels, not automatic classifications. The comparison excludes quarantine releases and does not measure downstream model quality or grant training admission. Older runs show that distributions were not collected; start a new inspection with collection enabled to obtain them. ### Scratch space for sources, outputs and indexes Selected HF imports and governance Parquet readers reserve temporary-file space before downloading. A busy or nearly full scratch volume refuses new source files; existing checksum and publication checks still apply. Treated, quarantined and reviewed-release outputs share the same pool and acquire capacity as they grow. Administrators can set `TUNEPLANE_SCRATCH_DIR` to an absolute, private directory, `TUNEPLANE_SCRATCH_MAX_BYTES` (default 64 GiB) and `TUNEPLANE_SCRATCH_MIN_FREE_BYTES` (default 1 GiB). Set `TUNEPLANE_SCRATCH_OUTPUT_MAX_BYTES` to bound each output file (default 8 GiB). Empty outputs do not reserve their full maximum; capacity is acquired before writes. Exceeding the output limit fails the operation without truncating records or publishing a partial Dataset. Split the input or adjust the limits before retrying. Processes sharing that directory must use the same limits. Without this configuration, imports use their Hub cache and governance readers/writers use their process temporary directory as separate pools. Disposable findings, duplicate, asset and image-action indexes also reserve this pool. Set `TUNEPLANE_SCRATCH_INDEX_MAX_BYTES` to their per-database ceiling (default 1 GiB). Unlike outputs, each index reserves its entire ceiling before loading records. Unsafe queries requiring extra sorting files are refused. Size the pool for all overlapping source files, indexes and outputs, or reduce concurrency. For authorized governance workers on local Docker/Podman Fleets, explicitly configuring the directory now provisions its shared mount and numeric limits automatically. It must be a private directory owned by the service UID and contain only scratch state. The directory must exist at the same path on the Docker host; a containerized console needs a corresponding host bind. Workers run under the service UID/GID and verify the pool identity before claiming work. Ordinary jobs cannot request this writable mount through environment variables. This does not provide OS filesystem quotas or whole-attempt reservation before launch. Production mounts and representative TB-scale workloads still require validation. Retry a refused import after capacity becomes available. ## Reference it from a job For scripts that accept platform metadata in their configuration, declare the reference in `experiments//config.yaml`. Merge with an existing `data` section rather than duplicating the YAML key. Strict trainer configs such as TRL SFTConfig do not accept this section; use the CLI flags below instead. Depending on the recipe, `--train-data` is still required to select the input file: ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} data: train: dataset: alice/gsm8k-zh@v2 # omit the version to take the latest file: train.parquet ``` Or override for one submission: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp submit my-grpo --profile h200:8 \ --train-dataset alice/gsm8k-zh@v2 --train-data train.parquet ``` At job startup the platform pulls the dataset into the cluster's shared cache, verifies it, and injects `_DATA_DIR`. So `--train-data` is a path *inside* the dataset, not a path on your machine. To use a validation file from the same version, also add: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} --validation-dataset alice/gsm8k-zh@v2 --validation-data validation.parquet ``` Use a path that actually appears in that version. The console guide offers the files from the version you select. Pinning `@v2` makes the input reproducible; omitting the version resolves the latest at submission. For a custom script, read the injected path at runtime: ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} import os from pathlib import Path dataset_dir = Path(os.environ["GSM8K_ZH_DATA_DIR"]) input_file = dataset_dir / "train.parquet" ``` Pass `input_file` to your data loader. The variable uses the dataset name in upper case, with `-` and `.` replaced by `_`, followed by `_DATA_DIR`; it includes neither the owner nor the version. It is not a variable on your laptop. Write processed outputs to `$TUNEPLANE_OUT_DIR`, or `$TUNEPLANE_DATASET_OUT_DIR` when the job declares an output dataset. The dataset manifest, presigned URLs included, is generated at submission and written into the job package. The job side needs no object-storage credentials and no route to the console. ## Write a card A dataset with nothing but a name is a black box to everyone except whoever pushed it, and picking the wrong data costs hundreds of GPU-hours. Put a `README.md` in the directory and `tp dataset push` publishes it with the data: ```markdown theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} # gsm8k-zh Chinese math problems rewritten from GSM8K, deduplicated by question text. ## Columns | Column | Type | Description | | --- | --- | --- | | `question` | string | The problem statement | | `answer` | string | Reference solution, including the reasoning | ## Provenance Produced by `common/data/prepare_gsm8k_zh.py`. 200 samples spot-checked by hand after translation. ``` Update the card without pushing data again: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp dataset card alice/gsm8k-zh -f README.md ``` The first paragraph becomes the one-line description on every list row. The body renders on the dataset's Card tab. ## Manage ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp dataset ls # public datasets plus your own tp dataset ls alice/gsm8k-zh -v v2 # one version's file list tp dataset visibility alice/gsm8k-zh --public # owner or admin ``` The console's [Data page](https://docs.tuneplane.ai/en/console/data.md) lists description, latest version, file count, size, formats and update time on every row. Opening one gives four tabs: | Tab | The question it answers | | - | - | | Card | What this data is, which columns, where it came from | | Files | What this version contains, how large, with which checksums | | Preview | Column types, how many empties, roughly how long the text is | | Referenced by | Which jobs reference it and which version each used | ## Why parquet previews better Parquet's footer carries the row count and the declared column types, and reading it costs a few kilobytes. So a parquet dataset shows an exact row count and real types — `int64`, `string`, `timestamp[us]` — rather than types guessed from the first few rows. jsonl and csv would need a full scan for a row count, so they show `—` instead of a guess. The platform reads the footer and the first row group, with a byte cap. A file written with very large row groups, which 64 to 128 MB per group commonly is, yields no sample rows; the columns and the row count are still shown, with a note where the samples would be. Column lengths in the preview are measured **before** truncation. Sample cells are cut to 160 characters on the way to the browser, so an average taken from the samples would always equal the truncation cap — a number you cannot size `max_seq_len` with. ## Catch contamination before you push ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp dataset check ./out/my-train --against ./eval/gsm8k-test.jsonl tp dataset check ./out/my-train --against ./eval/test.jsonl --fields question ``` This guards against accidentally training on the test set: slicing train and test from the same public dataset, or picking up somebody's cleaned mixture that happens to contain GSM8K. The damage is a score that is inflated and nobody notices. It names the rows, because the action you need is "delete these": ```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} ! 3 / 12000 training rows overlap the eval set (0.0%) · Natalia sold clips to 48 of her friends in April, and then she sold… → These rows inflate your eval scores. Remove them, or use a different eval set. ``` It exits 1 when there is overlap, so it drops into CI and a contaminated push gets stopped. `--fields` narrows it to the question text: the same phrase appearing in an answer is not contamination. Matching uses normalised 13-grams, the window the GPT-3 and PaLM contamination analyses used. Case, punctuation and full-width forms do not change the verdict. Chinese is shingled by character, because splitting on spaces would make a whole sentence one token and find nothing. ## Read the quality report ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp dataset quality ./out/my-train ``` ```text theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} 12000 records duplicates 842 (7.0%), of which 310 byte-identical empty 0 (0.0%) text length median 180 · p95 1400 · longest 9000 characters fields answer, prompt ! duplicate rate 7.0% is high → the model will overfit to the repeated samples. Deduplicate before pushing, or confirm this is a deliberate sampling weight. ``` `tp dataset push` computes the same report and stores it with the version, where the console shows it at the top of the dataset page. Run it by hand to look before you push. | Metric | What it guards against | | - | - | | Duplicate rate | A training set that is 30% repeats overfits to those repeats, and nothing else here would show you that | | Length p95 | Whether `max_seq_len` is enough is decided by p95, not the mean. The mean is dragged down by short samples, and 5% of the data gets truncated silently | | Empty records | Wholly empty samples dilute the gradient | | Field drift | Half the records have an `answer` field and half do not, and the half that does not reads `None` during training | It counts only what a machine can definitively count. Whether an answer is *good* is a judgement for a person or a judge, which is [preference annotation](https://docs.tuneplane.ai/en/guides/annotation.md). Near-duplicates use a small MinHash. Template-generated data — fifty records differing by one number — is reported as duplicated, and that is not a false positive. It is the thing this report exists to say out loud. ## Compare two datasets without moving the bytes `tp dataset push` computes an n-gram **fingerprint** alongside the checksums and uploads it with the version. Push already reads every file once, so shingling there costs almost nothing; computing it later would mean pulling gigabytes back out of object storage. With fingerprints on both sides, comparing two datasets is a set intersection: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} curl -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN" \ "https://tuneplane.your-company.com/api/datasets/alice/gsm8k-zh/contamination?against=alice/gsm8k-eval" ``` For data that does not live on the platform, such as a benchmark's test split: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} tp dataset fingerprint ./gsm8k-test.jsonl -o gsm8k.fingerprint.json ``` > [!NOTE] > > Fingerprint comparison is an **estimate**. Sampling allows false negatives, and a very small > dataset may have nothing sampled at all. For a small set use `tp dataset check`, which names > rows rather than giving a ratio. A version pushed before fingerprints existed has none, and the > endpoint says so rather than reporting "no contamination". ## Sensitive-value scanning After a version is published, the platform scans it once and the result appears on the dataset page. It produces a **report** and never rewrites what you uploaded. Three things the report says that most scanners do not: * **How many rows it read.** A hundred-gigabyte dataset cannot be read on the way to a submission, so the scan reads bounded windows spread across each file. A sampled scan can never say "clean", only "nothing in what I looked at". * **A severity per finding.** An API key, a private key or a checksum-valid identity number is `high`. Email and phone are `medium`. The generic long-token rule is `low`, because it hits every base64 blob and git SHA in a code corpus. * **Nothing at all, when it has not run.** An absent report renders as *not scanned*, never as clean. A deployment that blocks on findings (`TUNEPLANE_DATA_SCAN_POLICY=block`) blocks only on `high`. An administrator can add deployment-specific terms with `TUNEPLANE_DATA_SCAN_DENY_TERMS`, and suppress known false positives with `TUNEPLANE_DATA_SCAN_ALLOW_TERMS`. Those are terms, not patterns — the platform escapes and compiles them. To actually clean a dataset, run a [Processing Run](https://docs.tuneplane.ai/en/guides/volumes.md): read it, treat it with your own code, publish a new version. ## Hub datasets work too A Hugging Face or ModelScope id in the config works directly: ```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}} data: train: dataset: hf:nvidia/OpenMathInstruct-2 ``` The platform runs a preflight at submission, and a gated dataset checks your authorisation. Use a platform dataset for internal data and for preprocessed derivatives, where you do not want to push to a public hub and permissions should stay inside. ## Check it worked `tp dataset ls alice/gsm8k-zh` lists the version you pushed, and the console's Files tab shows the file count and checksums. In a running job, `echo $GSM8K_ZH_DATA_DIR` prints the cache path. ## Next [Data shapes and splits](https://docs.tuneplane.ai/en/guides/data-formats.md) · [Volumes](https://docs.tuneplane.ai/en/guides/volumes.md) for material a job may read but not download · [Annotation](https://docs.tuneplane.ai/en/guides/annotation.md) · [Submitting a job](https://docs.tuneplane.ai/en/guides/submit.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.