Preprocess
common/data/prepare_*.py by convention and write a local directory of parquet
or jsonl.
Push a version
v2.
Use it from the console
- Open Data → Datasets. Upload files to publish a version, or open an existing dataset.
- The Use in a job section shows a copyable, versioned reference immediately. Open Usage guide, choose a version and Training or Validation, then choose a file from that version.
- Copy the generated flags into your existing
tp submitcommand. Keep your experiment, model, hardware profile and other required flags. - For your own preprocessing script, copy the Python path example. Its environment variable is available inside the job after the dataset is prepared.
@version.
Import selected Hugging Face shards
In Data → Datasets → Import from a hub, choose Hugging Face and enable Select config and split. Enter the names declared in the repository’s data card. The importer fixes the repository commit before copying the selected Parquet or JSONL files; their names, shard boundaries and embedded images remain unchanged. Use Record file patterns to narrow a declared split, for exampledata/train-*.parquet. If the data card has no configs declaration, explicit
patterns are required. This mode records your file selection and labels; it does
not infer the repository’s split layout. Each pattern must match a file.
For records that reference separate images, also enter Companion image
patterns, such as images/**/*.png. These paths are relative to the repository
root, which becomes the Dataset root. Embedded image bytes need no extra files.
The importer copies files without decoding images or executing dataset scripts.
Selected imports currently support public repositories, Parquet/JSONL record
files and explicit companion images, with an 8 GiB limit per file. Other formats
need preparation before upload. Private Hub credentials, automatic format
conversion and physical resharding are not available in this mode. Leave the
selection switch off to use the existing whole-repository import.
The import list and dataset origin show the selected configuration/split and the
resolved commit. Successful import creates a private dataset by default, but does
not imply a clean scan or grant training admission; run the required governance
checks on the new version.
CPU OCR for image governance
Image governance defaults togovernance_ocr_engine=rapidocr-v6 on CPU.
The worker needs the server’s ocr-v6 extra and a verified PP-OCRv6 small model
bundle. To build a prepared worker image from this checkout, use
docker build --build-arg BUILD_NATIVE=1 --build-arg BUILD_GOVERNANCE_OCR=1 -t governance-worker .
and configure its registry digest as governance_worker_image. The ordinary
control-plane build does not include OCR models. Explicit existing settings stay
in effect: rapidocr selects the legacy engine; an empty value disables OCR.
Upgrade the approved worker image before relying on the new default; a missing
bundle fails the job instead of silently choosing another engine.
Prepare the bundle with scripts/prepare_governance_ocr.py --output <directory>
on a connected build machine, then include it in the worker image at
/opt/tuneplane/ocr-v6. Keep the directory read-only and readable by the worker.
The worker refuses missing or changed models instead of downloading them.
The v6 profile limits ONNX and OpenCV to one thread and checks CPU providers.
JPEG, PNG, static WebP and supported TIFF OCR inputs honor EXIF orientation. Redaction regions
refer to original image pixels; derived images are oriented upright with metadata
removed. Invalid or incomplete OCR output cannot count as a completed inspection.
TIFF support is limited to classic single-page, unsigned 8-bit grayscale, RGB
or straight-alpha RGBA images, with at most 4096 strips/tiles. Multipage,
BigTIFF, pyramids, planar, grayscale-alpha and higher-bit-depth images remain
unsupported and cannot count as fully inspected. Image size/decode limits apply.
The native decoder shares oriented RGBA pixels with CPU OCR to avoid a second
image decode. Validate Chinese recognition and sensitive
region coverage against your own documents before enabling production policies;
the included Simplified/Traditional Chinese fixtures are regression checks.
Parquet governance reuses bounded Arrow batch projections and coalesces output
chunks before encoding. Typed untouched values, source row identities and
embedded image checks are preserved. Exact-duplicate scratch indexes store
compact binary fingerprints; report identities and survivor order stay unchanged.
These optimizations do not replace deployment-specific TB-scale acceptance.
Compare governance distributions
When starting an inspection, enable Collect distribution comparisons. Length is collected automatically. For language, task and source, optionally declare a metadata field path and up to 64 comma-separated category labels. For example, use/language with en, zh, fr. Undeclared values are grouped as Other;
missing/null values and unconfigured dimensions are shown separately.
After treatment completes, Distribution comparison shows source/output
counts, shares and changes in percentage points. Both JSONL and Parquet collect
these statistics during processing. Length measures mapped Unicode characters
per record, not tokens or OCR text. Empty denominators show a dash.
Language/task/source are declared metadata labels, not automatic classifications.
The comparison excludes quarantine releases and does not measure downstream
model quality or grant training admission. Older runs show that distributions
were not collected; start a new inspection with collection enabled to obtain them.
Scratch space for sources, outputs and indexes
Selected HF imports and governance Parquet readers reserve temporary-file space before downloading. A busy or nearly full scratch volume refuses new source files; existing checksum and publication checks still apply. Treated, quarantined and reviewed-release outputs share the same pool and acquire capacity as they grow. Administrators can setTUNEPLANE_SCRATCH_DIR to an absolute, private directory,
TUNEPLANE_SCRATCH_MAX_BYTES (default 64 GiB) and
TUNEPLANE_SCRATCH_MIN_FREE_BYTES (default 1 GiB). Set
TUNEPLANE_SCRATCH_OUTPUT_MAX_BYTES to bound each output file (default 8 GiB).
Empty outputs do not reserve their full maximum; capacity is acquired before
writes. Exceeding the output limit fails the operation without truncating records
or publishing a partial Dataset. Split the input or adjust the limits before retrying.
Processes sharing that directory
must use the same limits. Without this configuration, imports use their Hub cache
and governance readers/writers use their process temporary directory as separate pools.
Disposable findings, duplicate, asset and image-action indexes also reserve this
pool. Set TUNEPLANE_SCRATCH_INDEX_MAX_BYTES to their per-database ceiling (default
1 GiB). Unlike outputs, each index reserves its entire ceiling before loading
records. Unsafe queries requiring extra sorting files are refused. Size the pool
for all overlapping source files, indexes and outputs, or reduce concurrency.
For authorized governance workers on local Docker/Podman Fleets, explicitly
configuring the directory now provisions its shared mount and numeric limits
automatically. It must be a private directory owned by the service UID and contain
only scratch state. The directory must exist at the same path on the Docker host;
a containerized console needs a corresponding host bind. Workers run under the
service UID/GID and verify the pool identity before claiming work. Ordinary jobs
cannot request this writable mount through environment variables.
This does not provide OS filesystem quotas or whole-attempt reservation before
launch. Production mounts and representative TB-scale workloads still require
validation. Retry a refused import after capacity becomes available.
Reference it from a job
For scripts that accept platform metadata in their configuration, declare the reference inexperiments/<experiment>/config.yaml. Merge with an existing data
section rather than duplicating the YAML key. Strict trainer configs such as
TRL SFTConfig do not accept this section; use the CLI flags below instead.
Depending on the recipe, --train-data is still required to select the input file:
<NAME>_DATA_DIR. So --train-data is a path inside the dataset, not a path on your
machine.
To use a validation file from the same version, also add:
@v2 makes the input reproducible;
omitting the version resolves the latest at submission.
For a custom script, read the injected path at runtime:
input_file to your data loader. The variable uses the dataset name in upper
case, with - and . replaced by _, followed by _DATA_DIR; it includes neither
the owner nor the version. It is not a variable on your laptop. Write processed
outputs to $TUNEPLANE_OUT_DIR, or $TUNEPLANE_DATASET_OUT_DIR when the job declares
an output dataset.
The dataset manifest, presigned URLs included, is generated at submission and written into the
job package. The job side needs no object-storage credentials and no route to the console.
Write a card
A dataset with nothing but a name is a black box to everyone except whoever pushed it, and picking the wrong data costs hundreds of GPU-hours. Put aREADME.md in the directory and
tp dataset push publishes it with the data:
Manage
Why parquet previews better
Parquet’s footer carries the row count and the declared column types, and reading it costs a few kilobytes. So a parquet dataset shows an exact row count and real types —int64, string,
timestamp[us] — rather than types guessed from the first few rows.
jsonl and csv would need a full scan for a row count, so they show — instead of a guess.
The platform reads the footer and the first row group, with a byte cap. A file written with very
large row groups, which 64 to 128 MB per group commonly is, yields no sample rows; the columns
and the row count are still shown, with a note where the samples would be.
Column lengths in the preview are measured before truncation. Sample cells are cut to 160
characters on the way to the browser, so an average taken from the samples would always equal
the truncation cap — a number you cannot size max_seq_len with.
Catch contamination before you push
--fields narrows it to the question text: the same phrase appearing in an answer is not
contamination.
Matching uses normalised 13-grams, the window the GPT-3 and PaLM contamination analyses used.
Case, punctuation and full-width forms do not change the verdict. Chinese is shingled by
character, because splitting on spaces would make a whole sentence one token and find nothing.
Read the quality report
tp dataset push computes the same report and stores it with the version, where the console
shows it at the top of the dataset page. Run it by hand to look before you push.
It counts only what a machine can definitively count. Whether an answer is good is a judgement
for a person or a judge, which is preference annotation.
Near-duplicates use a small MinHash. Template-generated data — fifty records differing by one
number — is reported as duplicated, and that is not a false positive. It is the thing this report
exists to say out loud.
Compare two datasets without moving the bytes
tp dataset push computes an n-gram fingerprint alongside the checksums and uploads it with
the version. Push already reads every file once, so shingling there costs almost nothing;
computing it later would mean pulling gigabytes back out of object storage.
With fingerprints on both sides, comparing two datasets is a set intersection:
Fingerprint comparison is an estimate. Sampling allows false negatives, and a very small
dataset may have nothing sampled at all. For a small set use
tp dataset check, which names
rows rather than giving a ratio. A version pushed before fingerprints existed has none, and the
endpoint says so rather than reporting “no contamination”.Sensitive-value scanning
After a version is published, the platform scans it once and the result appears on the dataset page. It produces a report and never rewrites what you uploaded. Three things the report says that most scanners do not:- How many rows it read. A hundred-gigabyte dataset cannot be read on the way to a submission, so the scan reads bounded windows spread across each file. A sampled scan can never say “clean”, only “nothing in what I looked at”.
- A severity per finding. An API key, a private key or a checksum-valid identity number is
high. Email and phone aremedium. The generic long-token rule islow, because it hits every base64 blob and git SHA in a code corpus. - Nothing at all, when it has not run. An absent report renders as not scanned, never as clean.
TUNEPLANE_DATA_SCAN_POLICY=block) blocks only on high.
An administrator can add deployment-specific terms with TUNEPLANE_DATA_SCAN_DENY_TERMS, and
suppress known false positives with TUNEPLANE_DATA_SCAN_ALLOW_TERMS. Those are terms, not
patterns — the platform escapes and compiles them.
To actually clean a dataset, run a Processing Run: read it, treat it with
your own code, publish a new version.
Hub datasets work too
A Hugging Face or ModelScope id in the config works directly:Check it worked
tp dataset ls alice/gsm8k-zh lists the version you pushed, and the console’s Files tab shows
the file count and checksums. In a running job, echo $GSM8K_ZH_DATA_DIR prints the cache path.