> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuneplane.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# tp dataset

> Preprocess, push, describe and inspect a dataset version.

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp dataset push gsm8k-zh v2 ./out/gsm8k-zh
```

Uploads a directory as an immutable version. The walkthrough is
[publishing and using a dataset](https://docs.tuneplane.ai/en/guides/datasets.md).

## Worked examples

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
# preprocess locally
tp dataset prepare                          # the scripts this project has
tp dataset prepare gsm8k-zh                 # runs common/data/prepare_gsm8k_zh.py

# push a version
tp dataset push gsm8k-zh v2 ./out/gsm8k-zh
tp dataset push gsm8k-zh v2 ./out/gsm8k-zh --public
tp dataset push alice/gsm8k-zh v2 ./out --readme CARD.md

# describe it
tp dataset card alice/gsm8k-zh              # print the current card
tp dataset card alice/gsm8k-zh -f README.md # update it without pushing data

# inspect before pushing
tp dataset check ./out/my-train --against ./eval/gsm8k-test.jsonl
tp dataset check ./out/my-train --against ./eval/test.jsonl --fields question
tp dataset quality ./out/my-train
tp dataset fingerprint ./gsm8k-test.jsonl -o gsm8k.fingerprint.json

# list and share
tp dataset ls
tp dataset ls alice/gsm8k-zh -v v2          # one version's file manifest
tp dataset visibility alice/gsm8k-zh --public
tp dataset visibility alice/gsm8k-zh --private
```

## Notes the flag table cannot carry

* `DATASET` without an owner lands in your own namespace. Only an administrator can push across
  namespaces.
* `VERSION` is immutable. A version of the same name cannot be overwritten; push a new one.
* `--readme` defaults to `README.md` inside `PATH` when there is one. It becomes the dataset
  **card**, which belongs to the dataset rather than to one version.
* `tp dataset check` exits 1 when there is overlap, so it drops into CI as a gate.
* Upload is chunked, checksummed per file, and resumable: re-running after an interruption skips
  what already landed.

## Check it worked

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp dataset ls alice/gsm8k-zh
```

The new version appears with its file count and size. Then reference it:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp submit my-grpo --profile h200:8 \
  --train-dataset alice/gsm8k-zh@v2 --train-data train.parquet
```

## `tp dataset`

Manage datasets

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp dataset <subcommand>
```

| Command | Description |
| - | - |
| `tp dataset card` | Show/update a dataset description (README.md) |
| `tp dataset check` | Check training data for evaluation overlap |
| `tp dataset fingerprint` | Compute a dataset fingerprint |
| `tp dataset ls` | List the visible datasets |
| `tp dataset prepare` | Preprocess a dataset locally |
| `tp dataset push` | Upload one dataset version |
| `tp dataset quality` | Produce a dataset quality report |
| `tp dataset visibility` | Change a dataset's visibility (owner or admin) |

### `tp dataset card`

Show/update a dataset description (README.md)

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp dataset card <DATASET> [options]
```

| Argument | Description |
| - | - |
| `DATASET` | Dataset id (\<owner>/\<name>) (**required**) |

| Option | Description |
| - | - |
| `--file`, `-f` | Update the description from this markdown file; omit to print the current one |

### `tp dataset check`

Check training data for evaluation overlap

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp dataset check <PATH> [options]
```

| Argument | Description |
| - | - |
| `PATH` | Training data directory or file (jsonl / json / csv / parquet) (**required**) |

| Option | Description |
| - | - |
| `--against`, `-a` | Evaluation set directory or file; the question is whether its questions leaked into training (**required**) |
| `--fields` | Only these fields (comma separated), e.g. question,prompt. Defaults to every string field |

### `tp dataset fingerprint`

Compute a dataset fingerprint

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp dataset fingerprint <PATH> [options]
```

| Argument | Description |
| - | - |
| `PATH` | Data directory or file (**required**) |

| Option | Description |
| - | - |
| `--out`, `-o` | Write it to this file; omit to print a summary |
| `--fields` | Only these fields (comma separated) |

### `tp dataset ls`

List the visible datasets

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp dataset ls [DATASET] [options]
```

| Argument | Description |
| - | - |
| `DATASET` | Dataset id (\<owner>/\<name>); omit to list everything visible |

| Option | Description |
| - | - |
| `--version`, `-v` | Show one version's file list; defaults to the latest |

### `tp dataset prepare`

Preprocess a dataset locally

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp dataset prepare [DATASET]
```

| Argument | Description |
| - | - |
| `DATASET` | Dataset name; leave empty to list the options |

No options beyond `--help`.

### `tp dataset push`

Upload one dataset version

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp dataset push <DATASET> <VERSION> <PATH> [options]
```

| Argument | Description |
| - | - |
| `DATASET` | Dataset name (lands in your own namespace), or a full \<owner>/\<name> (an admin can cross namespaces) (**required**) |
| `VERSION` | Version, e.g. v1 / 20260812 (**required**) |
| `PATH` | Local directory (**required**) |

| Option | Description |
| - | - |
| `--public` | Make it public on first creation (anyone can reference it in training) |
| `--readme` | Dataset description (markdown); defaults to the README.md in the directory |
| `--concurrency` `<int>` | Uploads in flight. Left alone, it is chosen from the size of what you are sending; the right number depends on the link, which nothing here can see (default `0`) |

### `tp dataset quality`

Produce a dataset quality report

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp dataset quality <PATH> [options]
```

| Argument | Description |
| - | - |
| `PATH` | Data directory or file (**required**) |

| Option | Description |
| - | - |
| `--fields` | Only these fields (comma separated) |

### `tp dataset visibility`

Change a dataset's visibility (owner or admin)

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
tp dataset visibility <DATASET> [options]
```

| Argument | Description |
| - | - |
| `DATASET` | Dataset id (\<owner>/\<name>) (**required**) |

| Option | Description |
| - | - |
| `--public` | Make it public |
| `--private` | Make it private |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.