Turn production traffic into training data

curl -X PUT https://tuneplane.your-company.com/api/model-deployments/<id>/reflow \
  -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sample_rate": 0.05, "retention_days": 30, "capture_system_prompt": true}'

That turns capture on for 5% of requests. The console’s Deployments page has the same three settings under a Reflow tab, which is where most people set them. Reflow is the governed path from a deployment’s traffic back to the training data of its next version. It closes the loop the platform otherwise leaves open: data goes in, a model comes out, it serves, and what it learns from serving goes nowhere.

What is captured is real user input, and it is the most sensitive data this platform will ever hold. Capture is off until somebody turns it on, that person is recorded, and every change to these settings is audited.

The shape of it

traffic → Reflow Buffer → mining → Candidate Set → [ you review ] → [ you promote ] → Dataset version
          (expires)                 (replaced)       (audited)       (audited)        (immutable)

Only the last step is irreversible. Everything before it expires or gets replaced, which is what makes it safe to be wrong about sampling, redaction or mining. A direct deployment captures nothing. A proxied one is captured at the gateway, and an external one reports its own traffic — see deployments.

The three settings

Setting	Default	What it decides
`sample_rate`	`0.0`, off	Sampling is decided before a request is issued, so an unsampled request costs nothing
`retention_days`	`30`	Expiry is real deletion. There is no archive tier
`capture_system_prompt`	`true`	Often your own prompt engineering, and the most sensitive field here. Leaving it out makes records weaker for training, because the behaviour was conditioned on text the record would be missing

Buffer bytes count against the owner’s storage quota, and the quota wins over retention: over the limit, the oldest buffer files are reclaimed early and the loss is counted. A full shared disk stops training for everyone on the cluster. Never captured: request headers of any kind, so an Authorization header cannot arrive by accident; the deployment token itself, only its id, which is the tenant boundary inside one deployment; and anything but chat completions.

Redaction

Masking runs before anything reaches disk. Redaction that runs afterwards is not redaction, it is filtering. The default pack covers emails, phone numbers in mainland-China and E.164 forms, US SSNs, PRC resident ids, payment-card-shaped digits and key-shaped strings. It is versioned, and the version is stamped on every record, so you can tell later which rules cleaned what. It errs toward masking too much. A long hex string in a code sample is masked as a possible key. That costs a slightly worse training example; the opposite error puts a credential in a corpus.

Report traffic from a service you run elsewhere

curl -X POST https://tuneplane.your-company.com/inference/<id>/reflow/records \
  -H "Authorization: Bearer $DEPLOYMENT_TOKEN" \
  -d '{"records": [{
    "id": "log-8841", "user": "<subject>", "model": "qwen2.5-32b", "latency_ms": 812,
    "request":  {"messages": [{"role": "user", "content": "..."}]},
    "response": {"choices": [{"message": {"content": "..."}, "finish_reason": "stop"}]},
    "feedback": {"rating": "down", "correction": "..."}
  }]}'

For an external deployment: the gateway never sees this traffic, so the service reports it, in the OpenAI chat-completions shape its own log already has. A reported record joins the same redaction, buffer, retention, mining and review as one the gateway captured, and nothing downstream can tell them apart. At most 500 records per request; the response counts what was accepted, sampled out, dropped and refused, and the four always sum to the batch.

Field	What it is
`request`, `response`	The request body and the completion object. `request.messages` is required; `{"text": ...}` is accepted for the response
`user`	Who asked, as your identity provider names them. Stored only as a digest, so two people asking the same thing are not read as one person retrying
`model`	The model that answered. It stands where a revision would, and a promoted row carries it
`feedback`	Optional `rating` (`up`, `down`, or 0 to 1), `correction`, `comment`

Feedback is evidence, never a label. A complaint makes the record a candidate, and a reported correction is shown to the reviewer as a starting point; neither reaches a dataset until somebody on this side has read it. With the sample rate at 0 a report is refused rather than dropped, so a misconfigured integration finds out. A record carrying feedback is kept whatever the rate.

Mine the hard cases

curl -X POST https://tuneplane.your-company.com/api/reflow/<id>/mine \
  -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN" \
  -d '{"rubric": "alice/answer-quality", "threshold": 0.5, "limit": 200}'

The useful half of a buffer is small, and it is not the average request.

Signal	What it means
Request failed	The cheapest possible flag
Answer truncated	The model hit the length ceiling, so it did not finish
Malformed tool call	Arguments that are not valid JSON, or a tool the request never declared
User complained	A reported rating below the midpoint, or a correction. The only signal that is a statement rather than an inference
User retried	The same prompt again, minutes later. The customer already made this judgement
Scored low	A rubric you name scores what the free signals missed. The only signal that spends judge tokens

rubric is optional; leave it out and only the free signals are used. Mining replaces the previous candidate set: it is a working suggestion, not a ledger.

Review

curl https://tuneplane.your-company.com/api/reflow/<id>/candidates/41/exchange \
  -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN"

curl -X POST https://tuneplane.your-company.com/api/reflow/<id>/candidates/review \
  -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN" \
  -d '{"candidate_ids": [41], "disposition": "accepted"}'

Every signal above marks an answer as suspect, and a promoted record ends in that answer as the assistant turn — the target a fine-tune learns to reproduce. A candidate is therefore a suspected wrong answer until somebody has read it, and promotion takes nothing else.

Disposition	What it means
`pending`	Nobody has looked. Where every candidate starts
`accepted`	You read the exchange and the answer is right as it stands
`corrected`	You read it and wrote the answer it should have been
`discarded`	You read it and it should not become data

A signal is only a suspicion: a user who pastes a prompt twice did not necessarily dislike the first answer, and a judge has false positives. Accepting is how those are told apart. Reading an exchange is audited, a decision records who made it, and it survives the next mining run as long as the record is flagged for the same reason. Administrators are not exempt — a role does not know whether an answer is right.

Correct

curl -X POST https://tuneplane.your-company.com/api/reflow/<id>/candidates/41/correction \
  -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN" \
  -d '{"text": "A latch is level-sensitive; a flip-flop samples on a clock edge."}'

The hard cases mining finds are the ones the model got wrong, so accepting or discarding alone throws most of them away. A correction keeps the prompt production traffic supplied and replaces the answer. It also gives a preference pair for free — your answer chosen, the model’s own rejected — with no second annotation pass. A correction is stored beside the captured traffic it answers, never in the database, and expires with it: promote what you want to keep. The buffer is redacted when it is written, so a correction that introduces a sensitive value the exchange does not already contain is refused. An empty correction is refused too (discard instead), as is one identical to the original answer (accept instead).

Promote

curl -X POST https://tuneplane.your-company.com/api/reflow/<id>/promote \
  -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN" \
  -d '{
    "candidate_ids": [41, 42, 57],
    "dataset": "alice/support-reflow",
    "against": ["alice/support-eval"],
    "note": "Q3 truncations and retries"
  }'

Four checks, all fail-closed:

Review. Every selected candidate must be accepted or corrected. One that is not refuses the whole promotion rather than being skipped, so a version is never quietly smaller than the selection.
Visibility. A deployment is readable by its owner and by administrators, so the only non-escalating destination is that owner’s own private dataset. Promoting into a public dataset publishes your users’ traffic to every account on the platform. An administrator can do it, and the audit record says so.
Contamination. Name the evaluation datasets in against, or set skip_contamination_check explicitly. The choice is recorded on the version, because nobody checked and checked and clean must not look alike six months later. Production traffic overlapping an evaluation set is the normal case — that set was probably built from the same distribution.
Not empty. A version that exists, is referenceable and teaches nothing is worse than no version.

shape chooses what is written, and one version holds one shape. sft (the default) writes every selected candidate as a conversation ending in the answer a person vouched for: the original when accepted, the correction when corrected. preference writes only the corrected candidates as prompt / chosen / rejected; an accepted candidate has no pair to give and is left out. Every row carries a source — the record, the deployment, the revision that produced the answer, when, why it was flagged and who reviewed it — so one wrong row can be traced to the model that wrote it. The contamination estimate uses 13-gram fingerprints sampled at 1/64. A small promotion of short prompts produces too few n-grams for the estimate to mean anything, and is reported inconclusive rather than as a pass.

Check it worked

curl -H "Authorization: Bearer $TUNEPLANE_CLIENT_TOKEN" \
  https://tuneplane.your-company.com/api/reflow/<id>

That returns the buffer’s size, its retention clock, and the counters for what was dropped and trimmed. What was lost is as visible as what was kept, on purpose. After a promotion, tp dataset ls alice/support-reflow shows the new version, and the next run can reference it.

See the whole loop

Models → expand a model → Data flywheel. It shows one version’s chain: sources, the training run, the evaluation verdict, serving and captured volume, and the datasets that came back. “Closed” is strict there. Capturing traffic is not closing the loop, and neither is mining it. The loop closes when the traffic became a dataset somebody can train on.

Deployments · Datasets · Rubrics

Get started

Core concepts

Train a model

Evaluate and serve

Agent environments

Extend the platform

Collaborate

Integrations

Reference

Help

Turn production traffic into training data

The shape of it

The three settings

Redaction

Report traffic from a service you run elsewhere

Mine the hard cases

Review

Correct

Promote

Check it worked

See the whole loop

Next

​The shape of it

​The three settings

​Redaction

​Report traffic from a service you run elsewhere

​Mine the hard cases

​Review

​Correct

​Promote

​Check it worked

​See the whole loop

​Next

The shape of it

The three settings

Redaction

Report traffic from a service you run elsewhere

Mine the hard cases

Review

Correct

Promote

Check it worked

See the whole loop

Next