Lesson 7 · 8 min
Privacy in observability — what to redact, what to keep
Traces are gold for debugging and a privacy disaster waiting to happen. The redaction rules that keep both true.
What you keep
- Token counts, costs, latencies — never PII.
- Hashed user/tenant identifiers — group without identify.
- Model + prompt version — you need this for postmortems.
- Structural shape of inputs (length, language, has-attachment) — without the literal text.
- Outcome signals (thumbs-up, schema-ok, refusal) — not the output content.
What you redact at write time
- Raw user input — store a redacted version (regex-strips emails, phones, IBANs, CCNs) by default.
- Raw model output — same redaction pass.
- Retrieved chunks — redact in the same pass; chunks often contain user-uploaded content.
- Tool arguments and results — these are often the highest-PII zone.
The rule: structured metadata is fine; literal user content is gated.