Learn AI Create the Future — with Pranay
Main site

Observability for LLM Applications

Redacting personal data from LLM logs

Redaction means finding and blacking out personal details before a log is stored anywhere, because every trace, prompt and feedback comment in this section can carry real personal data.

Read these first

On this page 8
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. A real example you have seen
  6. The honest part
  7. Remember this
  8. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Redaction means finding personal details in text and blacking them out before it gets stored anywhere.

PII stands for personally identifiable information. It is anything that could point back to one real person: a name, phone number, email, or ID number.

The analogy you have already lived

Getting an Aadhaar card photocopied for a form often means the shopkeeper blacks out most of the number first. Only the last four digits stay visible.

The copy is still useful — it still proves who you are. It no longer exposes your entire number to whoever ends up holding that photocopy later.

Redacting a log does the exact same thing to a stored conversation, before it sits on a server for months.

Why it exists

Every lesson so far in this section has logged something: a trace, a prompt, a rating, a comment. Real conversations that pass through those systems routinely contain phone numbers, emails, ID numbers, and addresses.

Storing all of that in plain text, forever, in logs many engineers can query, is a real risk to real people. In many places, it is a real legal obligation to avoid too.

How it works

   raw text            "My number is 9876543210 and my Aadhaar
                         is 2345 6789 1234."
        |
        v
   [ find personal data patterns ]
        |
        v
   redacted text        "My number is [REDACTED_PHONE] and my Aadhaar
                          is [REDACTED_AADHAAR]."

The redacted version keeps the shape of the conversation useful for debugging, while removing the specific personal details from it.

A real example you have seen

A support team debugging a chatbot bug needs to read real conversations to understand what went wrong. They should never need to see a customer's actual card number to do that — the bug is almost never about the number itself.

The honest part

Automatic redaction is never perfect. A pattern-based approach, the kind shown in the developer tab, catches things that follow a predictable shape — phone numbers, emails, ID numbers.

It does not reliably catch a name typed in free text, or an address written in a sentence. Treat redaction as risk reduction, not a guarantee, and combine it with limiting who can read raw logs at all.

Remember this

  • PII is anything that could identify one specific real person.
  • Redaction blacks out personal details while keeping the rest of the log useful.
  • Pattern-based redaction catches structured data well, and free-text names poorly.

What to learn next

Developer — Code and libraries.

Setup

No install needed. re ships in Python's standard library.

A pattern-based redactor

Four patterns, applied in a deliberate order, so a longer number (a card) is not partially swallowed by a shorter pattern (Aadhaar) matching first.

redact.py
import re

PATTERNS = {
    "EMAIL": re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"),
    "PHONE": re.compile(r"\b(?:\+91[- ]?)?[6-9]\d{9}\b"),
    "AADHAAR": re.compile(r"\b\d{4}\s?\d{4}\s?\d{4}\b"),
    "CARD": re.compile(r"\b(?:\d{4}[- ]?){3}\d{4}\b"),
}

# Run the more specific patterns before the more general ones, so a card
# number is not partially caught by the Aadhaar pattern first.
ORDER = ["EMAIL", "CARD", "AADHAAR", "PHONE"]


def redact(text):
    for label in ORDER:
        text = PATTERNS[label].sub(f"[REDACTED_{label}]", text)
    return text


transcript = (
    "Hi, I'm Ramesh. My email is ramesh.k99@gmail.com and my number is "
    "9876543210. My Aadhaar is 2345 6789 1234 in case you need it for "
    "verification, and I tried paying with card 4111 1111 1111 1111 but it failed."
)

print("BEFORE:")
print(transcript)
print("\nAFTER:")
print(redact(transcript))
Output
BEFORE:
Hi, I'm Ramesh. My email is ramesh.k99@gmail.com and my number is 9876543210. My Aadhaar is 2345 6789 1234 in case you need it for verification, and I tried paying with card 4111 1111 1111 1111 but it failed.

AFTER:
Hi, I'm Ramesh. My email is [REDACTED_EMAIL] and my number is [REDACTED_PHONE]. My Aadhaar is [REDACTED_AADHAAR] in case you need it for verification, and I tried paying with card [REDACTED_CARD] but it failed.

Exact, deterministic output — every pattern here is a fixed regular expression run against fixed text. Notice what did not get redacted: the name "Ramesh". That gap is real, and it is exactly why the honest part above matters.

Line-by-line walkthrough

ORDER matters more than it looks. Running AADHAAR before CARD would let the twelve-digit Aadhaar pattern eat the first twelve digits of the sixteen-digit card number, leaving four stray digits behind, unredacted and confusing.

The \b word boundaries stop a pattern from matching in the middle of a longer, unrelated number, keeping each pattern honest about what it actually claims to catch.

Each PATTERNS[label].sub(...) pass is independent and simple on purpose. A production redactor adds many more patterns over time, as new kinds of personal data get spotted in real logs.

Common mistakes

Assuming regex catches everything personal. As shown above, a plain name in free text sails through untouched. For a system that truly needs name detection, pair regex with a named-entity recognition model, and still expect imperfect recall.

Redacting logs after they are already stored raw. Retroactively cleaning existing logs is real work, and every day of delay is a day of exposure. Redact before the first write, not after.

Redacting so aggressively that debugging becomes impossible. A log with every number blacked out cannot help debug a payment failure, either. Keep enough structure (the last four digits, the redaction type) for the log to remain useful.

Applying the same redaction rules to every field, regardless of purpose. A request_id should never be redacted. A free-text feedback comment often needs the most redaction of any field in the whole system.

Try it yourself

Add a NAME heuristic that redacts any capitalised word directly following "I'm" or "My name is". Test it on the transcript above, and try to break it with a sentence where it wrongly redacts something else.

What to learn next

Researcher — Mathematics and papers.

Regex versus NER for PII detection

Pattern-based detection (the developer example) has high precision and low recall on unstructured personal data: it almost never redacts something that is not real PII of the targeted kind, and it almost always misses a name, address, or other entity with no fixed format. Named entity recognition (NER) models trade this the other way — higher recall on free-text entities, at the cost of occasional false positives and a real inference cost per log line.

Production PII detection typically layers both: fast regex for structured, high-confidence patterns (card numbers, emails, government ID formats), and a NER pass (spaCy, Presidio, or a fine-tuned transformer) for names, addresses, and organisations that regex cannot express.

Microsoft Presidio's design, as a reference architecture

Presidio (an open-source PII detection and anonymisation toolkit) separates recognizers (pattern-based or model-based detectors, each scoped to one entity type) from an anonymizer (the redaction, masking, or synthetic-replacement step applied once entities are found), and assigns each detected entity a confidence score rather than a binary match — allowing a system to redact high-confidence matches automatically while routing low-confidence ones to human review.

Regulatory context

Several regulatory frameworks bear directly on LLM log retention, independent of any specific jurisdiction: purpose limitation (data collected for one purpose should not be repurposed without basis), storage limitation (personal data should not be kept longer than needed), and data minimisation (collect and retain the least data sufficient for the purpose). India's Digital Personal Data Protection Act (2023) and the EU's GDPR both encode versions of these principles; a log redaction policy is one concrete mechanism for satisfying data minimisation specifically, though not a substitute for a retention policy or an access-control policy.

Reversible versus irreversible redaction

Irreversible redaction (as in the developer example, replacing text with a fixed token) destroys information permanently, appropriate for logs with no legitimate future need for the original value. Tokenisation (replacing a value with a reversible token, mapped back via a separately-secured lookup table) preserves the ability to recover the original value under strict access control, useful when a legitimate downstream need — a fraud investigation, a legal request — might require the real value later, at the cost of maintaining that secured mapping table as its own security-critical system.

Papers and tools

  • Lison et al., Anonymisation Models for Text Data: State of the Art, Challenges and Future Directions, ACL 2021 — aclanthology.org/2021.acl-long.323, a survey of the regex-versus-NER trade-off at more depth than fits here.
  • Microsoft Presidio documentation — microsoft.github.io/presidio, the reference architecture described above.
  • Carlini et al., Extracting Training Data from Large Language Models, USENIX Security 2021 — arxiv.org/abs/2012.07805, a reminder that unredacted personal data in logs used for future fine-tuning can resurface in a model's own outputs later.

What to learn next