On this tutorial, we construct an entire, production-style pipeline for detecting and redacting personally identifiable data utilizing the OpenAI Privacy Filter. We start by organising the atmosphere and loading a token classification mannequin that identifies a number of classes of delicate information, together with names, emails, cellphone numbers, addresses, and secrets and techniques. We then design helper capabilities to normalize labels, extract structured spans, and rework uncooked mannequin outputs into usable codecs. From there, we implement a configurable redaction system that permits us to exchange delicate entities with significant placeholders, preserving privateness and offering contextual readability. All through the method, we take a look at the pipeline on curated examples, convert outputs into structured dataframes, and put together the system for batch processing and real-world utilization.
!pip set up -q -U transformers speed up torch pandas matplotlib huggingface_hub
import os, re, json, time, textwrap, warnings
from pathlib import Path
from collections import Counter
import pandas as pd
import matplotlib.pyplot as plt
import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline
warnings.filterwarnings("ignore")
MODEL_ID = "openai/privacy-filter"
OUT_DIR = Path("/content material/privacy_filter_outputs")
OUT_DIR.mkdir(mother and father=True, exist_ok=True)
machine = 0 if torch.cuda.is_available() else -1
torch_dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
print("System:", "GPU" if torch.cuda.is_available() else "CPU")
print("Torch dtype:", torch_dtype)
print("Mannequin:", MODEL_ID)We set up all required libraries and arrange the pipeline’s runtime atmosphere. We configure machine choice and initialize paths for storing outputs. We additionally print system particulars to verify that all the things is prepared earlier than loading the mannequin.
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
mannequin = AutoModelForTokenClassification.from_pretrained(
MODEL_ID,
torch_dtype=torch_dtype,
device_map="auto" if torch.cuda.is_available() else None
)
classifier = pipeline(
activity="token-classification",
mannequin=mannequin,
tokenizer=tokenizer,
aggregation_strategy="easy",
machine=machine if not torch.cuda.is_available() else None
)
LABEL_MASKS = {
"account_number": "[ACCOUNT_NUMBER]",
"private_address": "[PRIVATE_ADDRESS]",
"private_email": "[PRIVATE_EMAIL]",
"private_person": "[PRIVATE_PERSON]",
"private_phone": "[PRIVATE_PHONE]",
"private_url": "[PRIVATE_URL]",
"private_date": "[PRIVATE_DATE]",
"secret": "[SECRET]"
}
We set up all required libraries and arrange the pipeline’s runtime atmosphere. We configure machine choice and initialize paths for storing outputs. We additionally print system particulars to verify that all the things is prepared earlier than loading the mannequin.
def normalize_label(label):
label = label.exchange("B-", "").exchange("I-", "").exchange("E-", "").exchange("S-", "")
return label.strip()
def detect_pii(textual content):
uncooked = classifier(textual content)
spans = []
for merchandise in uncooked:
label = normalize_label(merchandise.get("entity_group", merchandise.get("entity", "")))
if label == "O" or not label:
proceed
spans.append({
"label": label,
"rating": float(merchandise["score"]),
"textual content": merchandise["word"],
"begin": int(merchandise["start"]),
"finish": int(merchandise["end"])
})
spans = sorted(spans, key=lambda x: (x["start"], x["end"]))
return spans
def redact_text(textual content, spans, min_score=0.50, mode="typed"):
filtered = [s for s in spans if s["score"] >= min_score]
filtered = sorted(filtered, key=lambda x: x["start"], reverse=True)
redacted = textual content
for span in filtered:
substitute = LABEL_MASKS.get(span["label"], "[PII]") if mode == "typed" else "[REDACTED]"
redacted = redacted[:span["start"]] + substitute + redacted[span["end"]:]
return redacted
def privacy_report(textual content, min_score=0.50):
spans = detect_pii(textual content)
redacted = redact_text(textual content, spans, min_score=min_score)
return {
"original_text": textual content,
"redacted_text": redacted,
"span_count": len([s for s in spans if s["score"] >= min_score]),
"spans": [s for s in spans if s["score"] >= min_score]
}We outline helper capabilities to normalize labels and extract PII spans from mannequin predictions. We implement a redaction operate that replaces delicate segments primarily based on confidence thresholds. We mix all the things right into a single reporting operate that returns structured outputs.
sample_texts = [
"My name is Alice Smith and my email is [email protected]. Name me at +1 415 555 0189.",
"Affected person Rohan Mehta visited on 2025-04-11 and lives at 221B Baker Avenue, London.",
"Use API key sk-test-51HxYzDemoSecret987 and ship the bill to [email protected].",
"The general public web site is https://instance.com, however Jane Doe's personal portal is https://jane-private.instance.internet.",
"Account quantity 123456789012 was linked to Ahmed Khan on 12 March 2024.",
"This sentence has no personal data and will largely stay unchanged."
]
studies = []
for i, textual content in enumerate(sample_texts, 1):
report = privacy_report(textual content, min_score=0.50)
report["example_id"] = i
studies.append(report)
for r in studies:
print("n" + "=" * 100)
print("Instance:", r["example_id"])
print("Unique:", r["original_text"])
print("Redacted:", r["redacted_text"])
print("Detected spans:")
print(json.dumps(r["spans"], indent=2, ensure_ascii=False))
rows = []
for r in studies:
for s in r["spans"]:
rows.append({
"example_id": r["example_id"],
"label": s["label"],
"rating": s["score"],
"detected_text": s["text"],
"begin": s["start"],
"finish": s["end"],
"original_text": r["original_text"],
"redacted_text": r["redacted_text"]
})
df = pd.DataFrame(rows)
show(df)We create pattern inputs and run them by means of the pipeline to check detection and redaction. We acquire structured outcomes and print each unique and redacted textual content for comparability. We additionally convert the outputs right into a dataframe for simpler evaluation.
json_path = OUT_DIR / "privacy_filter_reports.json"
csv_path = OUT_DIR / "privacy_filter_spans.csv"
with open(json_path, "w", encoding="utf-8") as f:
json.dump(studies, f, indent=2, ensure_ascii=False)
df.to_csv(csv_path, index=False)
print("nSaved JSON:", json_path)
print("Saved CSV:", csv_path)
if len(df):
label_counts = df["label"].value_counts()
plt.determine(figsize=(10, 5))
label_counts.plot(sort="bar")
plt.title("Detected PII Classes")
plt.xlabel("PII Class")
plt.ylabel("Detected Span Depend")
plt.xticks(rotation=35, ha="proper")
plt.tight_layout()
plt.present()
plt.determine(figsize=(10, 5))
df["score"].plot(sort="hist", bins=10)
plt.title("Detection Confidence Distribution")
plt.xlabel("Confidence Rating")
plt.ylabel("Frequency")
plt.tight_layout()
plt.present()
def compare_thresholds(textual content, thresholds=(0.30, 0.50, 0.70, 0.90)):
spans = detect_pii(textual content)
outcomes = []
for threshold in thresholds:
stored = [s for s in spans if s["score"] >= threshold]
outcomes.append({
"threshold": threshold,
"span_count": len(stored),
"redacted_text": redact_text(textual content, spans, min_score=threshold)
})
return pd.DataFrame(outcomes)
threshold_demo = compare_thresholds(sample_texts[0])
show(threshold_demo)We save the processed outputs into JSON and CSV codecs for persistence and reuse. We visualize detected PII classes and confidence distributions utilizing plots. We additionally analyze how altering thresholds impacts detection and redaction conduct.
long_document = """
Buyer Assist Transcript:
Agent: Hi there, could I affirm your identify?
Buyer: My identify is PSP.
Agent: Thanks. May you affirm your e mail?
Buyer: [email protected].
Agent: And your cellphone quantity?
Buyer: +91 xxxxx xxxxx.
Agent: Your service handle is 45 MG Highway, Bengaluru, Karnataka.
Buyer: Sure. Additionally, my backup e mail is [email protected].
Agent: Please don't share passwords or OTPs.
Buyer: The short-term token I obtained is ghp_demoSecretToken123456.
"""
long_report = privacy_report(long_document, min_score=0.50)
print("nLONG DOCUMENT REDACTION")
print("=" * 100)
print(long_report["redacted_text"])
print("nStructured spans:")
print(json.dumps(long_report["spans"], indent=2, ensure_ascii=False))
def pii_audit_table(texts, min_score=0.50):
audit_rows = []
for idx, textual content in enumerate(texts, 1):
consequence = privacy_report(textual content, min_score=min_score)
labels = Counter([s["label"] for s in consequence["spans"]])
audit_rows.append({
"id": idx,
"original_chars": len(textual content),
"redacted_chars": len(consequence["redacted_text"]),
"span_count": consequence["span_count"],
"labels_found": dict(labels),
"redacted_text": consequence["redacted_text"]
})
return pd.DataFrame(audit_rows)
audit_df = pii_audit_table(sample_texts + [long_document], min_score=0.50)
show(audit_df)
audit_path = OUT_DIR / "privacy_filter_audit.csv"
audit_df.to_csv(audit_path, index=False)
print("Saved audit CSV:", audit_path)
custom_text = enter("nEnter your personal textual content for PII redaction, or press Enter to skip:n")
if custom_text.strip():
custom_report = privacy_report(custom_text, min_score=0.50)
print("nOriginal:")
print(custom_report["original_text"])
print("nRedacted:")
print(custom_report["redacted_text"])
print("nSpans:")
print(json.dumps(custom_report["spans"], indent=2, ensure_ascii=False))
else:
print("Skipped customized enter.")
print("nTutorial full.")We take a look at the pipeline on an extended, lifelike doc to guage robustness. We generate an audit-style abstract exhibiting counts and classes of detected PII. We additionally permit customized consumer enter so we will run the privateness filter interactively.
In conclusion, we developed a sturdy and extensible privateness filtering workflow that goes past easy detection. We systematically evaluated mannequin predictions, utilized confidence thresholds, and in contrast totally different redaction methods to know their impression. We additionally generated structured studies, visualized detection patterns, and exported ends in JSON and CSV codecs for auditing and downstream integration. This strategy permits us to construct dependable privateness safeguards into information pipelines, guaranteeing that delicate data is persistently recognized and dealt with responsibly whereas sustaining the usability of the underlying information.
Take a look at the Full Codes here. Additionally, be happy to observe us on Twitter and don’t neglect to affix our 130k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Must associate with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and so forth.? Connect with us
