Protecting PII in Data Pipelines: Masking, Hashing, Tokenisation and Encryption

PII (personally identifiable information) is any data that can identify a person, directly or combined with other data: names, emails, phone numbers, addresses, government ids, card numbers, precise locations, and often device ids and IP addresses. Data pipelines copy data between many systems, so they are where PII most often ends up somewhere it should not be: a raw bucket everyone can read, a debug log, a test database, a spreadsheet export.

Protecting it rests on two ideas: collect and copy less, and transform what you must keep so that most people and systems never see the real values.

The techniques

Technique	What it does	Reversible?	Joins still work?	Typical use
Masking	Hides part of a value: a***@example.com	No	No	Showing data to support staff or analysts
Plain hashing	Replaces a value with a fixed fingerprint	Not directly, but guessable values can be found	Yes	Rarely safe on its own (see below)
Keyed hashing (HMAC)	A hash that needs a secret key	Not without the key	Yes	Pseudonymous ids for analytics
Tokenisation	Swaps the value for a random token; a secure vault keeps the mapping	Yes, through the vault only	Yes, if the same token is reused	Payments, systems that occasionally need the real value
Encryption	Scrambles data with a key	Yes, with the key	Not directly	Storage at rest, data in transit, sensitive columns
Generalisation	Reduces detail: birth date to birth year, address to city	No	Partly	Publishing statistics, sharing data
Deletion	Do not keep the field at all	No	No	The best option when the data is not needed

See them in code

This example masks values for display, shows why a plain hash of a phone number is not protection, and compares it with a keyed hash and a token vault. The phone numbers are from a range reserved for fiction:

import hashlib, hmac, secrets

customers = [("C1", "[email protected]", "+447700900456"), ("C2", "[email protected]", "+447700900123")]

# 1. Masking: hide most of the value, keep enough to be recognisable. One-way, for display.
def mask_email(email):
    name, domain = email.split("@")
    return name[0] + "***@" + domain

def mask_phone(phone):
    return phone[:3] + "*" * (len(phone) - 7) + phone[-4:]

for cid, email, phone in customers:
    print(cid, mask_email(email), mask_phone(phone))

# 2. A plain hash looks safe, but phone numbers are guessable: try every candidate and compare.
target = hashlib.sha256(b"+447700900456").hexdigest()
known_prefix = "+4477009"                       # an attacker can often guess the first digits
tries = 0
for n in range(100_000):
    tries += 1
    guess = f"{known_prefix}{n:05d}"
    if hashlib.sha256(guess.encode()).hexdigest() == target:
        print(f"plain SHA-256 reversed in {tries} guesses: {guess}")
        break

# 3. Keyed hashing (HMAC): same input gives the same value, so joins still work,
#    but without the secret key nobody can test guesses. Keep the key in a secrets manager.
key = secrets.token_bytes(32)
def pseudonymise(value):
    return hmac.new(key, value.encode(), hashlib.sha256).hexdigest()[:16]

print("same input, same token:", pseudonymise("+447700900456") == pseudonymise("+447700900456"))

# 4. Tokenisation with a vault: a random token replaces the value; only the vault can map it back.
vault = {}
def tokenise(value):
    token = "tok_" + secrets.token_hex(8)
    vault[token] = value
    return token

t = tokenise("[email protected]")
print("token looks random:", t.startswith("tok_") and len(t) == 20, "| vault can reverse it:", vault[t])

C1 a***@example.com +44******0456
C2 t***@example.com +44******0123
plain SHA-256 reversed in 457 guesses: +447700900456
same input, same token: True
token looks random: True | vault can reverse it: [email protected]

The key lesson is the third line. A hash cannot be “decrypted”, but anyone holding the hash can guess and check. Phone numbers, national ids, dates of birth and many emails come from small or predictable sets, so the guesses run out quickly: here it took 457 tries, and all 100,000 candidates take well under a second. Adding a secret key (HMAC) stops this, because without the key an attacker cannot compute the hash of a guess. The key then needs the same care as a password: stored in a secrets manager, rotated with a plan, and never in code.

Pseudonymous is not anonymous

Replacing an email with a keyed hash or token is pseudonymisation: the data can still be linked back to a person by whoever has the key, the vault or enough other columns. Privacy laws such as the GDPR still treat pseudonymised data as personal data. Truly anonymous data, which cannot reasonably be linked to a person by anyone, is much harder to achieve: a combination such as postcode, birth date and gender can identify many people even with no name present.

Where PII protection goes in a pipeline

1. Classify at the door. When a new source is added, tag columns as PII or sensitive in the catalog. Warehouses and catalogs can scan for likely PII automatically, but people should confirm.
2. Minimise early. Drop fields nobody needs before they land in the lake.
3. Restrict the raw zone. If raw data must keep PII, only the pipeline’s service account and a few named engineers can read it.
4. Pseudonymise in the clean layer. Replace direct identifiers with keyed hashes or tokens so analysts can count and join customers without seeing who they are.
5. Mask or filter at query time. Use the platform’s column masking and row access policies so the same table shows different values to different roles. See Snowflake security and Unity Catalog.
6. Encrypt everywhere. At rest (storage encryption, often on by default in the cloud) and in transit (TLS), plus column-level encryption for the most sensitive fields.
7. Keep it out of logs and tests. Never log full records; use synthetic or masked data in development.
8. Plan for deletion. Know every table that holds a person’s data so you can delete it when required.

Common mistakes

1. Hashing without a key and calling it anonymised.
2. PII in logs, error messages and dead-letter queues, which are rarely protected like tables.
3. Copies of production data in development environments.
4. Keys stored next to the data they protect, or in the pipeline code.
5. Masking only in the BI tool, while the underlying table is open to everyone with SQL access.

What interviewers look for

For “how do you protect PII in a pipeline?”:

• The difference between masking, hashing, keyed hashing, tokenisation and encryption, and one use for each.
• Why an unkeyed hash of an email or phone number can be reversed.
• Where in the pipeline each control goes: classify, minimise, restrict raw, pseudonymise, mask by role, encrypt, keep out of logs.
• That pseudonymised data is still personal data.

Practise with the data engineering interview questions.

Key takeaway

Collect less, and transform what you keep. Mask for display, use keyed hashing or tokenisation so analysts can join without seeing identities, encrypt data at rest and in transit, and apply column masking and row policies by role. A plain hash of a guessable value is not protection, and pseudonymised data is still personal data. Next: data governance and data mesh.