TOKENIZATION

Designing the system around a substitute value: domains, reversibility, lifecycle and the authority to recover the original.

Tokenization is often introduced with a simple diagram: a sensitive value enters a service and an unrelated value comes back. The application stores or transports the token; the original remains somewhere else.

That substitution is useful, but it is not yet a system design.

The questions that determine the real properties of a token appear immediately afterwards. Must the same input produce the same token? Must the token preserve length or character set? Can it be used in more than one application? Who can recover the original? What happens when a token is copied, revoked or presented after its intended context has ended?

A commissioned design for a generic tokenization service began from those questions. The initial setting was financial, but the useful result was broader: tokenization is not primarily a choice of cipher. It is the design of a controlled relationship between an original value, its substitute and the systems allowed to move between them.

A token is a reference with policy

An opaque random value can be an effective token. If a protected service maintains the mapping, the token reveals no mathematical relationship to the original. But the mapping also becomes part of the critical infrastructure. Its confidentiality, integrity, availability and recovery procedures determine whether the token remains useful and whether the original remains protected.

Other constructions avoid a central lookup table by deriving a replacement cryptographically. That can reduce one operational dependency, but it changes the problem rather than removing it. Key management, collision handling, domain separation, rotation and the consequences of algorithm compromise now become central.

In both cases, the token needs a policy. A useful token is not merely different data. It is data with a defined scope:

• the system or domain in which it is meaningful;
• the operations for which it may be accepted;
• the period for which it remains valid;
• whether repeated tokenization is stable or intentionally produces a new value;
• whether detokenization is permitted and under whose authority.

Without those rules, a token may quietly become another long-lived identifier—less visibly sensitive than the original, but just as useful for correlation or abuse.

Reversible and irreversible are different designs

Some uses require the original value to be recovered. A downstream process may still need to submit an account reference to an established network, reconcile a legacy record or fulfil a narrowly authorised operation. The tokenization service must then support detokenization.

That reverse path deserves more attention than the forward API. It concentrates authority. Authentication of the caller is not enough: the service must decide whether that caller may recover this class of value for this purpose, record the event and avoid returning originals through logs, traces or error messages.

Other uses require only stable comparison. If the objective is to recognise that two inputs are equal without recovering them, a one-way construction may be more appropriate. Even then, a plain hash is rarely sufficient for low-entropy values such as document numbers, telephone numbers or account identifiers. An attacker can enumerate the likely input space. A keyed construction, a protected key and explicit domain separation are needed to prevent simple guessing and cross-system correlation.

Calling both designs “tokenization” can hide the most important architectural decision: whether an authority to recover the original exists at all.

Stability creates both utility and linkage

Deterministic tokens make joins and duplicate detection convenient. The same input can be recognised across transactions without repeated access to the original. That convenience also creates linkability.

If the same token is valid in unrelated applications, data collected for one purpose can be correlated with data from another. A better design normally separates token domains. The same original value can produce different tokens for different merchants, applications, datasets or purposes.

Domain separation may be implemented through independent mapping spaces, distinct keys or explicit context included in the derivation. The mechanism matters less than the invariant: a token copied out of one domain should not automatically become a useful identifier in another.

This also limits compromise. An exposed token population from one system need not invalidate every other system that refers to the same underlying values.

Format preservation is not tokenization

Legacy fields often impose fixed lengths, numeric-only values or validation rules. Preserving that shape can reduce integration work, but it introduces constraints that should be made explicit.

Format-preserving encryption transforms data while retaining an allowed format. NIST SP 800-38G specifies approved modes for this purpose. It is reversible encryption and depends on cryptographic keys; it should not be treated as synonymous with tokenization.

A vault-generated token can also imitate the original format, but a small output space raises the probability of collisions and can reveal structural information. If a six-digit field must contain a unique token for millions of active records, the arithmetic fails before cryptography becomes relevant.

The correct sequence is to establish the required population, lifetime and collision tolerance, then decide whether the inherited field constraint is viable. Preserving an obsolete database column at the cost of ambiguous or predictable tokens is not a security improvement.

The vault becomes a security boundary

In a vault-based design, the mapping store is not an ordinary database with an encryption flag. It is the boundary that separates applications from sensitive originals.

That boundary includes more than storage encryption. It includes:

• narrowly defined tokenize and detokenize operations;
• caller identity and purpose-specific authorisation;
• separation between service administration and access to clear values;
• key rotation and backup procedures that preserve existing mappings;
• audit records that identify sensitive operations without recording sensitive data;
• controls against bulk enumeration and abnormal recovery patterns;
• explicit behaviour when dependencies are unavailable.

Availability needs particular care. If every routine read requires detokenization, the vault becomes both a privacy boundary and a universal runtime dependency. A better system keeps most operations in the token domain and crosses the boundary only when the original is genuinely required.

Lifecycle is part of the token

Tokens accumulate. They are copied into backups, messages, analytics systems and logs precisely because they are considered safer to handle. The architecture therefore needs an answer for expiry, revocation and deletion.

Deleting a mapping can make a reversible token effectively irreversible, but only if no other copies of the original or mapping exist. Rotating a derivation key may change every deterministic token unless versioning or a migration mechanism was designed in advance. Reissuing a token may break references in systems that assumed permanence.

A token format may need a non-sensitive version or domain indicator so that the service can select the correct policy and key generation. That metadata should not disclose the original’s type more broadly than necessary, and it must be authenticated where an attacker could modify it.

These are data-governance questions expressed through engineering. A token service cannot decide retention policy for an organisation, but it must make that policy implementable.

Payment tokens are a specialised example

EMV payment tokenisation provides a useful illustration of scope. A payment token replaces a primary account number, but its value comes from the surrounding framework: token requestors, token service providers, assurance processes and controls that can restrict where or how a token is used.

A generic internal token should not be described as an EMV payment token merely because the original is financial data. The term carries a defined ecosystem and roles. What can be borrowed is the design principle: the substitute becomes safer when its domain and permitted use are narrower than those of the original.

What the original design clarified

The commissioned design did not proceed into a production deployment. That is relevant, not an inconvenience to hide. It means the work should be presented as system design rather than operational evidence.

Its lasting value was the separation of concerns that a credible implementation would need:

1. classify the values and determine whether recovery is genuinely required;
2. define token domains and the consequences of stable linkage;
3. calculate the output space and collision behaviour;
4. place detokenization behind stronger authorisation than token creation;
5. design key, mapping and token lifecycle together;
6. ensure observability never recreates the sensitive dataset in logs;
7. document what compromise of a token, a mapping store or a derivation key would reveal.

The token itself may be generated in milliseconds. The substantive work lies in deciding what authority that small string carries throughout its life.

---

Context and scope

This note develops lessons from a commissioned design for a generic tokenization system initially considered for use in a financial environment. The design did not proceed to production. It is discussed as an architectural study, without identifying the commissioning organisation or presenting it as a deployed service.

References

1. EMVCo — EMV Payment Tokenisation
2. EMVCo — The What, Why and How of EMV Payment Tokenisation
3. NIST SP 800-38G — Methods for Format-Preserving Encryption
4. NIST SP 800-57 Part 1 Rev. 5 — Recommendation for Key Management
5. OWASP — Logging Cheat Sheet