Configure anonymization

Published transform reference

The transform names and parameters below are shared across the product family, so teams can apply the same treatment to related values in different systems.

MongoDB, Kafka, SQL Server, Oracle and CouchDB draw the same transforms from the same library and accept the same parameters, with two exceptions worth knowing before you write a rule: CouchDB does not accept json as a rule’s transform, because a CouchDB document already is JSON and a nested path belongs in the rule’s field_path; and MongoDB accepts json only on a string field that holds a JSON document, since a BSON subdocument is reached with a field path instead.

Deterministic keyed transforms

With the same resolved hash key and parameters, equal inputs map alike. Stable mappings preserve equality for matching; they do not all preserve uniqueness.

noise and dnoise keep distributions usable through bounded shifts, but do not promise an unchanged distribution, ordering or uniqueness. They are not a guarantee that every output differs from its input.

Shape-preserving fake values: phone replaces each digit with a keyed digit; postcode also replaces letters while keeping their case. Both preserve separators and format shape, not the original digits or letters. A value with nothing to replace fails.

Unkeyed transforms

Nested documents

json takes paths, an array of 1–16 entries, each with a path, a transform, and that leaf transform's parameters. Every non-json kind above is accepted as a leaf. It transforms selected values and re-serializes the document; unselected values stay unchanged. See nested JSON paths for path syntax and shape checks.

Joinable integer pseudonyms in every product

pseudo_id is a deterministic, injective, format-preserving permutation over a declared integer domain. Set {"digits":9,"domain":"customer_id"}; digits is required (1–18), and domain is optional. Use the same resolved hash key, digit width and domain for a primary key and its foreign keys. The result stays numeric within the declared width; it is not a fixed-width padded string.

An identifier outside the configured digit range stops replication instead of being changed unpredictably or copied unchanged. Non-integer input also stops the run. Zero is transformed like any other identifier, so review applications that use zero as a special value. Ordering, gaps and issue rate are not preserved.

PostgreSQL fields

Each row identifies a schema, table, and column. The transform and its JSON parameters are applied during initial sync, streaming replication, and backfill.

INSERT INTO anon_field
  (schema_name, table_name, column_name, transform, params)
VALUES
  ('public','users','email',    'hash', '{"length":16}'),
  ('public','users','ssn',      'mask', '{"pattern":"XXX-XX-","keep_last":4}'),
  ('public','users','full_name','fake', '{"kind":"name"}'),
  ('public','users','notes',    'null', NULL),
  ('public','users','metadata', 'json',
   '{"paths":[{"path":"$.contact.email","transform":"fake","kind":"email"}]}');

Configure partitioned PostgreSQL tables at the partition root. A json transform is accepted only on json, jsonb, or text columns.

MySQL / MariaDB fields

MySQL calls a database a schema. Include the cluster name because one configuration database can manage multiple source/replica pairs.

INSERT INTO anon_field
  (cluster_name, schema_name, table_name, column_name, transform, params)
VALUES
  ('default','appdb','users','email',    'hash', '{"length":16}'),
  ('default','appdb','users','ssn',      'mask', '{"pattern":"XXX-XX-","keep_last":4}'),
  ('default','appdb','users','full_name','fake', '{"kind":"name"}'),
  ('default','appdb','users','notes',    'null', NULL),
  ('default','appdb','users','metadata', 'json',
   '{"paths":[{"path":"$.contact.email","transform":"fake","kind":"email"}]}');

A newly inserted MySQL field defaults to backfill_pending=1. Initial sync applies it while copying and clears the flag. On an existing replica, run backfill before treating the field as protected.

Nested JSON paths

Paths begin at $. Use .key to descend into an object and [*] to visit every item in an array. Numeric indexes such as [0] are not supported. A path can contain up to eight steps, and each configuration can contain 1–16 paths.

{
  "paths": [
    {"path":"$.ssn", "transform":"mask", "pattern":"XXX-XX-", "keep_last":4},
    {"path":"$.contact.email", "transform":"hash", "length":12},
    {"path":"$.members[*].email", "transform":"fake", "kind":"email"},
    {"path":"$.optional_note", "transform":"null"}
  ]
}

• An absent object key is left alone because no value is present.
• A present scalar where the path expects an object or array is a fatal shape error.
• A JSON null at the final step stays null.
• Invalid JSON or a non-scalar value at the leaf fails the row rather than leaking it unchanged.

Redis rules

A Redis rule combines a key glob, container type, optional selector, transform, parameters, and optional value_kind. Rules are checked in insertion order; the first matching rule wins.

INSERT INTO anon_rule
  (cluster_name,key_pattern,value_type,selector,transform,params,value_kind)
VALUES
  ('default','user:*','string','','hash','{"length":16}','text'),
  ('default','profile:*','hash','email','fake','{"kind":"email"}','text'),
  ('default','session:*','list','','mask','{"pattern":"***","keep_last":4}','text'),
  ('default','events:*','stream','payload','hash','{"length":14}','text'),
  ('default','doc:*','json','','json',
   '{"paths":[{"path":"$.ssn","transform":"mask","pattern":"***","keep_last":4}]}','doc');

Use value_kind for cross-engine consistency

Redis stores bytes without a type catalog. If a Redis value mirrors a SQL date, timestamp, UUID, decimal, or another typed value, declare its kind so the canonical rendering matches the SQL product before hashing. Accepted values are text, bool, int, uint, decimal, float, char, bytes, timestamp, date, time, uuid, and doc. A type mismatch stops replication and identifies the affected key.

Strict policy mode

INSERT INTO settings(cluster_name,key,value) VALUES
  ('default','unmatched_key_policy','reject'),
  ('default','passthrough_allowlist','health:*,public-cache:*');

Strict mode turns a coverage gap into a stopped sync or replication process instead of a plaintext copy. Test it on representative data and keep the allowlist narrow.

MongoDB fields

A rule names a database glob, a collection glob and one exact field path. . descends into a document and [*] into every element of an array; there is no positional index and no wildcard key, so the only fan-out is “every element”. Keys may not contain ., [ or ]. Where overlapping globs bring two rules to the same leaf, the first in id order wins.

INSERT INTO anon_field
  (db_name, collection_name, field_path, transform, params)
VALUES
  ('app','users','email',                   'partial_email', NULL),
  ('app','users','name',                    'fake',  '{"kind":"name"}'),
  ('app','users','profile.address.street',  'mask',  '{"pattern":"***","keep_last":0}'),
  ('app','users','phones[*]',               'hash',  '{"length":12}'),
  ('app','users','cards[*].number',         'null',  NULL),
  ('app','users','born',                    'dnoise','{"days":30}'),
  ('tenant_*','orders','customer_id',        'pseudo_id','{"digits":9,"domain":"customers"}');

BSON carries the type, so nothing has to be declared: each value is canonicalised under the kind its type implies and the transform’s text result is stored back under the original type wherever the text still is one. That is what keeps an anonymized MongoDB replica joinable with an anonymized SQL replica of the same data — an account id stored as Int64 here and BIGINT there canonicalises to the same text and hashes to the same value under the same key.

Kafka record parts

A Kafka rule names a topic glob, which part of the record it addresses, how that part’s bytes are read, and — for JSON — which leaf. The part is value, key or header:<name>; partition, offset and timestamp are never rule targets. The format is json (the leaf is rewritten and the document re-serialized compact, every other leaf untouched), text (the whole part is one string) or bytes (opaque octets, canonicalised to lowercase hex before the transform runs).

INSERT INTO anon_field
  (cluster_name, topic_pattern, part, format, field_path, transform, params)
VALUES
  ('default','users',    'value',      'json', '$.email',            'hash', '{"length":16}'),
  ('default','users',    'value',      'json', '$.cards[*].number',  'mask', '{"pattern":"***","keep_last":4}'),
  ('default','users',    'key',        'text', '',                   'hash', '{"length":16}'),
  ('default','audit-*',  'header:pii', 'text', '',                   'mask', '{"pattern":"***"}'),
  ('default','blobs',    'value',      'bytes','',                   'hash', '{"length":32}');

SQL Server columns

A rule names the SQL Server schema, the table and the column. A rule on a column that does not exist, on a computed column, a null rule on a NOT NULL column, or a transform whose output cannot be stored in the column’s type is refused at init-sync with the reason.

INSERT INTO anon_field
  (schema_name, table_name, column_name, transform, params)
VALUES
  ('dbo','users','id',        'pseudo_id',  '{"digits":6,"domain":"users"}'),
  ('dbo','users','email',     'hash',       '{"length":16}'),
  ('dbo','users','ssn',       'mask',       '{"pattern":"XXX-XX-","keep_last":4}'),
  ('dbo','users','full_name', 'fake',       '{"kind":"name"}'),
  ('dbo','users','notes',     'null',       NULL),
  ('dbo','users','born',      'generalize', '{"unit":"year"}'),
  ('sales','orders','user_id', 'pseudo_id', '{"digits":6,"domain":"users"}');

Every value is rendered by the server with a fixed CONVERT style — ISO 8601 for the temporal types, 17 significant digits for float, hex for binary — and canonicalised before the transform sees it, so char(10) padding, tinyint against int, or a datetimeoffset’s zone never change what is hashed. A constant or pick value that does not fit the column is refused at the first row, before anything is written. sql_variant, geometry, geography and hierarchyid are not supported and refuse the table outright.

Oracle columns

A rule names the cluster, the owner, the table and the column, spelled exactly as the catalog spells them — upper case unless the object was created with a quoted identifier. A rule that matches no column of any replicated table fails init-sync: a column the configuration says is protected, and which silently is not, is the one failure this product exists to prevent.

INSERT INTO anon_field
  (cluster_name, schema_name, table_name, column_name, transform, params)
VALUES
  ('default','APP','USERS','EMAIL',     'hash',       '{"length":16}'),
  ('default','APP','USERS','SSN',       'mask',       '{"pattern":"XXX-XX-","keep_last":4}'),
  ('default','APP','USERS','FULL_NAME', 'fake',       '{"kind":"name"}'),
  ('default','APP','USERS','BORN',      'generalize', '{"unit":"year"}'),
  ('default','APP','USERS','NOTES',     'null',       NULL);

Types map as you would expect: VARCHAR2, NVARCHAR2, CHAR, NCHAR, CLOB, NCLOB and LONG are text; a NUMBER with scale 0 is an integer and any other NUMBER a decimal; FLOAT and the binary floats are floats; DATE and TIMESTAMP, with and without time zone, are timestamps — and an Oracle DATE carries a time; RAW, LONG RAW and BLOB are bytes; INTERVALs are text; BOOLEAN (23ai) is a boolean; JSON is a document. Every session runs in UTC with fixed NLS formats, so TIMESTAMP WITH LOCAL TIME ZONE reads the same on both sides.

CouchDB fields

A rule names a database glob and one JSON field path. The path starts at $, each step is .key or [*], up to eight steps, and the first step must be a key because a CouchDB document is an object. There is no per-document filter: a database has no schema, and a path a document does not carry is a no-op for that document, so documents of several shapes share one rule set as long as they agree on what the sensitive field is called. These are the same paths the SQL products’ json transform takes, walked by the same code.

INSERT INTO anon_field
  (cluster_name, database_pattern, field_path, transform, params)
VALUES
  ('default','app*','$.email',               'hash', '{"length":16}'),
  ('default','app*','$.contacts[*].phone',   'fake', '{"kind":"phone"}'),
  ('default','app*','$.address.street',      'null', NULL),
  ('default','app*','$.orders[*].lines[*].sku', 'hash', '{"length":12}');

Activate a new field safely