Bring the latest anonymized data to a developer machine

Download the latest anonymized database

Choose any of the eight engine sections below. The common download step below selects one immutable artifact; each engine section gives host and Docker restore commands. These are local development recipes around the product’s generated restore files, not extra product CLI commands.

1. Obtain an RS256 JWT from the operator’s OAuth2/OIDC identity provider, with the download scope below. serve validates tokens; it does not mint them.
2. List published databases with GET /databases and a global download token. A database-scoped token cannot list: ask the operator for the database name.
3. Fetch GET /databases/{db}/latest. Continue only on 200; 202 means preparing, 404 means no published artifact, and 500 means an unreadable manifest.
4. Pin the returned id. GET /databases/{db}/artifacts/{id} re-reads its manifest; GET /databases/{db}/artifacts/{id}/download downloads its bytes.
5. Verify sha256 and size_bytes, then restore using the declared format and engine_min_version.

The general database claim is <scope>:<database>. Policy read/write scopes do not grant downloads. See API authentication setup.

The manifest fields are id, database, engine, format, capture_time, replication_position, config_version, size_bytes, sha256, file_name, download_path, plus restore guidance and the version floor. The actual names are capture_time and replication_position. PostgreSQL records LSN and timeline; MySQL records GTID, binlog position or unknown; Redis RDB records replication id/offset per node; MongoDB records a resume token, operation time or unknown; Kafka records per-partition source high-watermarks; SQL Server records its checkpoint LSN; Oracle records the pinned SCN; CouchDB records the database update sequence.

Publish artifacts

Operators run these C binaries using their SQLite configuration database. Configure artifact_dir and, for base-backed bundles, backup_dir. PostgreSQL also needs the required archived WAL. The enterprise API serves already-published artifacts; it does not create backups on request.

patroni-anonymizer backup-base --config /etc/patroni-anonymizer.db
patroni-anonymizer backup-publish --config /etc/patroni-anonymizer.db

mysql-anonymizer backup-base --config config.db
mysql-anonymizer backup-publish --config config.db

redis-anonymizer backup-publish --config config.db

mongodb-anonymizer backup-base config.db
mongodb-anonymizer backup-publish config.db

kafka-anonymizer backup-capture config.db --cluster prod
kafka-anonymizer backup-publish config.db --cluster prod

mssql-anonymizer backup-base --config config.db --cluster default
mssql-anonymizer backup-publish --config config.db --cluster default

oracle-anonymizer backup-base --config config.db --cluster prod
oracle-anonymizer backup-publish --config config.db --cluster prod

couchdb-anonymizer backup-publish config.db --cluster default

Use --cluster to select a configured cluster when needed. MongoDB creates a dump base before publication; Kafka captures topic records before publication; the other commands publish the product’s current artifact workflow. Physical PostgreSQL and MySQL backups cover an instance, and Redis snapshots cover a complete target instance, so review the full dataset before distributing it.

Download and verify once

Install curl, jq and sha256sum. Export API_BASE as your HTTPS API origin without a trailing slash (no /v1 prefix), TOKEN as the JWT, and DATABASE as a name returned by the API. With a global download token, list first:

curl --fail --silent --show-error   -H "Authorization: Bearer $TOKEN" "$API_BASE/databases" | jq .

Save fetch-dev-data.sh beside your Makefile. Run ARTIFACT=$(sh ./fetch-dev-data.sh). Its stdout is the verified artifact’s absolute path; manifest details go to stderr. It stores each id separately so a new latest artifact cannot be appended to an older download.

#!/bin/sh
# Download one immutable artifact; stdout is its verified absolute path.
set -eu
umask 077
: "${API_BASE:?set API_BASE to the HTTPS API origin, without a trailing slash}"
: "${TOKEN:?set TOKEN to an RS256 download token}"
: "${DATABASE:?set DATABASE to the published database name}"
case "$API_BASE" in https://*) ;; *) echo 'HTTPS API_BASE required' >&2; exit 1;; esac
case "$DATABASE" in ''|*[!A-Za-z0-9_.-]*|.|..) echo 'Invalid database name' >&2; exit 1;; esac
mkdir -p "${OUT_DIR:-.dev-data}/$DATABASE"
root=$(cd "${OUT_DIR:-.dev-data}/$DATABASE" && pwd)
manifest=$(mktemp "$root/manifest.XXXXXX")
trap 'rm -f "$manifest"' 0 HUP INT TERM
status=$(curl -sS --fail -H "Authorization: Bearer $TOKEN" \
  -w '%{http_code}' -o "$manifest" "$API_BASE/databases/$DATABASE/latest")
[ "$status" = 200 ] || { echo "Latest artifact not ready: HTTP $status" >&2; exit 1; }
id=$(jq -er '.id | strings' "$manifest")
case "$id" in ''|*[!A-Za-z0-9_.-]*|.|..) echo 'Invalid artifact id' >&2; exit 1;; esac
expected=$(jq -er '.sha256 | select(test("^[0-9a-f]{64}$"))' "$manifest")
size=$(jq -er '.size_bytes | select(type == "number" and . >= 0 and floor == .)' "$manifest")
mkdir -p "$root/$id"
target="$root/$id/artifact.tar"
cp "$manifest" "$root/$id/manifest.json"
verify() {
  [ -f "$target" ] && [ "$(wc -c < "$target" | tr -d ' ')" = "$size" ] &&
    printf '%s  %s\n' "$expected" "$target" | sha256sum -c - >&2
}
if ! verify; then
  # --fail keeps an HTTP error body out of the artifact. A 416 is not success.
  curl --fail --silent --show-error -C - -H "Authorization: Bearer $TOKEN" \
    -o "$target" "$API_BASE/databases/$DATABASE/artifacts/$id/download"
  verify || { echo 'Verification failed; do not restore' >&2; exit 1; }
fi
jq '{id, engine, format, capture_time, replication_position, engine_min_version, restore_notes}' "$manifest" >&2
printf '%s\n' "$target"

Transport: HTTPS served by the enterprise API, with RFC 7233 single-byte-range resume. Accept-Ranges: bytes advertises support; a valid range returns 206 Partial Content and Content-Range. An unsatisfiable range returns 416 with Content-Range: bytes */size. Unsupported or malformed ranges may return a full 200 response.

On 416 or checksum failure, stop. Re-read the pinned id’s manifest; accept an existing file only if its size and digest match. Otherwise remove that local artifact file and rerun to download from byte zero. The helper skips download for an already verified file and never treats an HTTP error as successful restoration.

Restore locally

Run one engine recipe after the common download step. Use a new, disposable restore directory. Set ARTIFACT from the helper and inspect its adjacent manifest.json. The tar itself is not gzip-compressed for any engine; do not use tar -z. Match the engine and its compatibility requirements before starting it.

PostgreSQL / Patroni: download the latest anonymized database

Use pa:download to list, or pa:download:orders for only orders. Follow token setup and download above with DATABASE=orders. Format base-wal-tar: uncompressed tar containing base/, wal/, restore.sh and README.txt.

Plain host

Stop the target PostgreSQL instance first. Install the matching major version (read the manifest and base/PG_VERSION), with pg_ctl and psql on PATH. Run as an unprivileged PostgreSQL-capable user, never root. The script starts PostgreSQL, replays bundled WAL, and waits for promotion. Do not reuse an existing data directory.

set -eu
: "${ARTIFACT:?download and verify first}"
mkdir pg-dev
tar -xf "$ARTIFACT" -C pg-dev
cd pg-dev
./restore.sh 5433

Docker

Use the same matching major and any extensions required by the backup. Run the host shell as a non-root user. No host port is published: the generated restore config uses loopback and may trust local connections. Set PGUSER to an existing role in the backup. A successful container launch is not proof of completed recovery; inspect its logs.

set -eu
: "${ARTIFACT:?download and verify first}"
: "${PG_IMAGE:?set a PostgreSQL image matching the backup major, with pg_ctl and psql on PATH}"
mkdir pg-dev
tar -xf "$ARTIFACT" -C pg-dev
docker run -d --name dev-postgres --user "$(id -u):$(id -g)"   -v "$PWD/pg-dev:/restore" -w /restore --entrypoint /bin/sh   "$PG_IMAGE" -ec './restore.sh 5433; exec tail -f /dev/null'
# Check restore completion in logs before connecting.
docker logs dev-postgres
# Connect inside the container; restore.sh binds to loopback.
docker exec -it dev-postgres psql -h 127.0.0.1 -p 5433 -U "$PGUSER" postgres

Keep the restore path free of single quotes, as required by the bundled recovery configuration.

MySQL / MariaDB / Percona: download the latest anonymized database

Use mya:download to list, or mya:download:app for only app. Follow token setup and download above with DATABASE=app. The uncompressed tar contains either data.sql.gz (format logical-sql-gz) or a gzip-compressed physical stream, data.xb.gz (format physical-mariabackup or physical-xtrabackup).

Plain host: logical SQL import

Start an empty local MySQL/MariaDB database and export the connection variables below. The bundled restore.sh validates gzip, decompresses the SQL and imports it with the installed client. Logical dumps have no physical datadir or same-major requirement; review SQL compatibility.

set -eu
: "${ARTIFACT:?download and verify first}"
: "${MYSQL_PWD:?set the local database password}"
: "${MYSQL_USER:?set the local database user}"
: "${MYSQL_DATABASE:?set the existing empty local database}"
mkdir mysql-dev
tar -xf "$ARTIFACT" -C mysql-dev
cd mysql-dev
./restore.sh 127.0.0.1 3306 "$MYSQL_USER" "$MYSQL_DATABASE"

Docker: logical SQL import

Select a compatible image supporting MYSQL_ROOT_PASSWORD and MYSQL_DATABASE initialization. The named disposable server has no published host port. The second container runs the bundle’s import against that server.

set -eu
: "${ARTIFACT:?download and verify first}"
: "${MYSQL_IMAGE:?set a compatible MariaDB/MySQL image with mysql or mariadb client}"
: "${MYSQL_PWD:?set the disposable root password}"
: "${MYSQL_DATABASE:?set the local database name}"
export MYSQL_ROOT_PASSWORD="$MYSQL_PWD"
mkdir mysql-dev
tar -xf "$ARTIFACT" -C mysql-dev
docker run -d --name dev-mysql -e MYSQL_ROOT_PASSWORD -e MYSQL_DATABASE "$MYSQL_IMAGE"
# Wait at most 60 seconds for the server to accept SQL.
n=0
until docker exec -e MYSQL_PWD dev-mysql sh -ec   'client=$(command -v mariadb || command -v mysql); "$client" -h 127.0.0.1 -u root -e "SELECT 1"' >/dev/null 2>&1; do
  n=$((n + 1)); [ "$n" -lt 60 ] || exit 1
  sleep 1
done
docker run --rm --network container:dev-mysql   -v "$PWD/mysql-dev:/restore:ro" -w /restore -e MYSQL_PWD -e MYSQL_DATABASE   --entrypoint /bin/sh "$MYSQL_IMAGE"   -ec './restore.sh 127.0.0.1 3306 root "$MYSQL_DATABASE"'

Plain host: physical prepare

Stop the destination server. Install mariabackup and mbstream matching MariaDB major.minor, or xtrabackup and xbstream matching MySQL/Percona (8.0 for 8.0, 8.4 for 8.4). MariaDB and MySQL physical backups are not interchangeable.

mkdir mysql-physical
tar -xf "$ARTIFACT" -C mysql-physical
cd mysql-physical
./restore-physical.sh "$PWD/datadir"

The generated script decompresses and extracts the stream into the empty target, then runs mariabackup --prepare or xtrabackup --prepare. Follow its printed host steps: set ownership to the server user, point the server’s datadir at the prepared directory, then start the server. It deliberately does not guess your service manager or overwrite a running server.

Docker: physical prepare

Set BACKUP_IMAGE to an image containing the matching backup and stream tools. This restores and prepares the datadir; apply the generated ownership/start instructions for your chosen server container before mounting it there. The publisher’s original database credentials remain in a physical backup.

mkdir mysql-physical
tar -xf "$ARTIFACT" -C mysql-physical
: "${BACKUP_IMAGE:?set an image with matching mariabackup/mbstream or xtrabackup/xbstream}"
docker run --rm -v "$PWD/mysql-physical:/restore" -w /restore   --entrypoint /bin/sh "$BACKUP_IMAGE"   -ec './restore-physical.sh /restore/datadir'

Full physical bases only; incremental physical delivery and binlog archiving are out of scope.

Redis: download the latest anonymized database

Use ra:download to list, or ra:download:cache for only cache. Follow token setup and download above with DATABASE=cache. Format rdb-snapshot is an uncompressed tar with one ordinary .rdb per node and RESTORE.txt; it is not gzip.

Valkey targets (8.1 and later) always produce rdb-snapshot: Valkey has no BACKUP command family. The manifest then says "engine":"valkey" and engine_min_version is the Valkey version (not the fixed redis_version 7.2.4 Valkey reports for compatibility). Restore with valkey-server at least that new. A Valkey 9 snapshot uses Valkey's own RDB format and does not load on Redis or on Valkey 8; a Redis 7.4+ snapshot does not load on Valkey.

Plain host: RDB snapshot

Use Redis at least as new as the writer (engine_min_version). Stop any instance using the destination directory. The fresh instance below loads the extracted RDB on startup with AOF disabled, so an old AOF cannot take precedence.

set -eu
: "${ARTIFACT:?download and verify first}"
mkdir redis-dev
tar -xf "$ARTIFACT" -C redis-dev
# This one-node recipe refuses multi-shard artifacts.
set -- "$PWD"/redis-dev/*.rdb
[ "$#" -eq 1 ] && [ -f "$1" ] || { echo 'Follow RESTORE.txt per shard' >&2; exit 1; }
redis-server --dir "$PWD/redis-dev" --dbfilename "$(basename "$1")"   --appendonly no --bind 127.0.0.1 --port 6380

Docker: RDB snapshot

Use the same minimum version requirement. Connect with docker exec; no host port is published.

set -eu
: "${ARTIFACT:?download and verify first}"
: "${REDIS_IMAGE:?set a Redis image at least as new as engine_min_version}"
mkdir redis-dev
tar -xf "$ARTIFACT" -C redis-dev
set -- "$PWD"/redis-dev/*.rdb
[ "$#" -eq 1 ] && [ -f "$1" ] || { echo 'Follow RESTORE.txt per shard' >&2; exit 1; }
docker run -d --name dev-redis -v "$PWD/redis-dev:/data" "$REDIS_IMAGE"   redis-server --dir /data --dbfilename "$(basename "$1")"   --appendonly no --bind 127.0.0.1 --port 6380
docker exec dev-redis redis-cli -p 6380 INFO keyspace

RDB cluster snapshots are per shard, taken sequentially, not cluster-wide consistent. Follow RESTORE.txt for shard placement; do not merge RDB files.

MongoDB: download the latest anonymized deployment

Use moa:download to list, or moa:download:default for only default. Format mongodump-archive-gz is an uncompressed tar containing data.archive.gz, restore.sh, RESTORE.txt, MANIFEST.txt and POLICY.txt.

backup-base runs mongodump --archive against the anonymized target and refuses while a rule is pending. A live dump is consistent per collection, not one deployment-wide instant; the manifest records the target checkpoint from before the dump so later idempotent streaming can converge. There is no incremental form for the Community-compatible dump path.

Restore

Install mongorestore and use a target MongoDB version compatible with the manifest. The generated script requires an explicit destination URI, verifies the gzip stream before writing, uses --drop, and excludes __devreplicate.* so a development copy does not inherit the replication checkpoint.

set -eu
: "${ARTIFACT:?download and verify first}"
mkdir mongodb-dev
tar -C mongodb-dev -xf "$ARTIFACT"
cd mongodb-dev
./restore.sh 'mongodb://127.0.0.1:27017'

Supported source deployments are MongoDB 6.0+ replica sets or sharded clusters; targets may be MongoDB 4.4+ standalone or replica-set deployments.

Apache Kafka / Redpanda: download an anonymized topic snapshot

Use ka:download to list, or ka:download:prod for only prod. Format ka-topics-v1 is an uncompressed tar with a manifest, executable restore.sh, guidance and one KAREC1 record file per topic partition.

The archive preserves topic partition counts and every record’s partition, key, value, headers, timestamp, tombstone state and null key. It intentionally does not preserve exact offsets, consumer-group positions or transaction markers. backup-capture refuses pending rules; backup-publish refuses a capture whose policy fingerprint is stale.

Restore

Unpack into a new directory and pass a fresh cluster’s bootstrap.servers. The restore creates absent topics with the recorded partition count and refuses an existing topic whose partition count differs.

set -eu
: "${ARTIFACT:?download and verify first}"
mkdir kafka-dev
tar -C kafka-dev -xf "$ARTIFACT"
cd kafka-dev
./restore.sh 127.0.0.1:9092

Restore into a fresh Kafka or Redpanda cluster whose version is compatible with the artifact manifest.

Microsoft SQL Server: native backup and restore

Use msa:download to list, or msa:download:app for only app. Format mssql-bak-tar is an uncompressed tar containing the native data.bak, MANIFEST.txt, RESTORE.txt and executable restore.sh.

backup-base asks SQL Server to write a COPY_ONLY, checksummed full backup and verifies it with RESTORE VERIFYONLY. SQL Server and the command therefore need the same writable backup directory through their respective filesystem paths. Differential and log backup delivery are out of scope.

Restore

set -eu
: "${ARTIFACT:?download and verify first}"
mkdir mssql-dev
tar -C mssql-dev -xf "$ARTIFACT"
cd mssql-dev
MSSQL_PWD=... ./restore.sh app_dev 127.0.0.1 1433 sa /msa-backup /var/opt/mssql/data

The generated script uses RESTORE DATABASE ... WITH MOVE and requires a running SQL Server 2019 or 2022 instance that can read the server-side backup path.

Oracle Database: Data Pump schema export

Use oa:download to list, or oa:download:prod for only prod. Format datapump-schema-tar is an uncompressed tar containing data.dmp, the Data Pump log, manifest and policy record, restore guidance and executable restore.sh.

backup-base takes a full Oracle Data Pump export of the anonymized target schemas pinned to one SCN, then fetches the dump from the server-side DIRECTORY object. Data Pump has no incremental export mode, so every published base is full.

Restore

set -eu
: "${ARTIFACT:?download and verify first}"
mkdir oracle-dev
tar -C oracle-dev -xf "$ARTIFACT"
cd oracle-dev
./restore.sh --connect 'app/pw@//127.0.0.1:1521/FREEPDB1' --upload
# Or, where the target Data Pump directory is directly visible:
# ./restore.sh --connect ... --dump-dir /opt/oracle/dpdump
# Add --remap APP:SCRATCH to restore into another empty schema.

Restore requires sqlplus and a running Oracle 19c-or-later database at least as new as the manifest minimum. Production LogMiner operation requires ARCHIVELOG and sufficient redo retention.

CouchDB: documents and attachments

Use ca:download to list, or ca:download:app for only app. Format couchdb-docs-tar-v1 is an uncompressed tar with one NDJSON document stream per database, attachments inline, manifest.json, guidance and executable restore.sh.

backup-publish reads the anonymized target through _all_docs?include_docs=true&attachments=true and publishes one artifact per database. It preserves _id, the current _rev, attachments and design documents. It does not preserve revision ancestry, tombstones, _local documents or cluster shard placement.

Restore

set -eu
: "${ARTIFACT:?download and verify first}"
mkdir couchdb-dev
tar -C couchdb-dev -xf "$ARTIFACT"
cd couchdb-dev
./restore.sh http://admin:[email protected]:5984 app_dev

The destination database must be absent or empty; partitioned databases are recreated partitioned. Product support is CouchDB 3.x.

Choose a restore recipe

The Docker commands above restore into disposable containers or prepare a physical datadir with the matching tool. Choose your image from the manifest and installed extensions; image names are supplied through required environment variables, not guessed version tags. Export credentials from your secret store, keep tokens out of images and logs, and leave downloaded datasets out of source control.

One command: make dev-db

Save one complete host or Docker recipe above as restore-dev-db.sh with its set -eu line (add that line to the shorter physical/BACKUP recipes). Select a recipe matching the manifest’s format. Save the downloader beside it. Export the token, API origin, database name and the recipe’s engine variables. Recipes intentionally refuse an existing restore directory; stop and retire your old disposable instance before a fresh restore.

.PHONY: dev-db

dev-db:
	@set -eu; ARTIFACT=$$(sh ./fetch-dev-data.sh); export ARTIFACT; sh ./restore-dev-db.sh

Run make dev-db. Download failure or a size/digest mismatch stops the chain before restore. Host Redis runs in the foreground; use the Docker variant for a background service. Physical MySQL stops after prepare and prints the host-specific startup steps.

Safety checks

• Verify the SHA-256 digest before extracting, importing, or starting an engine.
• Restore only into a disposable development instance, never over production or another developer’s working database.
• Confirm the manifest’s configuration version is the approved anonymization configuration for the intended audience.
• Keep the bearer token out of logs, command tracing, Makefile output, images, and committed environment files.
• Apply least privilege: developers need the download scope, not field-management write scope.
• Delete stale artifacts according to your organization’s data-retention policy.