API authentication, policy, and metrics

Run serve as an OAuth2 resource server, validate scoped RS256 bearer tokens, manage anonymization policy, trigger backfills, and expose one authenticated Prometheus endpoint for every configured cluster.

Commercial license RS256 JWT JWKS or static PEM Prometheus

Authentication Scopes SQL fields Redis rules Metrics

One operational interface across the product family. serve provides authenticated policy routes and GET /metrics for every engine family. See developer downloads for product-specific scopes, backups and restore instructions.

The API never issues tokens. Connect it to an OAuth2/OIDC identity provider that signs RS256 JWTs. In production, configure the API certificate and key together; running without TLS is a development-only mode and still requires bearer authentication.

Configure token verification

Every product accepts exactly one key source: a static PEM public key or a JWKS URL. It verifies alg=RS256, requires exp, matches iss exactly, requires the configured value in aud, and checks the space-separated scope claim. A 60-second clock-skew allowance applies to exp and nbf.

Static public key

POSTGRESQL / MYSQL SETTINGS TABLE

openssl rsa -in idp_signing_key.pem -pubout \
  -out /etc/devreplicate/jwt_pub.pem

sqlite3 /etc/patroni-anonymizer/config.db <<'SQL'
INSERT INTO settings(key,value) VALUES
 ('api_listen','0.0.0.0'),
 ('api_port','8443'),
 ('api_tls_cert_file','/etc/devreplicate/tls/server.crt'),
 ('api_tls_key_file','/etc/devreplicate/tls/server.key'),
 ('jwt_public_key_file','/etc/devreplicate/jwt_pub.pem'),
 ('jwt_issuer','https://idp.example.com/'),
 ('jwt_audience','patroni-anonymizer-api');
SQL

Use the same keys in the MySQL configuration database and set the audience appropriate to that service, for example mysql-anonymizer-api.

REDIS SETTINGS TABLE

sqlite3 /etc/redis-anonymizer/config.db <<'SQL'
INSERT INTO settings(cluster_name,key,value) VALUES
 ('default','api_listen','0.0.0.0'),
 ('default','api_port','8443'),
 ('default','api_tls_cert_file','/etc/devreplicate/tls/server.crt'),
 ('default','api_tls_key_file','/etc/devreplicate/tls/server.key'),
 ('default','jwt_public_key_file','/etc/devreplicate/jwt_pub.pem'),
 ('default','jwt_issuer','https://idp.example.com/'),
 ('default','jwt_audience','redis-anonymizer-api');
SQL

JWKS with key rotation

Replace jwt_public_key_file with jwt_jwks_url=https://idp.example.com/.well-known/jwks.json. Startup fails if the first JWKS fetch does not succeed. After a successful load, refresh failures keep the last-known-good keys rather than silently accepting unknown keys.

Start the service

ONE FOREGROUND SERVICE PER CONFIG DATABASE

patroni-anonymizer serve --config /etc/patroni-anonymizer/config.db
mysql-anonymizer   serve --config /etc/mysql-anonymizer/config.db
redis-anonymizer   serve /etc/redis-anonymizer/config.db
mongodb-anonymizer serve /etc/mongodb-anonymizer/config.db
kafka-anonymizer   serve /etc/kafka-anonymizer/config.db

serve fronts every cluster stored in that configuration database. The cluster name belongs in the route; there is no --cluster option for the server.

Issue the narrowest scopes

Product	Read	Write	Download
PostgreSQL / Patroni	`pa:read`	`pa:write`	`pa:download` or `pa:download:<database>`
MySQL / MariaDB	`mya:read`	`mya:write`	`mya:download` or `mya:download:<database>`
Redis	`ra:read` or `ra:read:<cluster>`	`ra:write` or `ra:write:<cluster>`	`ra:download` or `ra:download:<database>`
MongoDB	`moa:read`	`moa:write`	`moa:download` or `moa:download:<database>`
Kafka / Redpanda	`ka:read`	`ka:write`	`ka:download` or `ka:download:<database>`

GET /health and HEAD /health are the only unauthenticated routes.
GET /metrics needs the global read scope because it covers every cluster.
GET /databases needs the global download scope. A database-scoped token can use that database's /latest, manifest, and download routes.
Write scope controls connection destinations and policy. Treat it as an infrastructure-administration credential.
Download scope hands out row or key data. Keep it separate from policy-read access.

CALL THE API

export API_BASE=https://devreplicate-api.example.com:8443
export TOKEN='short-lived-token-from-your-idp'
export CLUSTER=default

curl --fail-with-body \
  -H "Authorization: Bearer $TOKEN" \
  "$API_BASE/clusters"

Add, inspect, edit, and remove SQL fields

PostgreSQL and MySQL/MariaDB use the same routes. PostgreSQL defaults an omitted schema to public; MySQL requires the source database name as schema.

List and add

POSTGRESQL EXAMPLE · PA:READ / PA:WRITE

curl --fail-with-body \
  -H "Authorization: Bearer $TOKEN" \
  "$API_BASE/clusters/$CLUSTER/fields"

curl --fail-with-body -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "schema":"public",
    "table":"users",
    "column":"phone",
    "transform":"mask",
    "params":{"pattern":"XXX-XXX-","keep_last":4}
  }' \
  "$API_BASE/clusters/$CLUSTER/fields"

A created field is always returned with backfill_pending=true. It does not become a complete protection rule for existing data until initial sync or backfill finishes.

Edit safely: replace, then backfill

There is no in-place PATCH endpoint. An edit is an explicit policy migration: remove the old identity, recreate it with the new transform, start backfill, and keep the target unavailable until the pending count reaches zero.

REPLACE A SQL FIELD POLICY

curl --fail-with-body -X DELETE \
  -H "Authorization: Bearer $TOKEN" \
  "$API_BASE/clusters/$CLUSTER/fields/public/users/phone"

curl --fail-with-body -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"schema":"public","table":"users","column":"phone",
       "transform":"hash","params":{"length":16}}' \
  "$API_BASE/clusters/$CLUSTER/fields"

curl --fail-with-body -X POST \
  -H "Authorization: Bearer $TOKEN" \
  "$API_BASE/clusters/$CLUSTER/backfill"

curl --fail-with-body \
  -H "Authorization: Bearer $TOKEN" \
  "$API_BASE/clusters/$CLUSTER/backfill/status"

Deleting does not undo transformed data. It only changes future policy after the replicator reloads configuration. During any replacement, restrict access and verify the backfill result before reopening the replica.

Add, inspect, edit, and remove Redis rules

Redis addresses a rule by numeric ID. The route validates the same key pattern, container type, selector, transform, and parameters as the SQLite configuration loader.

LIST AND ADD A RULE · RA:READ / RA:WRITE

curl --fail-with-body \
  -H "Authorization: Bearer $TOKEN" \
  "$API_BASE/clusters/$CLUSTER/rules"

curl --fail-with-body -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "key_pattern":"profile:*",
    "value_type":"hash",
    "selector":"email",
    "transform":"fake",
    "params":{"kind":"email"}
  }' \
  "$API_BASE/clusters/$CLUSTER/rules"

The create response contains the new id, and the rule is always pending. To edit, delete that ID, create the replacement, and run the cluster backfill.

REMOVE AND ACTIVATE

export RULE_ID=12
curl --fail-with-body -X DELETE \
  -H "Authorization: Bearer $TOKEN" \
  "$API_BASE/clusters/$CLUSTER/rules/$RULE_ID"

# POST the replacement rule, then:
curl --fail-with-body -X POST \
  -H "Authorization: Bearer $TOKEN" \
  "$API_BASE/clusters/$CLUSTER/backfill"
curl --fail-with-body \
  -H "Authorization: Bearer $TOKEN" \
  "$API_BASE/clusters/$CLUSTER/backfill/status"

Use the metrics endpoint

GET /metrics returns Prometheus text exposition for all clusters. It needs the product's global read scope. A newly started server can briefly return 503 until its first background collection completes; let Prometheus retry normally.

MANUAL SCRAPE

curl --fail-with-body \
  -H "Authorization: Bearer $TOKEN" \
  "$API_BASE/metrics"

Engine	Representative series	Watch for
PostgreSQL	`pa_replication_lag_bytes`, `pa_pending_fields`, `pa_unhandled_ddl_total`, `pa_metrics_snapshot_age_seconds`	Lag, pending policy, unhandled DDL, stale collection, and `pa_scrape_error`.
MySQL / MariaDB	`mya_gtid_lag_events`, `mya_pending_fields`, `mya_unhandled_ddl_total`, `mya_metrics_snapshot_age_seconds`	GTID lag, pending policy, DDL queue, stale collection, and `mya_scrape_error`.
Redis	`ra_offset_lag_bytes`, `ra_replicate_up`, `ra_pending_rules`, `ra_backfill_running`	Offset lag, missing replication lease, pending rules, and `ra_scrape_error`.
MongoDB	`moa_replication_lag_seconds`, `moa_checkpoint_age_seconds`, `moa_pending_rules`, `moa_metrics_snapshot_age_seconds`	Lag, an old checkpoint, pending rules, a stale collector, and `moa_scrape_error`.
Kafka / Redpanda	`ka_consumer_lag_records`, `ka_pending_rules`, `ka_metrics_snapshot_age_seconds`	Per-cluster mirror lag, rules awaiting resync, a stale collector, and `ka_scrape_error`.

Prometheus scrape configuration

PROMETHEUS.YML

scrape_configs:
  - job_name: devreplicate-postgres
    scheme: https
    metrics_path: /metrics
    static_configs:
      - targets: ['devreplicate-api.example.com:8443']
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/devreplicate-read.token
    tls_config:
      ca_file: /etc/prometheus/devreplicate-ca.pem
    scrape_interval: 15s
    scrape_timeout: 10s

Use a dedicated, short-lived or regularly rotated token containing only the read scope. Protect the token file, trust the API certificate through the configured CA, and size the Redis scrape timeout for the number of clusters because Redis collection performs bounded source and target operations per cluster.

Recommended alerts

Replication lag remains above the threshold your development workflow can tolerate.
Pending fields or rules remain non-zero beyond the planned policy-change window.
replicate_up is absent/zero, or paused remains set unexpectedly.
Unhandled DDL is non-zero.
A scrape-error series appears, or the metrics snapshot age grows far beyond its refresh interval.
A triggered backfill exits non-zero.