Designing a Multi-Region DR Plan with Confluent Cluster Linking

A disaster recovery (DR) plan for Apache Kafka has to do more than copy bytes between data centers. For an active-passive, multi-region topology, you need consumer offsets, topic configurations, and ACLs to land on the standby cluster in a form that lets clients resume exactly where they left off — without offset translation, without rebuilt ACLs, and without a fragile runbook. Within the broader domain of Operations, Security & Observability, Confluent Cluster Linking is a broker-level primitive built for precisely this: it replicates data and the metadata clients depend on, collapsing much of the operational surface area that makes MirrorMaker 2 (MM2) painful for strict DR. This guide covers the architecture, configuration, security model, monitoring, and — most importantly — the failover and failback runbooks.

Architectural Foundations of Cluster Linking for DR

Cluster Linking differs from MM2 in one decisive way: it runs in the broker, not in a separate Kafka Connect cluster. MM2 consumes from a source and produces to a target as an external application; Cluster Linking establishes a direct link between two clusters and has destination brokers fetch directly from source-partition leaders. As part of that link it replicates consumer offsets, topic configurations, and ACLs as first-class metadata rather than as a bolt-on.

That design removes the offset-translation problem that defines MM2 failovers. Under MM2, committed offsets on the source do not map cleanly to the target’s mirrored partitions, so clients must translate offsets at cutover. With Cluster Linking, enabling consumer.offset.sync.enable=true on the link replicates committed offsets for matching consumer groups onto the destination, and mirror topics keep the same name as their source. A client that fails over therefore resumes from its last committed offset against an identically named topic, with no application-level translation. This is the single largest RTO win, because it deletes the most error-prone step from the cutover.

The link is also directional and configured on one side only. You create it on the DR (destination) cluster pointing back to the primary (source). The destination brokers initiate the connection and pull data and metadata, so network egress originates from the DR cluster. Mirror topics are read-only on the destination, which structurally prevents split-brain while the link is active. This maps cleanly onto an active-passive topology and has real consequences for firewall design, covered below. For a broader comparison of replication patterns, see Disaster Recovery Strategies for Kafka.

Assume a primary cluster in us-east-1 and a DR cluster in eu-west-1. All link and mirror configuration is applied on the DR cluster, since that is where the link lives. The examples use the Confluent CLI; the same operations are available via the REST API and the kafka-cluster-links / kafka-mirrors scripts on Confluent Platform.

First, create the link on the DR cluster, pointing at the primary’s bootstrap server and supplying source credentials:

confluent kafka link create dr-link \
  --cluster <dr-cluster-id> \
  --source-cluster <primary-cluster-id> \
  --source-bootstrap-server pkc-abc123.us-east-1.aws.confluent.cloud:9092 \
  --source-api-key <primary-api-key> \
  --source-api-secret <primary-api-secret> \
  --config consumer.offset.sync.enable=true,consumer.offset.sync.ms=10000

consumer.offset.sync.enable=true is the cornerstone of the DR strategy; it defaults to false, so you must set it explicitly. consumer.offset.sync.ms controls how often offsets sync (default 30000). Ten seconds is a reasonable starting point — tighten it toward 1000 for a smaller offset-replay window at the cost of more sync traffic. The tradeoff is concrete: with a 10-second interval, a consumer that fails over may reprocess up to ~10 seconds of records it had already committed on the primary, so set this against your downstream idempotency, not just your RPO.

Next, create a mirror topic for each topic you want to protect. The destination mirror topic must have the same name as the source topic, which is what makes failover transparent:

confluent kafka mirror create orders \
  --link dr-link \
  --cluster <dr-cluster-id>

confluent kafka mirror create operates on a single topic; it does not accept a regex or wildcard. To keep new topics protected automatically, enable auto-mirroring on the link with auto.create.mirror.topics.enable=true and scope it with auto.create.mirror.topics.filters (a JSON document of include/exclude rules) rather than relying on a per-topic loop. For an existing inventory, script mirror create over an explicit topic list.

Once created, mirror topics are read-only on the DR cluster — any direct produce is rejected, which is what enforces unidirectional flow. To inspect a mirror’s status and per-partition lag use mirror describe; to enumerate mirrors and filter by state use mirror list:

confluent kafka mirror describe orders --link dr-link --cluster <dr-cluster-id>
confluent kafka mirror list --link dr-link --cluster <dr-cluster-id> --mirror-status active

mirror describe reports each partition’s last-synced source offset and the destination’s offset, which is your direct RPO signal. A healthy mirror shows lag near zero under steady throughput.

Security Design: TLS, Network Boundaries, and ACL Propagation

Security here is more than encryption in transit. Cluster Linking uses TLS, and you should require mutual TLS where you control both ends. On Confluent Cloud, links authenticate with API keys; for self-managed Confluent Platform clusters you configure keystores and truststores on both clusters and point the link at them via its security configs.

The link’s direction simplifies firewall rules. Because the DR cluster initiates the connection, you only need outbound access from the DR brokers to the primary’s bootstrap and broker listeners. The primary never opens an inbound path to the DR site. That asymmetry is a genuine operational advantage over connector-based replication, which may require broader bidirectional reachability.

ACL propagation closes the last gap. With acl.sync.enable=true (default false), ACLs from the source migrate to the DR cluster on the interval set by acl.sync.ms (default 5000), and you scope exactly which ACLs migrate with an acl.filters JSON document, supplied on creation via --acl-filters-json-file. After failover, principals keep the permissions they had on the primary, so you avoid maintaining a parallel ACL set by hand.

confluent kafka link create dr-link \
  --cluster <dr-cluster-id> \
  --source-cluster <primary-cluster-id> \
  --source-bootstrap-server pkc-abc123.us-east-1.aws.confluent.cloud:9092 \
  --source-api-key <primary-api-key> \
  --source-api-secret <primary-api-secret> \
  --acl-filters-json-file acl-filters.json \
  --config consumer.offset.sync.enable=true,acl.sync.enable=true

ACL sync only covers ACLs that match your filters, going forward. After enabling it, verify that the pre-existing ACLs you care about are actually present on the DR cluster, and seed any gaps during the setup window — sync is not retroactive validation.

Monitoring, Alerting, and RPO Validation

A DR plan is only as good as your ability to detect degradation before you need to fail over. Cluster Linking exposes JMX metrics on the destination brokers that you scrape with a JMX exporter. Use the documented MBeans — do not guess at metric names:

  • Aggregate link lag (messages): kafka.server.link:type=ClusterLinkFetcherManager,name=MaxLag,clientId=ClusterLink,link-name={linkName} — the maximum lag, in messages, between destination replicas and the source leaders for the link.
  • Per-partition lag (messages): kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=ClusterLinkFetcherThread-{destBrokerId}-{linkName}-{sourceBrokerId},topic={topic},partition={partition},link-name={linkName}.
  • Mirror throughput (bytes/sec): kafka.server:type=BrokerTopicMetrics,name=MirrorBytesInPerSec,topic={topic} (available since Confluent Platform 7.6.1).
  • Mirror state counts: kafka.server:type=cluster-link-metrics,state={state},link-name={linkName},name=mirror-topic-count — watch for mirrors leaving the Mirror/active state.

Lag is reported in messages, not bytes, so size your alert thresholds in messages (or convert via your known average record size). A Prometheus alert built on the JMX-exporter-flattened MaxLag series looks like this — adjust the metric name to match your exporter’s naming rules:

groups:
  - name: kafka_cluster_linking
    rules:
      - alert: HighMirrorLag
        expr: max_over_time(kafka_server_link_clusterlinkfetchermanager_maxlag{name="MaxLag"}[5m]) > 100000
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Cluster link {{ $labels.link_name }} lag exceeds 100k messages"
          description: "MaxLag for link {{ $labels.link_name }} has been above 100,000 messages for 10 minutes. RPO at risk."

This fires when the 5-minute max lag stays above 100,000 messages for 10 minutes. Translate the threshold into time: at a sustained 50,000 records/sec, 100,000 messages of lag is roughly a 2-second replay window — tight enough for most RPO targets. Set the number from your steady-state throughput, not a copied constant: 100k messages on a 1,000 records/sec topic is a 100-second window and a very different RPO. Pair this alert with the per-partition ConsumerLag series and the mirror describe offsets so you can localize lag to specific partitions when it spikes. See the official Cluster Linking metrics reference for the complete MBean list.

Failover Runbook: From Primary Outage to DR Activation

When the primary is unavailable, the procedure must be deterministic. The decisive detail most runbooks get wrong is which command converts a mirror into a writable topic. There are two, and choosing the wrong one in an outage either stalls (waiting on a dead source) or silently widens your data loss:

mirror promote mirror failover
Source must be reachable Yes — verifies the destination has caught up No — converts using already-replicated data
Data loss None (waits for full sync) RPO = link lag at the moment of the outage
Use it for Planned switchovers, migrations, drills A real primary outage
Rehearsal Supports --dry-run to validate without committing Stops mirroring; one-way

In a genuine primary outage the source is gone, so promote cannot complete its catch-up check — use failover.

Step 1: Fail over mirror topics. On the DR cluster, force each protected mirror to a writable topic. This is one-way: the mirror relationship for that topic ends.

confluent kafka mirror failover orders --link dr-link --cluster <dr-cluster-id>

failover accepts multiple topics in one call, which is the fastest way to cut over a known set under stress:

confluent kafka mirror failover orders payments shipments \
  --link dr-link --cluster <dr-cluster-id>

To drive it from the current inventory, list the link’s mirrors as JSON and feed the topic names back in:

confluent kafka mirror list --link dr-link --cluster <dr-cluster-id> -o json | \
  jq -r '.[].mirror_topic_name' | \
  xargs confluent kafka mirror failover --link dr-link --cluster <dr-cluster-id>

Confirm the exact JSON field name against mirror list -o json in your CLI version before wiring it into automation, since the output schema is version-specific.

Step 2: Redirect clients. Because offsets are synced and topic names are preserved, redirection is a bootstrap-server swap. Repoint the DNS record clients use for the broker endpoint (or your service-discovery entries) at the DR cluster. You do not change consumer group IDs or reset offsets. Keep this DNS record on a short TTL (tens of seconds) before an incident — a multi-hour TTL caches the dead primary at clients and becomes your real RTO bottleneck, independent of how fast the mirrors fail over.

Step 3: Validate. Confirm that consumers resumed at their last committed offsets using the standard kafka-consumer-groups tool against the DR bootstrap:

kafka-consumer-groups --bootstrap-server <dr-bootstrap> \
  --group order-service --describe

CURRENT-OFFSET should sit just behind LOG-END-OFFSET, with LAG shrinking as consumers catch up — confirming they picked up where they left off rather than from earliest or latest. A CURRENT-OFFSET of 0 or a group rewound to the log start is the tell-tale that offset sync was not actually enabled on the link.

Failback and Re-Establishing the Primary

After the primary is restored, the topics promoted on the DR cluster hold writes that never reached the primary, so you must reverse replication before cutting back. The clean path is to reverse the link in place rather than rebuild it from scratch. With a bidirectional link, mirror reverse-and-pause flips the original DR mirror topics into sources and pauses the now-stale primary side, so the DR cluster keeps serving while the primary catches up:

# Run on the DR cluster, against the existing link
confluent kafka mirror reverse-and-pause orders payments shipments \
  --link dr-link --cluster <dr-cluster-id>

The original primary topics still hold their old data, so a same-named reverse mirror cannot overlay them directly. On bidirectional links running in KRaft mode, mirror truncate-and-restore rewinds the stale primary topic so it can mirror cleanly from the DR side; otherwise plan to retire the stale primary topics and re-seed them. Watch the reverse direction’s lag fall to zero with mirror describe, then promote on the primary — the DR cluster is reachable now, so the controlled promote (optionally --dry-run first) is appropriate — redirect clients back, and re-establish the original DR-to-primary mirror direction to restore the active-passive posture. Rehearse this whole sequence; the Confluent Cluster Linking documentation and the DR and failover guide cover the promote/failover and bidirectional semantics in detail.

Operational Considerations and Pitfalls

Schemas are not part of the link. Cluster Linking does not replicate Schema Registry. Run a Schema Registry in the DR region and replicate subjects with Schema Linking, which uses a schema exporter to sync schemas across registries. In steady state the primary registry is in READWRITE mode and the DR registry is in IMPORT mode, so only Schema Linking writes to it — an active-passive topology that mirrors the cluster setup. Without it, Avro, Protobuf, and JSON Schema deserialization fails on the DR cluster even when the records are present — and it fails after failover, exactly when you have no slack to debug it.

Size the DR cluster for full production load. It must absorb production throughput after failover and carry replication overhead during normal operation. Watch broker CPU and, especially, network egress/ingress on the DR brokers; under-provisioned DR network bandwidth is a common cause of chronic, RPO-eroding lag that the MaxLag alert above will catch only after it has already accumulated.

Guard against configuration drift. Cluster Linking syncs topic configs from source to destination on the topic.config.sync.ms interval (default 5000), so manual changes made directly on a mirror topic are overwritten. Make all topic-config changes on the primary — for example confluent kafka topic update <topic> --config retention.ms=... against the primary cluster — and let the link propagate them.

An untested plan is a hope. Schedule quarterly drills that exercise the full promote/redirect/validate path: use promote with the source online, run it with --dry-run first to rehearse without committing, and measure actual RTO against target. For the fetch and offset-commit semantics that make name- and offset-preserving failover work, the Kafka protocol reference is the authoritative source.

Cluster Linking removes a large share of the toil in Kafka DR — offset translation, ACL rebuilding, and bidirectional firewall rules — but it is not a turnkey guarantee. It demands the same discipline in monitoring, security, and rehearsal as any tier-zero system. Choose failover over promote when the primary is truly down, alert on the real MaxLag metric in messages sized to your own throughput, keep client DNS on a short TTL, and replicate schemas with Schema Linking — then you have a DR capability that holds up when you actually need it.

Back to Disaster Recovery Strategies For Kafka