Disaster Recovery Strategies for Kafka

Apache Kafka often sits on the critical path for data in modern organizations, which makes its resilience a hard requirement for platform and site reliability engineers. A sound disaster recovery (DR) strategy keeps business-critical pipelines alive through the loss of a region, a storage failure, or a prolonged network partition. Unlike stateless service failover, Kafka DR means coordinating stateful brokers, preserving ordering guarantees, managing consumer offsets, and ensuring that failover and failback do not introduce data loss or duplication. This page is a technical framework for designing, implementing, and validating DR for production Kafka clusters, with a focus on the trade-offs between recovery objectives, operational complexity, and cost.

Any Kafka DR plan rests on the broader principles of Operations, Security & Observability, which provide the monitoring and access-control layers that make a recovery procedure auditable and repeatable. Before you design replication topologies, make sure your authentication and authorization are applied consistently across both sites. A DR cluster whose Kafka ACLs and Authorization Best Practices were never replicated is an incident waiting to happen. The transport between regions must also be encrypted and mutually authenticated, as covered in Securing Kafka with mTLS and SASL/SCRAM. These primitives are the baseline on which replication streams are built.

Understanding Recovery Objectives and Architectural Trade-offs

Every DR design starts with two business-defined metrics: Recovery Time Objective (RTO) and Recovery Point Objective (RPO). RTO is the maximum tolerable downtime before service is restored; RPO is the maximum tolerable data loss, measured in time. In Kafka terms, an RPO of zero means no acknowledged message is lost during a disaster, which demands synchronous replication. An RTO under five minutes demands automated failover with pre-warmed infrastructure and continuous health probing. These are financial decisions as much as technical ones: they dictate the replication strategy, the distance between sites, and the acceptable performance penalty on producers.

The fundamental tension is between consistency, availability, and latency. Synchronous replication across regions guarantees zero data loss but adds write latency proportional to the inter-region round trip. Asynchronous replication decouples producer latency from the DR site but exposes a data-loss window bounded by replication lag. A stretched cluster, where brokers span multiple data centers within a metro area, can replicate synchronously at low latency but cannot survive a regional disaster. Choosing well means mapping your application’s consistency requirements onto Kafka’s replication semantics.

Kafka’s built-in replication protocol operates within a single cluster and is not designed for cross-region WAN links. Intra-cluster replication uses a leader-follower model where followers fetch from the leader, and the acks=all producer setting ensures that all in-sync replicas (ISRs) acknowledge a write before it is committed. When a broker fails, an ISR member is elected leader and the cluster continues without data loss. But this assumes the low-latency, reliable connectivity that rarely holds across regions separated by hundreds of miles. For DR you must layer an additional replication mechanism on top of intra-cluster replication.

Quantifying RPO and RTO for Kafka Workloads

To turn business requirements into technical parameters, instrument your replication pipeline and measure actual replication lag under production load. Replication lag is the time delta between a message being produced to the primary cluster and that message becoming available on the DR cluster. It directly determines your achievable RPO under asynchronous replication.

# Measure consumer group lag on the DR cluster to estimate RPO
kafka-consumer-groups.sh --bootstrap-server dr-broker:9092 \
  --group dr-replication-consumer \
  --describe

The output includes a LAG column showing the number of messages each partition is behind the log-end offset. A sustained lag of 10,000 messages at a production rate of 5,000 messages/second implies an RPO of roughly 2 seconds.

For RTO, measure the end-to-end time from detecting a primary failure to having producers and consumers successfully connected to the DR cluster. That includes failure detection, DNS propagation, client reconnection, offset translation, and any manual validation. Automating it requires a health-checking control plane that distinguishes transient network blips from genuine regional failures. The probes should read real Kafka MBeans exposed over JMX (for example via the Prometheus JMX exporter), such as kafka.controller:type=KafkaController,name=ActiveControllerCount (exactly one active controller cluster-wide) and kafka.network:type=RequestMetrics,name=TotalTimeMs per request type.

Replication Topologies for Cross-Region DR

Kafka offers several replication topologies for DR, each with distinct operational characteristics. The right choice depends on your RPO/RTO targets, the number of regions, and whether you need active-active or active-passive operation.

MirrorMaker 2: The Standard Asynchronous Replication Engine

MirrorMaker 2 (MM2) is the official Kafka Connect-based tool for replicating data between clusters. It builds on the original MirrorMaker and adds automatic topic-configuration sync, consumer-offset translation for failover, and heartbeat/health topics. MM2 runs on Kafka Connect, consuming from the primary cluster and producing to the DR cluster. It is asynchronous: a producer to the primary is never blocked on the DR cluster.

The replication flow uses a MirrorSourceConnector that reads from the primary cluster and a MirrorCheckpointConnector that periodically emits translated consumer-group offsets to the DR cluster. This offset translation is what lets consumers resume from the correct position after a failover, instead of relying on auto.offset.reset, which would cause duplication or loss.

# MirrorMaker 2 connector configuration (dedicated mode, primary -> dr)
name = primary-to-dr-mirror
connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
tasks.max = 16
source.cluster.alias = primary
target.cluster.alias = dr
topics = .*
replication.factor = 3
refresh.topics.interval.seconds = 30
sync.topic.configs.enabled = true
sync.topic.acls.enabled = true
emit.heartbeats.enabled = true
heartbeats.topic.replication.factor = 3
offset-syncs.topic.replication.factor = 3
# Offset translation for consumer failover
sync.group.offsets.enabled = true
offset.lag.max = 100

sync.topic.acls.enabled (default true) is important: it replicates topic-level ACLs to the DR cluster, keeping the authorization model in sync rather than relying on manual, error-prone synchronization. This ties directly into Kafka ACLs and Authorization Best Practices. Note that sync.group.offsets.enabled defaults to false, so you must set it explicitly to have MM2 write translated offsets into the DR cluster’s __consumer_offsets.

MM2’s offset translation maintains a mapping between primary and DR offsets in an internal topic. The mapping is updated by the checkpoint connector on a configurable interval, so the translated offset always lags the actual replication position by a small, bounded amount.

# Deploy MirrorMaker 2 in dedicated mode
bin/connect-mirror-maker.sh config/mm2.properties

# Verify replication progress for a remote (renamed) topic on the DR cluster
kafka-consumer-groups.sh --bootstrap-server dr-broker:9092 \
  --describe --group primary.orders

By default MM2 prefixes replicated topics with the source alias (primary.orders on the DR cluster for orders on the primary). Plan your topic-naming and consumer configuration around this, or configure a custom replication.policy.class if you require identical names.

Stretched Clusters and Synchronous Replication

For an RPO of zero you need synchronous replication, where a produce request is not acknowledged until the write is committed across both sites. Kafka does not support synchronous cross-cluster replication, but you can approximate it by stretching a single cluster across two (ideally three) data centers with low-latency links. Brokers are distributed across sites, each partition has replicas in multiple locations, and min.insync.replicas requires acknowledgment from enough replicas that at least one in another site has the write before the producer is acked.

# Site A brokers set broker.rack=dc1; Site B brokers set broker.rack=dc2
# Create a topic that forces cross-DC replication
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic critical-events \
  --partitions 12 \
  --replication-factor 4 \
  --config min.insync.replicas=3 \
  --config unclean.leader.election.enable=false

broker.rack (not rack.id) drives rack-aware replica placement, so replicas are spread across data centers. With replication-factor=4 and min.insync.replicas=3 you can lose one broker without losing availability, and with the racks above the ISR will span both sites. unclean.leader.election.enable=false prevents an out-of-ISR replica from becoming leader, which would violate the RPO guarantee.

The hard constraint is latency. The leader waits for ISR followers to acknowledge before responding to the producer, so if inter-DC latency exceeds a few milliseconds, producer throughput degrades sharply. A two-site stretch also has a quorum problem: losing one site can leave the surviving site unable to form a metadata quorum. In practice, stretched clusters are viable only for metro-area deployments (sites within roughly 50 km), and KRaft/ZooKeeper quorum members should be spread across at least three failure domains.

Confluent Cluster Linking for Managed DR

For organizations on Confluent Platform or Confluent Cloud, Cluster Linking provides a managed replication mechanism built into the brokers themselves — there is no MirrorMaker process or Kafka Connect cluster to deploy. Mirror topics are byte-for-byte replicas: the same messages land on the same partitions at the same offsets. Because offsets are preserved and consumer offsets can be synced, consumers fail over without any offset translation.

Cluster Linking is asynchronous: the source broker leads and the destination broker fetches, so it does not block producers on the primary and does not provide a synchronous, zero-RPO mode. Consumer-offset sync, ACL sync, and topic-config sync each run on a configurable frequency (at most once per second). This makes Cluster Linking simpler to operate than MM2 for active-passive DR, at the cost of being Confluent-specific.

For a deeper treatment, see Designing a Multi-Region DR Plan with Confluent Cluster Linking, which covers bidirectional links and controlled failovers.

Consumer Offset Management and Failover Procedures

The hardest part of Kafka DR is rarely replicating the data — it is making consumers resume correctly after a failover. A consumer’s position is its committed offsets in the __consumer_offsets topic on the primary cluster. After failing over to the DR cluster, those raw offsets are meaningless unless they have been translated (MM2) or preserved (Cluster Linking).

Offset Translation with MirrorMaker 2

MM2’s checkpoint connector records the correspondence between primary and DR offsets for each consumer group in offset-sync and checkpoint topics. The raw offset-sync data lives in an internal topic named mm2-offset-syncs.<source-alias>.internal (the alias is whatever you configured, e.g. primary). Translated checkpoints are emitted to a per-group checkpoint topic on the DR cluster. The accuracy of translation depends on the checkpoint interval and the replication lag.

# Inspect a consumer group's lag and state on the DR cluster
kafka-consumer-groups.sh --bootstrap-server dr-broker:9092 \
  --group primary.orders \
  --describe --state

A translated offset can point slightly behind the actual replicated data, so consumers may reprocess a few messages. If your application cannot tolerate duplicates, implement idempotent processing or use transactional, exactly-once semantics.

Consumer Reconfiguration for Failover

Consumers must be pointed at the DR cluster during failover, via DNS-based redirection (update the bootstrap hostname) or application-level configuration that knows about both clusters. The Kafka client does not perform automatic cross-cluster failover, so this logic lives in your application or platform layer.

// Consumer configuration; failover handled by the application
Properties props = new Properties();
props.put("bootstrap.servers", "primary-broker:9092");
props.put("group.id", "my-application");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
// Disable auto-commit so offsets are managed explicitly during failover
props.put("enable.auto.commit", "false");
// Static group membership reduces rebalances on transient failures
props.put("group.instance.id", "instance-1");

Listing both clusters in a single bootstrap.servers does not give you failover — the client treats all entries as one logical cluster, and a DR broker that answers metadata requests for a different cluster will confuse the client. Switch the endpoint deliberately on failover instead.

After failover, seek to the translated offset rather than trusting the primary’s committed offset. MM2 ships a RemoteClusterUtils helper for programmatic translation. Its translateOffsets method takes the connection properties, the remote (source) cluster alias, the consumer group, and a java.time.Duration timeout — there is no separate target-alias argument:

import org.apache.kafka.connect.mirror.RemoteClusterUtils;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.common.TopicPartition;
import java.time.Duration;
import java.util.Map;
import java.util.HashMap;

Map<String, Object> mm2Config = new HashMap<>();
// Points at the DR cluster, which hosts the checkpoint topics
mm2Config.put("bootstrap.servers", "dr-broker:9092");

// translateOffsets(properties, remoteClusterAlias, consumerGroupId, timeout)
Map<TopicPartition, OffsetAndMetadata> translatedOffsets =
    RemoteClusterUtils.translateOffsets(
        mm2Config, "primary", "my-application", Duration.ofMillis(10_000));

translatedOffsets.forEach((tp, offset) ->
    consumer.seek(tp, offset.offset()));

Validating DR Readiness Through Continuous Testing

A DR plan that has never been exercised is a hope, not a plan. Test regularly to confirm that replication is healthy, offset translation is accurate, and failover completes within RTO. Make it automated and scheduled, not improvised during an incident.

Chaos Engineering for Kafka DR

Chaos engineering injects controlled failures to verify graceful degradation and recovery. For Kafka DR, that means simulating network partitions, broker failures, and region outages. Tools such as the Chaos Toolkit and Litmus provide declarative experiment definitions. To cut MM2’s connectivity to the DR cluster, use the pod-network-loss experiment with DESTINATION_IPS scoped to the DR brokers (the pod-network-partition experiment, by contrast, severs all ingress/egress via a NetworkPolicy and does not take per-destination IPs):

# Drop MM2 -> DR cluster traffic to test replication-lag alerting
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: kafka-dr-network-loss
spec:
  engineState: "active"
  appinfo:
    appns: "kafka"
    applabel: "app=connect-mirror-maker"
    appkind: "deployment"
  chaosServiceAccount: pod-network-loss-sa
  experiments:
    - name: pod-network-loss
      spec:
        components:
          env:
            - name: TARGET_CONTAINER
              value: "connect-mirror-maker"
            - name: DESTINATION_IPS
              value: "10.20.0.11,10.20.0.12"   # DR broker IPs
            - name: NETWORK_INTERFACE
              value: "eth0"
            - name: NETWORK_PACKET_LOSS_PERCENTAGE
              value: "100"
            - name: TOTAL_CHAOS_DURATION
              value: "300"

During the experiment, assert your hypothesis from your own observability stack — for example, that the HighReplicationLag alert defined below fires within the chaos window and clears afterward — rather than embedding fragile inline parsing of CLI output in the experiment.

Data Integrity Validation

Beyond infrastructure testing, validate that DR data is correct and complete by comparing message counts (and, for the paranoid, sample payloads or checksums) between clusters. A simple periodic job can compare per-topic log-end offsets.

#!/bin/bash
# Compare total messages between primary and DR for a topic.
# kafka-get-offsets.sh prints "topic:partition:offset"; sum the offsets.
TOPIC="orders"
PRIMARY_COUNT=$(kafka-get-offsets.sh --bootstrap-server primary:9092 \
  --topic "$TOPIC" --time -1 | awk -F: '{sum += $3} END {print sum}')
DR_COUNT=$(kafka-get-offsets.sh --bootstrap-server dr:9092 \
  --topic "primary.$TOPIC" --time -1 | awk -F: '{sum += $3} END {print sum}')
LAG=$((PRIMARY_COUNT - DR_COUNT))
if [ "$LAG" -gt 1000 ]; then
  echo "CRITICAL: replication backlog for $TOPIC is $LAG messages"
  exit 1
fi

This is an approximation: log-end offsets count compacted/retained records and reset on retention, and MM2 renames topics on the DR side (primary.orders), so account for both. kafka-get-offsets.sh replaces the older kafka-run-class kafka.tools.GetOffsetShell --broker-list ... invocation, whose --broker-list flag is deprecated in favor of --bootstrap-server.

Operationalizing Failover and Failback

Failover redirects traffic from the primary to the DR cluster. Failback is the reverse, restoring the primary as the active site once the disaster is resolved. Both must be codified in version-controlled, reviewed, regularly exercised runbooks.

Automated Failover Triggers

Automated failover needs a control plane that watches cluster health and decides based on predefined criteria. Deploy it in a third, independent site so the same disaster cannot take it down with the primary. Combine multiple signals — controller presence, produce error rate, consumer lag, and inter-site connectivity — to avoid false positives.

# Simplified failover decision logic
def should_failover(primary_metrics, dr_metrics):
    """Return True only if the primary is clearly down and the DR is ready."""
    primary_unhealthy = (
        primary_metrics["active_controllers"] == 0 and
        primary_metrics["produce_error_rate"] > 0.5 and
        primary_metrics["under_replicated_partitions"] > 100
    )
    dr_healthy = (
        dr_metrics["active_controllers"] >= 1 and
        dr_metrics["replication_lag_seconds"] < 60 and
        dr_metrics["disk_usage_percent"] < 85
    )
    return primary_unhealthy and dr_healthy

Failback and Data Reconciliation

Failback is harder than failover because histories diverge: the primary may hold data it never replicated before the disaster, and the DR cluster has accepted writes during the outage. The safest approach is to treat the DR cluster as the source of truth after failover, replicate it back to the rebuilt primary with a second MM2 flow whose source and target are swapped, and switch traffic back only once the primary has caught up.

# Reverse replication from DR back to primary during failback.
# Use a separate dedicated-mode config with source/target swapped.
bin/connect-mirror-maker.sh config/mm2-reverse.properties

Consumer offsets need the same care on the way back. Consumers that ran on the DR cluster hold offsets valid only there; before switching back, translate them into the primary’s offset space (via the reverse flow’s checkpoints), or reset to the earliest unprocessed offset and accept some reprocessing. If you ran the original primary as a stretched or active-active topology, beware that data written on both sides during a split must be reconciled at the application level — Kafka will not merge it for you.

Monitoring and Alerting for DR Pipelines

DR monitoring needs metrics beyond standard cluster health: the replication pipeline’s throughput, lag, and error rates, plus the DR cluster’s own readiness to take production traffic. Key signals:

  • Replication lag in messages and time — the primary RPO indicator; alert when it exceeds your RPO budget.
  • MirrorMaker connector and task status — failed tasks mean replication has stopped.
  • Offset-translation lag — the gap between the latest checkpoint and the actual replicated position.
  • DR cluster under-replicated partitions — a DR cluster that is itself unhealthy cannot serve as a failover target.
  • Inter-site network throughput and latency — degradation causes replication backpressure.
# Prometheus alerting rules for Kafka DR
# Metric names assume the Prometheus JMX exporter with lower-cased,
# underscore-joined MBean naming.
groups:
  - name: kafka_dr_alerts
    rules:
      - alert: HighReplicationLag
        expr: kafka_connect_mirror_source_connector_replication_latency_ms > 10000
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "MirrorMaker source replication latency exceeds 10s"

      - alert: MirrorMakerTaskFailure
        expr: kafka_connect_connector_tasks{state="failed"} > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "A MirrorMaker connector task has failed"

      - alert: DRClusterUnderReplicated
        # From kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
        expr: kafka_server_replicamanager_underreplicatedpartitions{cluster="dr"} > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "DR cluster has under-replicated partitions"

Exact Prometheus metric names depend on your exporter and its naming rules, so confirm them against your own JMX scrape before wiring alerts. Route these to on-call, integrate with your incident platform, and tune thresholds against your specific RPO/RTO. Review them quarterly as part of your DR testing cadence.

The practices here are part of the broader discipline of Operations, Security & Observability. A DR strategy is only as strong as the operational rigor around it — the runbooks, the monitoring, the access controls, and the regular testing that turn a plan on paper into a capability you can trust.

In this section

1 guide in this area.

Back to Operations Security Observability