Geo-Replication with MirrorMaker 2

For platform and SRE teams operating multi-datacenter Apache Kafka deployments, geo-replication underpins business continuity, data locality, and regulatory compliance. MirrorMaker 2 (MM2), built on the Kafka Connect framework, is a topology-aware, offset-translating replication engine that understands Kafka metadata, consumer group state, and topic configuration — a departure from the legacy MirrorMaker 1 consumer-producer design. This page covers the operational realities of running MM2 in production: how it works internally, how to configure it for resilience, how to secure replication across untrusted networks, and how to monitor and troubleshoot the pipeline when it drifts from the happy path.

Geo-replication sits inside the broader discipline of Operations, Security & Observability, where replication intersects with authentication, authorization, monitoring, and capacity planning. A misconfigured topology can silently drop records, write offsets that resume consumers in the wrong place, or — in active-active setups — create replication loops that amplify rather than mitigate a failure domain. The goal here is to give you the mental models and concrete configuration patterns to avoid those outcomes.

How MirrorMaker 2 Differs from MirrorMaker 1

Legacy MirrorMaker 1 ran as a set of independent consumer-producer pairs: each pair consumed from a source cluster and produced to a target cluster using a separate consumer group. Three limitations became painful at scale.

First, MirrorMaker 1 had no awareness of topic configuration. Create a compacted topic on the source, and the mirrored topic on the target was created with the default delete cleanup policy, silently breaking the compaction contract. Operators pre-created topics with matching configs by hand — brittle and error-prone once topic counts grew into the hundreds.

Second, offset translation was absent. Source-cluster consumer offsets had no relationship to offsets in the target cluster, so failover meant either accepting data loss (resetting to earliest or latest) or building custom tooling to approximate the resume point.

Third, MirrorMaker 1 could replicate topics with a prefix but had no robust mechanism to prevent loops in active-active topologies. A record produced to cluster A, replicated to B, could be picked up by the reverse flow and written back to A — an amplification loop that saturated inter-datacenter links.

MM2 addresses all three by running on Kafka Connect, inheriting Connect’s distributed coordination, REST management API, and single-message transforms. It deploys three connector types: a MirrorSourceConnector that replicates topic data and configuration, a MirrorCheckpointConnector that translates and emits consumer group offsets, and a MirrorHeartbeatConnector that emits heartbeats to measure connectivity and latency. The design is described in KIP-382: MirrorMaker 2.0, the authoritative reference.

Core Replication Topology and Offset Translation

MM2’s model centers on the replication flow: a directional stream from a source cluster to a target cluster, configured with the source->target. prefix. Each flow’s MirrorSourceConnector consumes from the source and produces into the target, renaming topics according to a replication policy. The default DefaultReplicationPolicy prepends the source cluster alias (separator is a period by default, set via replication.policy.separator): topic orders on cluster us-west becomes us-west.orders on us-east.

The key capability is offset translation. As the MirrorSourceConnector replicates data, it records source-to-target offset mappings in the internal mm2-offset-syncs.<target-alias>.internal topic. By default this topic lives on the source cluster (offset-syncs.topic.location defaults to source). The MirrorCheckpointConnector reads committed consumer group offsets from the source cluster’s __consumer_offsets, uses the offset-sync mappings to translate each position into the corresponding offset on the mirrored topic, and writes those translated checkpoints to the <source-alias>.checkpoints.internal topic on the target cluster.

Translation alone does not move offsets into the target’s __consumer_offsets. That requires sync.group.offsets.enabled (default false). When enabled, MM2 periodically writes the translated offsets into the target cluster’s __consumer_offsets — but only while no active consumer of that group is connected to the target, so it never overwrites the position of a live consumer. The combination lets a consumer group fail over to the secondary cluster and resume near its last committed position without manual offset surgery.

The following snippet replicates from us-west to us-east with offset translation enabled:

# mm2.properties
clusters = us-west, us-east
us-west.bootstrap.servers = kafka-west-1:9092,kafka-west-2:9092
us-east.bootstrap.servers = kafka-east-1:9092,kafka-east-2:9092

us-west->us-east.enabled = true
us-west->us-east.topics = orders\..*, inventory\..*
us-west->us-east.emit.heartbeats.enabled = true
us-west->us-east.emit.checkpoints.enabled = true
us-west->us-east.sync.group.offsets.enabled = true

replication.policy.class = org.apache.kafka.connect.mirror.DefaultReplicationPolicy
replication.factor = 3
checkpoints.topic.replication.factor = 3
heartbeats.topic.replication.factor = 3
offset-syncs.topic.replication.factor = 3

replication.factor sets the replication factor of newly created remote (data) topics; the three *.topic.replication.factor keys above are valid top-level properties that set the replication factor of MM2’s internal checkpoint, heartbeat, and offset-sync topics. All four default — replication.factor to 2, the internal-topic factors to 3 — so override them to match your cluster size, especially in single-broker test setups where 3 will fail topic creation.

Launch a dedicated MM2 cluster with the bundled script:

./bin/connect-mirror-maker.sh mm2.properties

Securing Cross-Datacenter Replication Streams

Replication traffic often traverses untrusted networks or organizational boundaries. Securing it means TLS for transport encryption (optionally mutual TLS for peer authentication) plus SASL/SCRAM or Kerberos for client authentication. MM2 inherits Connect’s security configuration, and you set it per cluster alias, so source and target can use different protocols.

A common pattern is SASL_SSL: TLS encrypts the wire while SCRAM authenticates the client. Configure security for both clusters; this also applies to the producers, consumers, and admin clients MM2 uses internally. The broader mTLS/SCRAM setup is covered in Securing Kafka with mTLS and SASL/SCRAM.

# Security configuration for both clusters
us-west.security.protocol = SASL_SSL
us-west.ssl.truststore.location = /etc/kafka/secrets/truststore.jks
us-west.ssl.truststore.password = ${TRUSTSTORE_PASSWORD}
us-west.sasl.mechanism = SCRAM-SHA-512
us-west.sasl.jaas.config = org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="mm2-west" password="${WEST_SCRAM_PASSWORD}";

us-east.security.protocol = SASL_SSL
us-east.ssl.truststore.location = /etc/kafka/secrets/truststore.jks
us-east.ssl.truststore.password = ${TRUSTSTORE_PASSWORD}
us-east.sasl.mechanism = SCRAM-SHA-512
us-east.sasl.jaas.config = org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="mm2-east" password="${EAST_SCRAM_PASSWORD}";

If you also use mTLS, add ssl.keystore.location/ssl.keystore.password per alias so each side presents a client certificate.

Beyond transport and authentication, the MM2 principal needs ACLs on both clusters: Read and Describe on source topics and Describe on consumer groups (to read committed offsets), plus Create, Describe, and Write on the target’s mirrored and internal topics. The ACL model is detailed in Kafka ACLs and Authorization Best Practices.

A subtlety that bites people: in kafka-acls.sh, --topic is a literal resource name by default. A value like orders.* matches a topic literally named orders.*, not “everything starting with orders.”. For prefix matching you must pass --resource-pattern-type prefixed (and only the bare * acts as a true wildcard, with literal). The source-cluster ACLs:

# Source cluster ACLs for MM2 principal "User:mm2-west"
kafka-acls.sh --bootstrap-server kafka-west-1:9092 --command-config admin.conf \
  --add --allow-principal User:mm2-west \
  --operation Read --operation Describe \
  --topic orders --topic inventory --resource-pattern-type prefixed

kafka-acls.sh --bootstrap-server kafka-west-1:9092 --command-config admin.conf \
  --add --allow-principal User:mm2-west \
  --operation Describe --group "*"

On the target cluster, the principal needs to create and write the mirrored topics and MM2’s internal topics. With DefaultReplicationPolicy the mirrored topics carry the source alias prefix, so replication from us-west lands under us-west.; the checkpoint topic is us-west.checkpoints.internal:

# Target cluster ACLs for MM2 principal "User:mm2-east"
kafka-acls.sh --bootstrap-server kafka-east-1:9092 --command-config admin.conf \
  --add --allow-principal User:mm2-east \
  --operation Create --operation Describe --operation Write \
  --topic us-west --topic mm2-offset-syncs --topic heartbeats \
  --resource-pattern-type prefixed

(us-west as a prefix covers both us-west.<topic> data topics and us-west.checkpoints.internal. If offset-syncs.topic.location is left at its default of source, the offset-syncs topic is created on us-west and only needs an ACL there.)

Active-Active Replication and Loop Prevention

In an active-active topology, two clusters replicate bidirectionally, which raises the risk of a loop: a record produced in A is replicated to B, picked up by the reverse flow, and written back to A. How MM2 handles this depends entirely on the replication policy.

With the default DefaultReplicationPolicy, loop prevention is automatic and based on the topic-name prefix. A topic replicated from us-east arrives on us-west as us-east.orders; because that name already carries the us-east prefix, the us-west->us-east flow recognizes it as remote and does not replicate it back. This provenance check is a property of the prefixing policy itself.

# Active-active configuration with prefixing (loop-safe)
clusters = dc1, dc2
dc1.bootstrap.servers = kafka-dc1-1:9092,kafka-dc1-2:9092
dc2.bootstrap.servers = kafka-dc2-1:9092,kafka-dc2-2:9092

dc1->dc2.enabled = true
dc1->dc2.topics = .*
dc1->dc2.sync.group.offsets.enabled = true
dc1->dc2.emit.checkpoints.enabled = true

dc2->dc1.enabled = true
dc2->dc1.topics = .*
dc2->dc1.sync.group.offsets.enabled = true
dc2->dc1.emit.checkpoints.enabled = true

replication.policy.class = org.apache.kafka.connect.mirror.DefaultReplicationPolicy

IdentityReplicationPolicy (org.apache.kafka.connect.mirror.IdentityReplicationPolicy) preserves the original topic name with no prefix, which keeps application code simpler across failover. The trade-off is significant: with IdentityReplicationPolicy MM2 cannot detect or prevent cycles, because the prefix it would otherwise use as a provenance marker is gone. If you use it, you are responsible for keeping the topology acyclic — for example, by replicating disjoint topic sets in each direction (dc1->dc2.topics and dc2->dc1.topics must not overlap), or by running one-directional replication per topic. Treat IdentityReplicationPolicy as an active-passive / migration tool unless you have rigorously partitioned which topics flow which way. A complete bidirectional walkthrough is in MirrorMaker 2 Configuration for Active-Active Geo-Replication.

Monitoring the Replication Pipeline

MM2 is built on Connect, so it exposes all of Connect’s JMX metrics (e.g. source-record-poll-rate) plus its own metrics under the kafka.connect.mirror group. Scrape JMX with the Prometheus JMX exporter and alert on three things: replication lag, connector health, and offset-translation latency.

MM2’s source metrics are tracked per remote topic-partition on the target — note that the topic name and partition are part of the MBean’s ObjectName, and the measurements (record-age-ms, record-count, etc.) are attributes of that MBean, not separate name= entries:

# Prometheus JMX exporter scrape config for MM2 workers
scrape_configs:
  - job_name: 'mm2'
    static_configs:
      - targets: ['mm2-worker-1:8080', 'mm2-worker-2:8080']

The metrics to watch:

• kafka.connect.mirror:type=MirrorSourceConnector,target=<alias>,topic=<remote-topic>,partition=<n>
• record-age-ms (with -min/-max/-avg): age of a record when it is replicated — the practical replication-lag signal. A sustained rise means the pipeline is falling behind on throughput, bandwidth, or a stalled task.
• replication-latency-ms (-min/-max/-avg): time for records to propagate source→target.
• record-count: number of records replicated source→target.
• byte-rate: average bytes/sec of replicated records.
• kafka.connect.mirror:type=MirrorCheckpointConnector,source=<alias>,target=<alias>
• checkpoint-latency-ms (-min/-max/-avg): time to replicate consumer offsets, i.e. how stale your failover offsets are.
• kafka.connect:type=connector-metrics,connector="<name>" — status attribute reports the connector state (running, paused, failed, etc.); pair with the per-task kafka.connect:type=connector-task-metrics,connector="<name>",task="<id>" status attribute to catch individual task failures.

The Apache Kafka Monitoring documentation lists the baseline JMX metrics, and the Prometheus JMX Exporter project documents scrape configuration.

Handling Failure Modes and Recovery

Production MM2 deployments hit a predictable set of failure modes. Build runbooks around them.

Connector task failure. Tasks fail on transient network errors, credential expiry, or producer errors. Connect retries with backoff governed by errors.retry.timeout and errors.retry.delay.max.ms, but persistent failures need intervention via the Connect REST API:

# Inspect connector and task status
curl -s http://mm2-worker-1:8083/connectors/MirrorSourceConnector/status | jq .

# Restart a failed task
curl -X POST http://mm2-worker-1:8083/connectors/MirrorSourceConnector/tasks/0/restart

(Connector names are generated per flow, e.g. MirrorSourceConnector for the source connector of a given source->target pair; list them with GET /connectors.)

Offset-translation drift. If the MirrorCheckpointConnector falls behind or fails, translated offsets in the target lag the source and consumers resume from stale positions after failover. Inspect a group’s committed offsets on the target with the standard tool:

# Translated offsets for a consumer group on the target cluster
kafka-consumer-groups.sh --bootstrap-server kafka-east-1:9092 \
  --describe --group my-app-group

Compare these against the source group’s offsets to quantify drift, and watch checkpoint-latency-ms.

Network partitions. A partition between datacenters stalls replication. The MirrorHeartbeatConnector (toggled with emit.heartbeats.enabled, default true) writes records to the heartbeats topic at the interval set by heartbeats.topic.replication.factor for durability and emit.heartbeats.interval.seconds for cadence; if heartbeats stop arriving on the downstream side, you have lost connectivity.

Topic-configuration drift. With sync.topic.configs.enabled (default true), MM2 periodically reapplies source topic configs to the mirrored topics, so a manual change on the target is overwritten on the next sync rather than persisting. If you need a target-side config to stick, it must be carried on the source or excluded from sync — do not rely on editing the mirrored topic directly.

Capacity Planning and Performance Tuning

Geo-replication adds steady, predictable load to both clusters. Plan for the replication consumer’s read throughput, inter-datacenter bandwidth, and the target’s ingestion capacity.

tasks.max defaults to 1, which is rarely enough. For the source connector, the usable parallelism is capped at one task per replicated partition; for the checkpoint connector, one task per replicated consumer group. The actual task count is the lower of tasks.max and that ceiling, so set tasks.max to spread partitions across your Connect workers:

us-west->us-east.tasks.max = 16

On the producer side, MM2 uses the standard Kafka producer, tunable per flow with producer.override.*. Larger batches, a little linger, and compression cut bytes on high-latency links:

us-west->us-east.producer.override.batch.size = 1048576
us-west->us-east.producer.override.linger.ms = 50
us-west->us-east.producer.override.compression.type = zstd

Bandwidth is often the binding constraint. Rough math: 100 MB/s of replicated produce over a 1 Gbps link (~125 MB/s) already consumes ~80% of the pipe before TCP and TLS overhead — leave headroom for bursts and post-outage catch-up. After an outage, an unthrottled MM2 will try to drain its backlog at full speed and can saturate target-broker disks, causing produce timeouts that cascade back into the pipeline. Apply client quotas (via kafka-configs.sh) to the MM2 principal so replication catch-up never starves local producers and consumers. KIP-382 covers the throughput model in more depth.