MirrorMaker 2 Configuration for Active-Active Geo-Replication

Active-active geo-replication lets multiple Kafka clusters serve producers and consumers simultaneously while keeping their data bidirectionally synchronized. Getting MirrorMaker 2 (MM2) right for this pattern comes down to four concerns: topic naming, cycle prevention, consumer-group offset translation, and write-conflict handling. This guide gives production-ready configuration for platform and SRE engineers building resilient active-active replication across data centers or cloud regions, with the gotchas that bite most teams.

Understanding the Active-Active Replication Model

In an active-active topology, each Kafka cluster handles local producer and consumer traffic independently. MM2 replicates topics bidirectionally so that data produced in one region becomes available in the others. This differs from active-passive, where one cluster is an idle disaster-recovery standby.

The core challenge is preventing infinite replication loops. When cluster us-east replicates a record to eu-west, MM2 must not replicate that same record back to us-east. MM2 solves this with topic renaming, not with any per-record header inspection: the DefaultReplicationPolicy prefixes every replicated topic with its source-cluster alias. A topic that already carries a remote prefix is recognized as remote-origin and is excluded from being mirrored back, which breaks the cycle. This is the single most important fact to get right, and it directly determines which replication policy you should choose (covered below).

Before the configuration specifics, it helps to be familiar with the foundational concepts in Geo-Replication with MirrorMaker 2, which covers MM2 architecture, deployment modes, and basic replication flows. The broader Operations, Security & Observability principles also help you integrate these patterns into a production Kafka platform.

Choosing a Replication Policy (and Why It Matters for Cycles)

MM2’s replication policy controls how source topics are named on the target, and it is the mechanism that makes active-active safe. The two built-in policies differ on exactly one axis — whether the target topic name encodes its origin — and that difference dictates whether a bidirectional flow is loop-safe:

DefaultReplicationPolicy (the default) renames a replicated topic to {source}.{topic} — for example, orders.created from us-east appears as us-east.orders.created on eu-west. Because the prefix encodes provenance, MM2 will not re-replicate us-east.orders.created back toward us-east. This is the recommended policy for active-active. Its trade-off is that consumers that need the full global view must subscribe to both the local topic and the remote-prefixed topic (often via a regex subscription). The separator is configurable via replication.policy.separator (default .); only change it if your topic names already contain periods in a way that would make a prefixed name ambiguous — and if you do, change it identically on every cluster.

IdentityReplicationPolicy preserves the original topic name on the target, so orders.created stays orders.created. This is convenient for one-directional migrations, but per the Apache Kafka documentation it cannot prevent cycles: with identical names on both sides, MM2 has no way to tell a locally produced record from a mirrored one. Using IdentityReplicationPolicy in a bidirectional flow with overlapping topic sets will create a replication loop. Only use it when your topology is strictly acyclic, or when you isolate replicated topics by name so no flow can mirror a topic it already received.

The configuration below uses DefaultReplicationPolicy for a safe bidirectional topology between a cluster in us-east and one in eu-west:

# mm2-active-active.properties
clusters = us-east, eu-west

# US East cluster connection
us-east.bootstrap.servers = kafka-us-east-1:9092,kafka-us-east-2:9092,kafka-us-east-3:9092
us-east.security.protocol = SASL_SSL
us-east.sasl.mechanism = SCRAM-SHA-512
us-east.sasl.jaas.config = org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="mm2-user" \
  password="mm2-password";

# EU West cluster connection
eu-west.bootstrap.servers = kafka-eu-west-1:9092,kafka-eu-west-2:9092,kafka-eu-west-3:9092
eu-west.security.protocol = SASL_SSL
eu-west.sasl.mechanism = SCRAM-SHA-512
eu-west.sasl.jaas.config = org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="mm2-user" \
  password="mm2-password";

# Replication flow: us-east -> eu-west
us-east->eu-west.enabled = true
us-east->eu-west.topics = orders\..*, inventory\..*, shipments\..*

# Replication flow: eu-west -> us-east
eu-west->us-east.enabled = true
eu-west->us-east.topics = orders\..*, inventory\..*, shipments\..*

# Replication policy (default; prefixes target topics with the source alias)
replication.policy.class = org.apache.kafka.connect.mirror.DefaultReplicationPolicy

# Replication factors for MM2 internal topics
heartbeats.topic.replication.factor = 3
checkpoints.topic.replication.factor = 3
offset-syncs.topic.replication.factor = 3

# Connect (dedicated MM2) internal topics
offset.storage.replication.factor = 3
status.storage.replication.factor = 3
config.storage.replication.factor = 3

# Heartbeats, checkpoints, and refresh cadence
emit.heartbeats.enabled = true
emit.checkpoints.enabled = true
sync.topic.acls.enabled = false
refresh.topics.interval.seconds = 30
refresh.groups.interval.seconds = 30

The topics values are regular expressions, so orders\..* matches orders.created, orders.updated, and so on. The DefaultReplicationPolicy then mirrors us-east’s orders.created to us-east.orders.created on eu-west, and vice versa, with no loop.

If you must keep original names (for example, you need consumers to read orders.created unchanged on the failover side), switch to IdentityReplicationPolicy and make the topology acyclic — typically by giving each flow a non-overlapping topic set so the same topic is never mirrored in both directions:

replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy
# Acyclic: each flow replicates a disjoint topic set
us-east->eu-west.topics = us\..*
eu-west->us-east.topics = eu\..*

Whichever policy you choose, set replication.policy.class identically for the source, checkpoint, and heartbeat connectors — MM2 requires them to agree.

Offset Translation and Consumer-Group Failover

MM2 can translate consumer-group offsets between clusters so a consumer that fails over from us-east to eu-west resumes near the correct position in the replicated topic. The MirrorSourceConnector records source-to-target offset mappings in an internal offset-syncs topic, and the MirrorCheckpointConnector (enabled by emit.checkpoints.enabled = true) periodically emits per-group checkpoints into a checkpoints topic on the target. Both the checkpoint and the offset-sync cadence are tunable: emit.checkpoints.interval.seconds defaults to 60, and how aggressively the source connector records new sync points is bounded by offset.lag.max (default 100) — lower it if you need tighter translation accuracy at the cost of more writes to the offset-syncs topic.

With DefaultReplicationPolicy, the checkpoints topic on eu-west is named us-east.checkpoints.internal — the source alias prefixes it, just like data topics. (With IdentityReplicationPolicy it would be the unprefixed checkpoints.internal, which is one more reason that policy is ambiguous in bidirectional setups.)

There are two ways to consume these checkpoints at failover, and the right choice depends on whether the failover consumer group is ever active on the target at the same time as the source:

• Automated write to __consumer_offsets — Set sync.group.offsets.enabled = true (added in Kafka 2.7 by KIP-545; disabled by default and requires emit.checkpoints.enabled = true). The MirrorCheckpointConnector then writes translated offsets directly into the target’s __consumer_offsets, so a consumer can fail over with no manual seeking. Two safeguards matter operationally: it writes offsets only for groups that are not actively consuming on the target, and it will not move a group backward if the target has already consumed further than the translated position. The practical consequence is that for a symmetric active-active topology — where the same group ID runs on both clusters at once — automated sync is a no-op for those active groups by design; it is most useful for active-passive failover or for groups that are idle on the standby side.
• Programmatic translation at cutover — Use org.apache.kafka.connect.mirror.RemoteClusterUtils.translateOffsets(...), which reads the checkpoint topic and returns the corresponding target offsets for a remote group, and seek your consumers explicitly. RemoteClusterUtils does not perform translation itself — it surfaces the mappings the MirrorCheckpointConnector already produced — so checkpoints must be enabled for it to return anything. Choose this when failover is an orchestrated, point-in-time cutover and you want explicit control over exactly when the seek happens.

Inspect group offsets on both clusters to confirm translation is working:

# Source-cluster offsets for the group
kafka-consumer-groups --bootstrap-server kafka-us-east-1:9092 \
  --group orders-processor --describe \
  --command-config mm2-client.properties

# Target-cluster offsets for the same group (translated by MM2)
kafka-consumer-groups --bootstrap-server kafka-eu-west-1:9092 \
  --group orders-processor --describe \
  --command-config mm2-client.properties

The translated offsets will lag the source by the replication delay, and they point at the prefixed topic (us-east.orders.created) on the target. To audit the raw checkpoint records, consume the internal topic directly:

kafka-console-consumer --bootstrap-server kafka-eu-west-1:9092 \
  --topic us-east.checkpoints.internal \
  --from-beginning --property print.key=true \
  --consumer.config mm2-client.properties

See the Apache Kafka documentation on geo-replication and offset translation for the full API and config reference.

Conflict Handling for Concurrent Writes

Active-active replication makes it possible for two regions to write the same logical key concurrently. MM2 provides no built-in conflict resolution: it replicates records as produced, preserving their values, and — with DefaultReplicationPolicy — lands them in separate, source-prefixed topics rather than overwriting one another. Reconciliation is therefore an application concern.

The most reliable strategy is to avoid conflicts by design: give each logical entity a single writer region. Orders originating in Europe are produced in eu-west, orders from North America in us-east; MM2 mirrors each region’s stream to the other so every cluster has the complete picture, with no contended writes. This single-writer model also sidesteps the offset-sync caveat above, since the consuming group on each side reads a distinct prefixed stream rather than competing for the same group position.

When a single-writer model is not feasible, resolve at the application layer using record metadata. Producers can attach a region tag and a logical clock as headers; consumers reconcile conflicting states on read (last-writer-wins, or a domain-specific merge):

ProducerRecord<String, Order> record =
    new ProducerRecord<>("orders.created", orderId, order);
record.headers().add("region", "us-east".getBytes(StandardCharsets.UTF_8));
record.headers().add("logical-clock",
    Long.toString(clock.tick()).getBytes(StandardCharsets.UTF_8));
producer.send(record);

MM2 replicates record headers along with keys and values, so the region tag and clock survive the hop and remain available to consumers on every cluster. Last-writer-wins by wall-clock timestamp is the common default, but it silently drops one of two truly concurrent updates — prefer a logical clock or a CRDT-style merge when losing a write is unacceptable.

Monitoring and Alerting

Production active-active deployments need observability for replication lag, connector health, and connectivity before problems reach applications. MM2 exposes metrics under the kafka.connect.mirror JMX group, which you can scrape into Prometheus and Grafana as part of the broader Operations, Security & Observability practices.

The MBean ObjectNames are templated with target, topic, and partition (and, for checkpoints, source and group). Useful metrics include:

• kafka.connect.mirror:type=MirrorSourceConnector,target=*,topic=*,partition=* exposes record-count, record-rate, record-age-ms, byte-rate, byte-count, and replication-latency-ms (with -min/-max/-avg variants) — per-partition replication throughput and end-to-end delay.
• kafka.connect.mirror:type=MirrorCheckpointConnector,source=*,target=*,group=*,topic=*,partition=* exposes checkpoint-latency-ms — how stale the offset checkpoints are.
• kafka.connect:type=connector-metrics,connector=* exposes status (running, failed, paused, etc.) — connector health.

Two failure modes deserve dedicated alerts because each is invisible to the other’s signal: a rising replication-latency-ms means data is flowing but falling behind RPO, while a flat record-rate of zero on a topic that should be active often means a connector task has silently failed or a topics regex no longer matches — lag can look healthy precisely because nothing is moving. Alert on both, and watch checkpoint-latency-ms separately, since stale checkpoints mean your failover offsets are stale even when data replication is healthy.

A JMX exporter flattens these into Prometheus names such as kafka_connect_mirror_replication_latency_ms. Scrape each MM2 worker:

scrape_configs:
  - job_name: 'mirrormaker2-us-east'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['mm2-us-east-1:8080', 'mm2-us-east-2:8080']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'kafka_connect_mirror.*'
        action: keep

Alert when replication lag exceeds your recovery point objective (RPO). For a 30-second RPO:

groups:
  - name: mm2_alerts
    rules:
      - alert: HighReplicationLag
        expr: max(kafka_connect_mirror_replication_latency_ms_max{target="eu-west"}) > 30000
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "MirrorMaker 2 replication lag exceeds 30s for eu-west"

The Apache Kafka monitoring documentation lists the full set of MM2 metrics and their meanings.

Securing Cross-Cluster Communication

Active-active replication usually crosses network boundaries, so it needs authentication, authorization, and encryption. MM2 runs on Kafka Connect and inherits its security options: SASL/SCRAM, mTLS, and Kerberos for authentication, and TLS for wire encryption.

The example below configures mutual TLS between the two clusters, the recommended approach for cross-data-center links:

us-east.security.protocol = SSL
us-east.ssl.truststore.location = /etc/kafka/secrets/us-east.truststore.jks
us-east.ssl.truststore.password = ${TRUSTSTORE_PASSWORD}
us-east.ssl.keystore.location = /etc/kafka/secrets/us-east.keystore.jks
us-east.ssl.keystore.password = ${KEYSTORE_PASSWORD}
us-east.ssl.key.password = ${KEY_PASSWORD}

eu-west.security.protocol = SSL
eu-west.ssl.truststore.location = /etc/kafka/secrets/eu-west.truststore.jks
eu-west.ssl.truststore.password = ${TRUSTSTORE_PASSWORD}
eu-west.ssl.keystore.location = /etc/kafka/secrets/eu-west.keystore.jks
eu-west.ssl.keystore.password = ${KEYSTORE_PASSWORD}
eu-west.ssl.key.password = ${KEY_PASSWORD}

Mutual TLS (client authentication) is enforced on the brokers via ssl.client.auth=required in each broker’s server.properties, not in the MM2 client configuration. The MM2 principal then needs least-privilege ACLs: READ and DESCRIBE on source topics, WRITE and DESCRIBE (plus CREATE if MM2 auto-creates target topics) on the target, and READ/WRITE/DESCRIBE on the internal checkpoint, heartbeat, and offset-sync topics. If you enable sync.group.offsets.enabled, the principal also needs the group-offset write path on the target. Grant them with kafka-acls:

# Source cluster: read access to topics being replicated
kafka-acls --bootstrap-server kafka-us-east-1:9092 \
  --add --allow-principal User:mm2-user \
  --operation Read --operation Describe \
  --topic orders --resource-pattern-type prefixed \
  --topic inventory --resource-pattern-type prefixed \
  --command-config admin-client.properties

# Target cluster: write access to replicated topics plus MM2 internal topics
kafka-acls --bootstrap-server kafka-eu-west-1:9092 \
  --add --allow-principal User:mm2-user \
  --operation Write --operation Describe --operation Create \
  --topic us-east --resource-pattern-type prefixed \
  --topic us-east.checkpoints.internal \
  --topic us-east.heartbeats \
  --command-config admin-client.properties

Use --resource-pattern-type prefixed for wildcard grants — kafka-acls matches topic names by literal or prefix, not by regex. The Kafka security documentation covers TLS, SASL, and ACL setup for multi-cluster deployments in full.

Operational Practices and Failure Recovery

Run MM2 as a dedicated distributed Connect cluster — at least three workers per region for availability — and keep it separate from Connect clusters running application connectors to avoid resource contention.

When a regional link fails, MM2 connectors or their tasks can enter the FAILED state, and Connect does not auto-restart them. Implement recovery against the Connect REST API. The status endpoint reports both connector and task state; restart tasks (not just the connector) with restart?includeTasks=true&onlyFailed=true, available since Kafka 3.0 (KIP-745):

#!/bin/bash
# mm2-health-check.sh
CONNECTOR="MirrorSourceConnector"
HOST="http://mm2-us-east-1:8083"

state=$(curl -s "$HOST/connectors/${CONNECTOR}/status" | jq -r '.connector.state')
if [ "$state" != "RUNNING" ]; then
  echo "Connector $CONNECTOR is $state; restarting failed tasks..."
  curl -s -X POST \
    "$HOST/connectors/${CONNECTOR}/restart?includeTasks=true&onlyFailed=true"
fi

Run it every minute via cron or a Kubernetes CronJob. For production, gate the restart on confirmed network connectivity to avoid hammering a partitioned link, and route persistent failures to your incident-management platform.

Finally, exercise the topology with chaos testing: simulate network partitions, broker loss, and topic reconfigurations to confirm replication degrades gracefully and recovers without data loss. The two things to verify explicitly are that the prefix-based cycle prevention holds under a flapping link (no topic ever mirrors back onto itself) and that translated failover offsets land where consumers expect. Document the runbooks and make sure on-call engineers know where the prefixed topics, checkpoint topics, and failover procedure live.