Schema Evolution and Compatibility in Confluent Schema Registry

In event-driven architectures powered by Apache Kafka, schemas are the contract between producers and consumers. As requirements shift, schemas evolve—fields are added, types refined, structures reorganized. Without discipline, a single field addition cascades into deserialization failures, corrupted stream-processor state, and silent data loss in downstream sinks. The Confluent Schema Registry is the enforcement layer that makes evolution safe, but only when operators understand its compatibility modes and apply them correctly within their broader Schema Registry & Connect Ecosystem.

This page addresses the operational realities of schema evolution for platform engineers, SREs, and backend developers running Kafka in production: how to choose between backward, forward, and full compatibility for a given subject; what a safe Protobuf evolution looks like when field-number constraints collide with team workflows; how to gate compatibility checks in CI/CD so a breaking pull request cannot merge; and how to recover when a violation slips through—because in production, eventually one will.

We assume familiarity with the registry’s role as a centralized metadata store and its REST API. For deployment topologies and the registry/Connect relationship, see Kafka Connect Architecture and Deployment Modes. Here we focus on the lifecycle of a schema after its initial registration.

Understanding Compatibility Types as Operational Contracts

The Schema Registry supports seven compatibility levels (plus NONE), but three form the backbone of production strategies: BACKWARD, FORWARD, and FULL. The choice is not merely a configuration flag—it encodes a promise about the direction of data flow and the order in which producers and consumers are deployed.

Backward compatibility (BACKWARD) guarantees that a consumer using the new schema can read data written by producers using the previous schema. This is the default and the most common choice for data-at-rest topics where consumers are upgraded after producers. If you add a field with a default value, the new consumer can deserialize old records because the missing field is populated by the default. The operational contract is: deploy consumers first.

Forward compatibility (FORWARD) guarantees that a consumer using the previous schema can read data written by producers using the new schema. This is essential when producers are deployed before consumers—common in continuous delivery pipelines where services push updates independently. Under FORWARD you may delete fields and add fields; the operational contract is: deploy producers first. The How to Evolve Avro Schemas with Forward Compatibility page explores this pattern in depth.

Full compatibility (FULL) combines both guarantees. It is the strictest of the three and is the right choice for topics that serve as enterprise-wide integration points where deployment ordering cannot be controlled. Under FULL, the only safe changes are adding or removing optional fields (fields that have a default). The cost is reduced flexibility.

The transitive variants—BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, and FULL_TRANSITIVE—check against all previously registered schemas in the subject, not just the immediate predecessor. The non-transitive modes check only against the latest version. Use a transitive mode for long-lived subjects where a consumer might still hold any historical schema version.

Use this table to pick a mode from the deployment facts you already know:

Set the level globally or per-subject via the REST API. Compatibility is configured per subject (typically <topic>-value / <topic>-key), not per topic:

# Set the global default to backward compatibility
curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "BACKWARD"}' \
  http://schema-registry:8081/config

# Override for a specific subject to full compatibility
curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FULL"}' \
  http://schema-registry:8081/config/my-topic-value

The mode must align with your deployment topology. In a Change Data Capture with Debezium and Kafka pipeline, the Debezium source connector is the producer and typically evolves its schema backward-compatibly, because downstream consumers are often owned by separate teams with independent release cycles.

Avro Schema Evolution: Rules, Defaults, and Aliases

Avro remains the most widely used serialization format in the Confluent ecosystem, and its evolution rules are well-defined but nuanced. The core principle is Avro schema resolution: the reader projects the writer’s data onto its own schema, applying defaults, promotions, and aliases as specified in the Avro specification.

Adding and Removing Fields

Adding a field is backward-compatible only if the field has a default value. Without a default, a consumer using the new schema cannot deserialize records written without that field. The following addition is safe under BACKWARD:

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null}
  ]
}

Removing a field is not backward-compatible if that field had no default, because a reader of the old schema would encounter a missing field with no fallback. Removing a field is forward-compatible. Conversely, removing an optional field (a null union with a default) is backward-compatible because the new reader supplies the default. This asymmetry is why FULL permits only changes that are safe in both directions: adding or removing fields that carry defaults.

Type Promotions and Unions

Avro permits these type promotions during schema resolution: int → long, float, or double; long → float or double; float → double; and string ↔ bytes (both directions). These are safe because the reader can up-convert the writer’s value. The reverse numeric direction—narrowing a long to an int, for example—is not a valid promotion and will be rejected by the compatibility check.

Unions require care. Adding a branch to a union is backward-compatible (a new reader can read old data that used an existing branch), but removing a branch is not, because old data may have been written with the now-missing type. Records written with a type the reader’s union no longer contains will fail resolution.

Enum Evolution and the Forward-Compatibility Trap

Enums carry a subtle, version-dependent rule. Since Confluent Platform 5.4.0 (Avro 1.9.1), an enum may declare a default symbol: if the writer used a symbol the reader’s enum does not contain, the reader substitutes that default instead of erroring. Without a default, encountering an unknown symbol fails resolution.

The trap: adding a symbol to an enum passes the checker under FORWARD, yet it is not truly forward-safe—data written with the new symbol cannot be read by an older schema that lacks both the symbol and an enum default (confluentinc/schema-registry #601). Always give enums you expect to grow a default symbol, and test the specific change against the registry rather than trusting a green check.

Aliases for Renaming

Renaming a field breaks compatibility unless you use aliases. Per the Avro spec, aliases are declared on the reader’s schema and rewrite the writer’s schema during resolution—so the new field carries an alias of the old name:

{
  "name": "User",
  "fields": [
    {"name": "userId", "type": "string", "aliases": ["id"]}
  ]
}

This lets a consumer using the new schema (userId) read data written with the old field name (id): the reader treats the writer’s id as though it were named userId. Confluent’s compatibility checker validates renames via aliases on a best-effort basis; always test the specific change against the registry before relying on it.

Testing Compatibility via the REST API

Before registering a schema, validate it against the existing versions using the compatibility endpoint. This is the building block for CI/CD gating:

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"age\",\"type\":\"int\",\"default\":0}]}"}' \
  http://schema-registry:8081/compatibility/subjects/my-topic-value/versions/latest

A 200 response with {"is_compatible": true} confirms the evolution is safe under the subject’s configured level. {"is_compatible": false} (still a 200) means the change is rejected; add ?verbose=true to the URL to get the specific rule messages explaining why.

Protobuf and JSON Schema: Evolution Constraints

While Avro dominates the Kafka ecosystem, Protobuf and JSON Schema are increasingly common, especially in organizations standardizing on gRPC or RESTful APIs. Their evolution rules differ significantly from Avro’s, and misunderstanding these differences is a frequent source of production incidents.

Protobuf Field Numbering and Reserved Fields

Protobuf relies on field numbers, not field names, for wire serialization. You can rename a field freely without breaking wire compatibility, but you must never reuse a field number for a different field. The Protobuf Serialization with Kafka Streams: A Complete Example page demonstrates this with a full Kafka Streams topology.

To remove a field safely, mark its number (and optionally its name) as reserved to prevent accidental reuse:

message User {
  reserved 2;
  reserved "old_email";
  string id = 1;
  string name = 3;
}

Adding a field is generally both backward- and forward-compatible in Protobuf because unknown fields are preserved/ignored by readers. Be cautious with oneof: adding a member is safe at the wire level, but old consumers will see the new member as an unset case and may mishandle it.

JSON Schema and the Compatibility Checker

JSON Schema support applies the same backward/forward/full model, but the checker reasons over the JSON Schema structure. Adding a property is backward-compatible if it is not required. Removing a required property breaks readers that expect it. The following is safe under BACKWARD because the new email field is optional:

{
  "type": "object",
  "properties": {
    "id": {"type": "string"},
    "name": {"type": "string"},
    "email": {"type": "string"}
  },
  "required": ["id"]
}

The behavior of additionalProperties is the common trap. With the default additionalProperties: true, readers tolerate unknown fields, easing forward compatibility. If you set additionalProperties: false, adding a field becomes a forward-breaking change because an old (strict) reader rejects the extra data.

Schema References and Shared Types

Avro, Protobuf, and JSON Schema all support schema references, letting you define a type once and reuse it across subjects. When you evolve a referenced schema, compatibility is evaluated for the importing subjects that pin to it—so a change to a shared type can break a subject you didn’t touch. Enumerate the subjects that reference a schema before changing it:

# Find which subjects/versions reference a given subject
curl http://schema-registry:8081/subjects/my-shared-type-value/versions/latest/referencedby

This referencedby endpoint returns the schema IDs that import the given version—use it instead of grepping the subject list, which only matches names, not actual reference edges.

CI/CD Integration: Enforcing Compatibility in Pipelines

Schema evolution must be a gate in your deployment pipeline, not an afterthought. The REST API makes integration straightforward, but the operational details matter.

The Schema Promotion Workflow

The standard pattern, detailed in Promoting Schemas Across Dev, Staging, and Prod with CI/CD, is to validate a proposed schema against the target registry, then register it. Environments commonly run different compatibility levels—NONE in development for rapid iteration, BACKWARD (or stricter) in staging and production.

A CI step for a pull request that modifies a schema can call the compatibility endpoint directly. The endpoint returns HTTP 200 whether or not the schema is compatible, so check the is_compatible field in the body rather than the status code:

#!/usr/bin/env bash
set -euo pipefail

# Read the .avsc and embed it as a JSON string in the request body
SCHEMA_STRING=$(jq -Rs '.' < src/main/avro/user.avsc)

RESULT=$(curl -s \
  -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "{\"schema\": ${SCHEMA_STRING}}" \
  "http://staging-schema-registry:8081/compatibility/subjects/user-value/versions/latest")

if [ "$(echo "$RESULT" | jq -r '.is_compatible')" != "true" ]; then
  echo "Schema is not compatible with staging. Merge blocked."
  echo "$RESULT"
  exit 1
fi

Producer-Side Registration: Pin the Schema, Don’t Auto-Register

A breaking schema most often reaches production not through a forgotten CI check but through the producer itself silently registering a new version at runtime. By default the Avro/Protobuf/JSON serializers run with auto.register.schemas=true: the first record carrying an evolved schema registers it on the spot, bypassing your reviewed promotion flow. In any environment where schemas are promoted through CI, disable that and pin producers to the reviewed version:

• Set auto.register.schemas=false so the producer never registers; an unregistered schema fails fast instead of leaking a new version.
• Set use.latest.version=true (default false; only takes effect when auto-register is off) so the serializer serializes against the latest registered version rather than the object’s compile-time schema.
• Leave latest.compatibility.strict=true (the default) so the serializer still verifies the object is compatible with that latest version before writing; setting it false skips the check and serialization can fail downstream instead.

This makes the registry, fed by CI, the single place new versions appear—runtime producers only ever consume what was already reviewed.

Handling Multiple Subjects per Topic

Under the default TopicNameStrategy, a topic has two subjects: <topic>-key and <topic>-value. Your pipeline must check whichever subjects your data uses (most teams register only -value). If a schema imports references, also validate the referenced subject. Automate this by iterating over the schema files in your project and mapping each to its subject.

Maven and Gradle Plugins

The Confluent kafka-schema-registry-maven-plugin automates registration and compatibility checks in the build. The test-compatibility goal reads local schemas and checks them against a target registry; the configuration maps each subject name to a file path and, optionally, a schema type:

<plugin>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-schema-registry-maven-plugin</artifactId>
  <version>7.5.0</version>
  <configuration>
    <schemaRegistryUrls>
      <param>http://staging-schema-registry:8081</param>
    </schemaRegistryUrls>
    <subjects>
      <user-value>src/main/avro/user.avsc</user-value>
    </subjects>
    <schemaTypes>
      <user-value>AVRO</user-value>
    </schemaTypes>
  </configuration>
</plugin>

Run mvn io.confluent:kafka-schema-registry-maven-plugin:test-compatibility (or mvn schema-registry:test-compatibility once the plugin is bound) to fail the build on incompatible changes. The plugin checks against the registry’s configured compatibility level for each subject—there is no <compatibilityLevels> element on test-compatibility; set the level on the registry with the config endpoint or the plugin’s separate set-compatibility goal. The companion test-local-compatibility goal checks a schema against a previous local schema file without contacting a registry, and register pushes schemas in a deployment stage. This shifts validation left, catching issues before they reach a shared environment.

Operational Recovery: When Compatibility Violations Occur

Despite best efforts, violations will slip through—a hotfix bypasses the pipeline, a registry is misconfigured to NONE, or a manual registration introduces a breaking change. The response must be swift and structured.

Detecting the Violation

The first symptom is usually a spike in deserialization errors on consumers. In clients (including Kafka Streams) this surfaces as org.apache.kafka.common.errors.SerializationException; in Connect, sink tasks throw org.apache.kafka.connect.errors.DataException and may fail. Watch these JMX signals:

• Consumer lag/lead: kafka.consumer:type=consumer-fetch-manager-metrics,client-id=* attribute records-lead-min approaching zero (or records-lag-max climbing) as consumers stall.
• Connect errors: kafka.connect:type=task-error-metrics,connector=*,task=* attributes total-record-errors and total-record-failures increasing; if a dead letter queue is configured, deadletterqueue-produce-requests rising.
• Schema Registry: rising request-error-rate on the jersey-metrics per-endpoint counters (below) and Incompatible schema messages in the registry logs.

Immediate Mitigation

Do not hard-delete the offending schema version. Deletion has consequences—a soft delete hides the version but keeps the ID, and a hard delete (?permanent=true) is irreversible and frees the version number, which can confuse consumers still holding that schema ID. The cleaner fix is to register a new, compatible version that restores the structure consumers expect. For example, if someone added a required field without a default, register a version that makes the field optional:

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"email\",\"type\":[\"null\",\"string\"],\"default\":null}]}"}' \
  http://schema-registry:8081/subjects/user-value/versions

If consumers have already poisoned their offsets on records they cannot deserialize, you may need to reset the group to a point before the first bad record. Use kafka-consumer-groups with an explicit UTC timestamp (the consumer group must be stopped first):

kafka-consumer-groups --bootstrap-server kafka:9092 \
  --group my-consumer-group \
  --topic user \
  --reset-offsets --to-datetime 2024-01-15T10:00:00.000Z \
  --execute

The --to-datetime value is ISO 8601; appending Z pins it to UTC rather than the broker host’s local time zone. Resetting drops records produced after the timestamp from the group’s view, so coordinate with stakeholders—reprocessing or data loss is on the table.

Root Cause Analysis and Prevention

After restoring service, investigate how the change bypassed your gates. Common causes:

• Direct registration via the REST API by a developer with write access.
• A producer left on auto.register.schemas=true, registering an evolved schema at runtime outside CI.
• A pipeline that skipped or short-circuited the compatibility check.
• A shared/referenced schema changed without testing the importing subjects.

Strengthen the pipeline by restricting write access and routing every schema change through version control and CI. Confluent Platform supports RBAC and ACLs on Schema Registry; grant the registry write role only to your CI service accounts, and keep humans on read.

Multi-Environment Schema Management

Managing schemas across development, staging, and production registries requires a disciplined promotion strategy. The goal is that a schema version tested in staging is byte-identical to the one deployed to production—ideally down to the schema ID.

Schema ID Consistency Requires IMPORT Mode

Schema Registry assigns a globally unique ID to each registered schema, and that ID is embedded in every serialized record so consumers can fetch the writer’s schema. If you register the “same” schema independently in two registries, each assigns its own IDs, and records produced in one environment cannot be resolved against the other.

A plain POST .../versions does not let you choose the ID—the registry allocates it. To preserve IDs and versions across registries, the target subject must be put into IMPORT mode and the registration payload must include the explicit id and version. IMPORT mode can only be set on an empty subject (attempting it on a populated subject returns 422), and it bypasses compatibility checks during import:

# 1) Export schema, id, and version from the source registry
curl -s http://staging-schema-registry:8081/subjects/user-value/versions/latest \
  > user-value.json   # contains {"subject","version","id","schema"}

# 2) Put the (empty) target subject into IMPORT mode
curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"mode": "IMPORT"}' \
  http://prod-schema-registry:8081/mode/user-value

# 3) Register with the preserved id and version
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data @user-value.json \
  http://prod-schema-registry:8081/subjects/user-value/versions

For anything beyond a one-off, use a purpose-built replication tool (such as Confluent’s Schema Linking or schema-exporter) rather than scripting IMPORT mode by hand—getting the ordering of versions and references right across many subjects is error-prone.

Environment-Specific Compatibility Levels

A common pattern is NONE in development, BACKWARD in staging, and BACKWARD or stricter in production. This allows prototyping while enforcing compatibility where it matters. Because a schema that registers freely under NONE may fail under BACKWARD, always run a compatibility check against the target registry (the /compatibility/... endpoint) before promoting—don’t assume success in a looser environment carries over.

Schema Versioning and GitOps

Treat schemas as code. Store them in a Git repository alongside the applications that produce and consume them, in a directory structure that mirrors your subjects:

schemas/
  user-value/
    v1.avsc
    v2.avsc
  order-value/
    v1.proto

Your CI pipeline should validate each proposed schema against both the latest version in the repository and the target registry, failing if either check fails. This keeps Git as the source of truth and the registry in sync with it.

Monitoring Schema Registry Health and Performance

Schema Registry is critical infrastructure: if it is unavailable, producers and consumers that rely on schema-based serialization stall on cache misses. Monitoring must cover both the registry’s own health and its impact on Kafka clients.

JMX Metrics

Schema Registry exposes metrics through JMX under the kafka.schema.registry domain. Browse them with JConsole, and set SCHEMA_REGISTRY_JMX_OPTS/SCHEMA_REGISTRY_JMX_PORT to expose the port. The metrics most worth alerting on:

• kafka.schema.registry:type=jersey-metrics — per-endpoint request metrics. Attributes are prefixed by the endpoint name (e.g. register.request-latency-avg, register.request-error-rate, subjects.get-schema.request-error-rate). A rising request-error-rate on the register/compatibility endpoints is your earliest signal that teams are pushing breaking changes; a climbing request-latency-avg warns of producer timeouts.
• kafka.schema.registry:type=jetty-metrics — attribute connections-active, the count of active TCP connections, useful for capacity planning.
• kafka.schema.registry:type=master-slave-role — indicates whether this node is the leader (the node that performs writes to the _schemas topic). Watch for unexpected leadership changes.

These are the documented MBeans; consult the Schema Registry monitoring reference for the full attribute list for your version. (There is no documented MBean named schema.registry.registered.count or schema.registry.compatibility.checks.failed.count—use the per-endpoint jersey-metrics and the registry logs instead.)

Health Check and Readiness Probe

In Kubernetes, point a readiness probe at the registry’s root endpoint:

readinessProbe:
  httpGet:
    path: /
    port: 8081
  initialDelaySeconds: 30
  periodSeconds: 10

A 200 means the HTTP layer is up, but it does not prove the _schemas topic is being consumed. For a deeper liveness signal, hit the subjects endpoint, which forces the registry to serve from its in-memory store:

curl -f http://schema-registry:8081/subjects || exit 1

Client-Side Caching

Clients cache schemas locally to avoid a registry round-trip on every record. The serializers/deserializers (subclasses of AbstractKafkaSchemaSerDeConfig) bound the per-subject identity map with max.schemas.per.subject (default 1000). If your application legitimately uses many schema versions per subject and you exceed this, the client throws IllegalStateException: Too many schema objects created for <subject>—usually a sign that schemas are being rebuilt per record rather than reused, but raise the limit if the usage is genuine. Caches that serve the latest schema lookups are bounded separately by latest.cache.size and latest.cache.ttl. Reuse a single CachedSchemaRegistryClient per application so these caches actually hit.

Backup and Disaster Recovery

The _schemas topic is the registry’s source of truth; back it up with your standard Kafka topic backup or replication procedures. As a secondary, human-readable backup, periodically export every subject and version to version control:

#!/usr/bin/env bash
set -euo pipefail
BASE=http://schema-registry:8081
mkdir -p backup
for subject in $(curl -s "$BASE/subjects" | jq -r '.[]'); do
  for version in $(curl -s "$BASE/subjects/$subject/versions" | jq -r '.[]'); do
    curl -s "$BASE/subjects/$subject/versions/$version/schema" \
      > "backup/${subject}-v${version}.avsc"
  done
done

In a rebuild, restore IDs and versions with IMPORT mode (above) rather than re-registering blindly—otherwise consumers holding old IDs cannot resolve them.

Conclusion

Compatibility modes are contracts, not flags: they dictate deployment ordering and data-flow direction across your event-driven architecture. The leverage points are concrete—pick the mode from the deployment facts in the table above, pin producers off auto.register.schemas so only CI introduces versions, gate the /compatibility endpoint in every pipeline, and keep a tested recovery path (compatible re-registration, then offset reset) for the violation that eventually arrives.

These principles apply across the Schema Registry & Connect Ecosystem—streaming ETL with Kafka Connect, change data capture, or stateful stream processors. As your schema estate grows, revisit your strategy: what works for ten subjects rarely scales to a thousand without per-subject overrides and transitive modes.

For authoritative references, see the Apache Avro specification, the Protocol Buffers language guide, and Confluent’s documentation on schema evolution and compatibility and the Schema Registry REST API.