[jira] [Created] (KAFKA-20762) Node replacements are not supported during KRaft migration

["Anton Agestam (Jira)" <ji...@apache.org>]

Anton Agestam created KAFKA-20762:
-------------------------------------

             Summary: Node replacements are not supported during KRaft migration
                 Key: KAFKA-20762
                 URL: https://issues.apache.org/jira/browse/KAFKA-20762
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 3.9.2
            Reporter: Anton Agestam


Hi, 

We are Aiven, a Kafka-as-a-service provider. We run a large fleet of managed Kafka services.

In a recent production incident where we had to use KRaft migration backup tooling, we discovered that node replacements are not supported during the migration sequence. As far as we have been able to find, this incompatibility is not stated anywhere in documentation, and is causing a regression in ability to recover clusters to a healthy state.

The problematic sequence is as follows:
 # Cluster is put into KRaft migration, migrates all the way to [dual-write phase|https://kafka.apache.org/37/operations/kraft/#migration-phases]. We linger here to allow discovery of any issues that may occur.
 # Loss of a single service node occurs, automation provisions a node replacement.
 # During provisioning of the replacement, we issue `kafka-storage.sh format` as per [the documentation.|https://kafka.apache.org/41/operations/kraft/#provisioning-nodes]
 # The new node joins the cluster and starts serving workloads.
 # Later, we discover an issue and an operator decides to trigger rollback of the migration. When configuring brokers back into migration mode, the newly provisioned broker goes into a crash loop with the below error log.

 
{code:java}
Jul 02 14:56:32 kafka[25156]: [2026-07-02 14:56:32,647] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
                              java.lang.RuntimeException: Found unexpected version in /srv/kafka/meta.properties. ZK-based brokers that are not migrating only support version 0 (which is implicit when the `version` field is missing).
                                      at org.apache.kafka.metadata.properties.MetaPropertiesEnsemble.verify(MetaPropertiesEnsemble.java:489)
                                      at kafka.server.KafkaServer.startup(KafkaServer.scala:258)
                                      at kafka.Kafka$.main(Kafka.scala:112)
                                      at kafka.Kafka.main(Kafka.scala)
Jul 02 14:56:32 kafka[25156]: [2026-07-02 14:56:32,650] INFO shutting down (kafka.server.KafkaServer) {code}
---

 

The issue happens because kafka-storage.sh format always outputs meta.properties with version=1. Once the rollback sequence is entered, this is rejected. The provisioning tooling does not provide a way to format the local storage in a way such that the broker spun up on top of it will support rolling back an ongoing migration. And documentation does not mention this incompatibility.

As these things are documented as orthogonal concepts, where the meta.properties version is an opaque implementation detail, we think it's fair to expect that there should be compatibility here. We'd like to propose as solution to provide a flag for the format tool that will make it create a meta.properties with version=0, and that supports rolling back. The issue could also be solved by deciding that this incompatibility is as designed, and documenting the status quo, but we think that would be unfortunate as it imposes increased risk and cumbersome operations during an already complex operational mode.

As there is already support for formatting storage in a way that supports rollback (the broker does this itself during the migration), it seems it should not be hard to implement the fix in the format command.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)