[jira] [Created] (KAFKA-16814) KRaft broker cannot startup when `partition.metadata` is missing

["Luke Chen (Jira)" <ji...@apache.org>]

Luke Chen created KAFKA-16814:
---------------------------------

             Summary: KRaft broker cannot startup when `partition.metadata` is missing
                 Key: KAFKA-16814
                 URL: https://issues.apache.org/jira/browse/KAFKA-16814
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 3.7.0
            Reporter: Luke Chen


When starting up kafka logManager, we'll check stray replicas to avoid some corner cases. But this check might cause broker unable to startup if `partition.metadata` is missing because when startup kafka, we load log from file, and the topicId of the log is coming from `partition.metadata` file. So, if `partition.metadata` is missing, the topicId will be None, and the `LogManager#isStrayKraftReplica` will fail with no topicID error.

The `partition.metadata` missing could be some storage failure, or another possible path is unclean shutdown after topic is created in the replica, but before data is flushed into `partition.metadata` file. This is possible because we do the flush, it's done async [here|https://github.com/apache/kafka/blob/5552f5c26df4eb07b2d6ee218e4a29e4ca790d5c/core/src/main/scala/kafka/log/UnifiedLog.scala#L229].

 

 
{code:java}
ERROR Encountered fatal fault: Error starting LogManager (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler)
java.lang.RuntimeException: The log dir Log(dir=/tmp/kraft-broker-logs/quickstart-events-0, topic=quickstart-events, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) does not have a topic ID, which is not allowed when running in KRaft mode.
    at kafka.log.LogManager$.$anonfun$isStrayKraftReplica$1(LogManager.scala:1609)
    at scala.Option.getOrElse(Option.scala:201)
    at kafka.log.LogManager$.isStrayKraftReplica(LogManager.scala:1608)
    at kafka.server.metadata.BrokerMetadataPublisher.$anonfun$initializeManagers$1(BrokerMetadataPublisher.scala:294)
    at kafka.server.metadata.BrokerMetadataPublisher.$anonfun$initializeManagers$1$adapted(BrokerMetadataPublisher.scala:294)
    at kafka.log.LogManager.loadLog(LogManager.scala:359)
    at kafka.log.LogManager.$anonfun$loadLogs$15(LogManager.scala:493)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1623) {code}
 

Because if we don't do the isStrayKraftReplica check, the topicID and the `partition.metadata` will get recovered after getting topic partition update and becoming leader or follower later. I'm proposing we skip the `isStrayKraftReplica` check if topicID is None, instead of throwing exception to terminate the kafka. `isStrayKraftReplica` check is just for a corner case only, it should be fine IMO.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)