[jira] [Commented] (KAFKA-17950) The leader requested truncation to below the current high watermark

["Viktor Somogyi-Vass (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-17950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17895963#comment-17895963 ] 

Viktor Somogyi-Vass commented on KAFKA-17950:
---------------------------------------------

It seems like that in the configs I use {{inter.broker.protocol.version}} and {{metadata.version}} too but they are either 3.9 or 21, so it doesn't really affect the repro.

> The leader requested truncation to below the current high watermark
> -------------------------------------------------------------------
>
>                 Key: KAFKA-17950
>                 URL: https://issues.apache.org/jira/browse/KAFKA-17950
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 3.9.0, 3.9.1
>            Reporter: Viktor Somogyi-Vass
>            Priority: Blocker
>         Attachments: broker1.log, broker2.log, broker3.log, controller-logs.zip, controller1-migration-enabled.properties, controller1.properties, controller2-migration-enabled.properties, controller2.properties, controller3-migration-enabled.properties, controller3.properties, kraft1.log, kraft2.log, kraft3.log, producer-perf.log, producer.properties, server1-migrated-to-kraft.properties, server1-migration-enabled.properties, server1.properties, server2-migrated-to-kraft.properties, server2-migration-enabled.properties, server2.properties, server3-migrated-to-kraft.properties, server3-migration-enabled.properties, server3.properties, zookeeper.log
>
>
> While testing the migration from 3.9 ZK Kafka to 3.9 KRaft, I find that in the last step (finalization) where I restart the controllers in non-migration mode, the last controller restart causes a fatal failure in the cluster: every node (broker and controller) stops beside the controller I restarted.
> The failing nodes throw the same exception at the time:
> {noformat}
> [2024-11-06 14:02:13,498] ERROR Encountered fatal fault: Unexpected error in raft IO thread (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler)
> org.apache.kafka.common.KafkaException: The leader requested truncation to offset 484, which is below the current high watermark LogOffsetMetadata(offset=508, metadata=Optional.empty)
>         at org.apache.kafka.raft.KafkaRaftClient.lambda$handleFetchResponse$11(KafkaRaftClient.java:1619)
>         at java.base/java.util.Optional.ifPresent(Optional.java:183)
>         at org.apache.kafka.raft.KafkaRaftClient.handleFetchResponse(KafkaRaftClient.java:1616)
>         at org.apache.kafka.raft.KafkaRaftClient.handleResponse(KafkaRaftClient.java:2457)
>         at org.apache.kafka.raft.KafkaRaftClient.handleInboundMessage(KafkaRaftClient.java:2613)
>         at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:3312)
>         at org.apache.kafka.raft.KafkaRaftClientDriver.doWork(KafkaRaftClientDriver.java:64)
>         at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136)
> {noformat}
> Setup:
> * single Zookeeper node
> * 3 brokers
> * 1 running producer-performance client
> * 3 controllers
> Repro:
> # Start Zookeeper with zookeeper.properties
> {noformat}
> bin/zookeeper-server-start.sh repro-conf/zookeeper.properties
> {noformat}
> # Start brokers with serverX.properties
> {noformat}
> bin/kafka-server-start.sh repro-conf/server1.properties
> bin/kafka-server-start.sh repro-conf/server2.properties
> bin/kafka-server-start.sh repro-conf/server3.properties
> {noformat}
> # Start the producer-performance tool
> {noformat}
> bin/kafka-producer-perf-test.sh --topic test1 --num-records 1000000 --throughput 100 --record-size 10000 --producer.config repro-conf/producer.properties
> {noformat}
> # Get the cluster ID and format all controller log dirs
> # Start the controllers in migration mode
> {noformat}
> bin/kafka-server-start.sh repro-conf/controller1-migration-enabled.properties
> bin/kafka-server-start.sh repro-conf/controller2-migration-enabled.properties
> bin/kafka-server-start.sh repro-conf/controller3-migration-enabled.properties
> {noformat}
> # Restart the brokers in migration mode with the following configs. (My restart order was 1,2,3.)
> {noformat}
> bin/kafka-server-start.sh repro-conf/server1-migration-enabled.properties
> bin/kafka-server-start.sh repro-conf/server2-migration-enabled.properties
> bin/kafka-server-start.sh repro-conf/server3-migration-enabled.properties
> {noformat}
> # Restart the brokers in migrated mode with the following configs (at this point they are connected to the controllers and not ZK). My restart order was 1,2,3.
> {noformat}
> bin/kafka-server-start.sh repro-conf/server1-migrated-to-kraft.properties
> bin/kafka-server-start.sh repro-conf/server2-migrated-to-kraft.properties
> bin/kafka-server-start.sh repro-conf/server3-migrated-to-kraft.properties
> {noformat}
> # At this point all brokers run with KRaft, let's restart the controllers to finalize. (The order was 3,2,1.)
> {noformat}
> bin/kafka-server-start.sh repro-conf/controller3.properties
> bin/kafka-server-start.sh repro-conf/controller2.properties
> bin/kafka-server-start.sh repro-conf/controller1.properties
> {noformat}
> At the last restart, when controller1 starts up, all other nodes crash at once. Attached all logs and configuration.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)