[jira] [Created] (KAFKA-19523) Gracefully handle error while building remoteLogAuxState

["Kamal Chandraprakash (Jira)" <ji...@apache.org>]

Kamal Chandraprakash created KAFKA-19523:
--------------------------------------------

             Summary: Gracefully handle error while building remoteLogAuxState
                 Key: KAFKA-19523
                 URL: https://issues.apache.org/jira/browse/KAFKA-19523
             Project: Kafka
          Issue Type: Task
            Reporter: Kamal Chandraprakash
            Assignee: Kamal Chandraprakash


Improve the error handling while building the remote-log-auxiliary state when a follower node with an empty disk begin to synchronise with the leader. If the topic has remote storage enabled, then the ReplicaFetcherThread attempt to build the remote-log-auxiliary state. Note that the remote-log-auxiliary state gets invoked only when the leader-log-start-offset is non-zero and leader-log-start-offset is not equal to leader-local-log-start-offset.

When the LeaderAndISR request is received, then the ReplicaManager#becomeLeaderOrFollower invokes 'makeFollowers' initially, followed by the RemoteLogManager#onLeadershipChange call. As a result, when ReplicaFetcherThread initiates the RemoteLogManager#fetchRemoteLogSegmentMetadata, the partition may not have been initialized at that time.

Introducing a new RetriableRemoteStorageException requires a KIP as it is a public API change, so wrap the IllegalStateException in RemoteStorageException to gracefully handle the error.

stacktrace: 
{code}
[2025-07-19 19:15:47,915] ERROR [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Error building remote log auxiliary state for orange-0 (kafka.server.ReplicaFetcherThread)
java.lang.IllegalStateException: This instance is in invalid state, initialized: false close: false
        at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.ensureInitializedAndNotClosed(TopicBasedRemoteLogMetadataManager.java:569) ~[kafka-storage-4.2.0-SNAPSHOT.jar:?]
        at org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager.remoteLogSegmentMetadata(TopicBasedRemoteLogMetadataManager.java:221) ~[kafka-storage-4.2.0-SNAPSHOT.jar:?]
        at org.apache.kafka.server.log.remote.storage.RemoteLogManager.fetchRemoteLogSegmentMetadata(RemoteLogManager.java:606) ~[kafka-storage-4.2.0-SNAPSHOT.jar:?]
        at kafka.server.TierStateMachine.buildRemoteLogAuxState(TierStateMachine.java:233) ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
        at kafka.server.TierStateMachine.start(TierStateMachine.java:114) ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
        at kafka.server.AbstractFetcherThread.handleOffsetsMovedToTieredStorage(AbstractFetcherThread.scala:785) ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
        at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:434) ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
        at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.16.jar:?]
        at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6(AbstractFetcherThread.scala:342) ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
        at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted(AbstractFetcherThread.scala:341) ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
        at scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry(JavaCollectionWrappers.scala:430) ~[scala-library-2.13.16.jar:?]
        at scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry$(JavaCollectionWrappers.scala:426) ~[scala-library-2.13.16.jar:?]
        at scala.collection.convert.JavaCollectionWrappers$AbstractJMapWrapper.foreachEntry(JavaCollectionWrappers.scala:344) ~[scala-library-2.13.16.jar:?]
        at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:341) ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
        at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:137) ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
        at java.base/java.util.Optional.ifPresent(Optional.java:178) [?:?]
        at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:136) [kafka_2.13-4.2.0-SNAPSHOT.jar:?]
        at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:117) [kafka_2.13-4.2.0-SNAPSHOT.jar:?]
        at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:96) [kafka_2.13-4.2.0-SNAPSHOT.jar:?]
        at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136) [kafka-server-common-4.2.0-SNAPSHOT.jar:?]
{code}

The exception gets thrown repeatedly until the RemoteLogMetadataManager#isReady(topicIdPartition) becomes true. This is a retriable error so we have to handle it gracefully. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)