[jira] [Commented] (KAFKA-12345) KIP-500: AlterIsrManager crashes on broker idle-state

["Alok Nikhil (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286810#comment-17286810 ] 

Alok Nikhil commented on KAFKA-12345:
-------------------------------------

Adding a bit more context. Seems like there was a controller quorum voting event just before this crash. Seems to be related


{code:java}
[2021-02-19 01:04:20,721] INFO [Controller 3000] Fenced broker: FenceBrokerRecord(id=3, epoch=2) (org.apache.kafka.controller.ClusterControlManager)
[2021-02-19 01:04:21,354] INFO [BrokerMetadataListener id=0] Tell Replica Manager to handle records (kafka.server.metadata.BrokerMetadataListener)
[2021-02-19 01:04:22,097] INFO [RaftManager broker=0,controller=3000] Completed transition to Unattached(epoch=2, voters=[3000, 3001, 3002, 3003, 3004], electionTimeoutMs=690) (org.apache.kafka.raft.QuorumState)
[2021-02-19 01:04:22,203] INFO [RaftManager broker=0,controller=3000] Completed transition to Unattached(epoch=3, voters=[3000, 3001, 3002, 3003, 3004], electionTimeoutMs=584) (org.apache.kafka.raft.QuorumState)
[2021-02-19 01:04:22,507] INFO [RaftManager broker=0,controller=3000] Completed transition to Unattached(epoch=4, voters=[3000, 3001, 3002, 3003, 3004], electionTimeoutMs=280) (org.apache.kafka.raft.QuorumState)
[2021-02-19 01:04:22,626] INFO [RaftManager broker=0,controller=3000] Completed transition to Unattached(epoch=5, voters=[3000, 3001, 3002, 3003, 3004], electionTimeoutMs=161) (org.apache.kafka.raft.QuorumState)
[2021-02-19 01:04:22,626] INFO [RaftManager broker=0,controller=3000] Completed transition to Voted(epoch=5, votedId=3004, voters=[3000, 3001, 3002, 3003, 3004], electionTimeoutMs=520) (org.apache.kafka.raft.QuorumState)
[2021-02-19 01:04:22,631] INFO [RaftManager broker=0,controller=3000] Completed transition to FollowerState(fetchTimeoutMs=2000, epoch=5, leaderId=3004, voters=[3000, 3001, 3002, 3003, 3004]) (org.apache.kafka.raft.QuorumState){code}

> KIP-500: AlterIsrManager crashes on broker idle-state
> -----------------------------------------------------
>
>                 Key: KAFKA-12345
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12345
>             Project: Kafka
>          Issue Type: Task
>          Components: core
>            Reporter: Alok Nikhil
>            Priority: Major
>              Labels: kip-500
>
> Occasionally, a scheduler thread on a broker crashes with this stack
>  
> {code:java}
> [2021-02-19 01:04:24,683] ERROR Uncaught exception in scheduled task 'send-alter-isr' (kafka.utils.KafkaScheduler)
>  java.lang.NullPointerException
>  at kafka.server.AlterIsrManagerImpl.sendRequest(AlterIsrManager.scala:117)
>  at kafka.server.AlterIsrManagerImpl.propagateIsrChanges(AlterIsrManager.scala:85)
>  at kafka.server.AlterIsrManagerImpl.$anonfun$start$1(AlterIsrManager.scala:66)
>  at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
>  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>  at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>  at java.base/java.lang.Thread.run(Thread.java:834){code}
>  
> After that the broker is unable to fetch any records from any other broker (and vice versa)
> {code:java}
> [2021-02-19 01:05:07,000] INFO [ReplicaFetcher replicaId=0, leaderId=4, fetcherId=0] Error sending fetch request (sessionId=164432409
>  2, epoch=957) to node 4: (org.apache.kafka.clients.FetchSessionHandler)
>  java.io.IOException: Connection to 4 was disconnected before the response was read
>  at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:100)
>  at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:110)
>  at kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:215)
>  at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:313)
>  at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:139)
>  at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:138)
>  at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:121)
>  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96){code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)