[jira] [Created] (KAFKA-6178) Broker is listed as only ISR for all partitions it is leader of

["AS (JIRA)" <ji...@apache.org>]

AS created KAFKA-6178:
-------------------------

             Summary: Broker is listed as only ISR for all partitions it is leader of
                 Key: KAFKA-6178
                 URL: https://issues.apache.org/jira/browse/KAFKA-6178
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 0.10.1.0
         Environment: Windows
            Reporter: AS
         Attachments: KafkaServiceOutput.txt, log-cleaner.log, server.log

We're running a 15 broker cluster on windows machines, and one of the brokers, 10, is the only ISR on all partitions that it is the leader of. On partitions where it isn't the leader, it seems to follow the leadeer fine. This is an excerpt from 'describe':

{{Topic: ClientQosCombined      Partition: 458  Leader: 10      Replicas: 10,6,7,8,9,0,1   Isr: 10
Topic: ClientQosCombined      Partition: 459  Leader: 11      Replicas: 11,7,8,9,0,1,10 Isr: 0,10,1,9,7,11,8}}

The server.log files all seem to be pretty standard, and the only indication of this issue is the following pattern that often repeats:

{{2017-11-06 20:28:25,207 [INFO] kafka.cluster.Partition [kafka-request-handler-8:] - Partition [ClientQosCombined,398] on broker 10: Expanding ISR for partition [ClientQosCombined,398] from 10 to 5,10
2017-11-06 20:28:39,382 [INFO] kafka.cluster.Partition [kafka-scheduler-1:] - Partition [ClientQosCombined,398] on broker 10: Shrinking ISR for partition [ClientQosCombined,398] from 5,10 to 10}}

For each of the partitions that 10 leads. This is the only topic that we currently have in our cluster. The __consumer_offsets topic seems completely normal in terms of isr counts. The controller is broker 5, which is cycling through attempting and failing to trigger leader elections on broker 10 led partitions. From the controller log in broker 5:

{{2017-11-06 20:45:04,857 [INFO] kafka.controller.KafkaController [kafka-scheduler-0:] - [Controller 5]: Starting preferred replica leader election for partitions [ClientQosCombined,375]
2017-11-06 20:45:04,857 [INFO] kafka.controller.PartitionStateMachine [kafka-scheduler-0:] - [Partition state machine on Controller 5]: Invoking state change to OnlinePartition for partitions [ClientQosCombined,375]
2017-11-06 20:45:04,857 [INFO] kafka.controller.PreferredReplicaPartitionLeaderSelector [kafka-scheduler-0:] - [PreferredReplicaPartitionLeaderSelector]: Current leader 10 for partition [ClientQosCombined,375] is not the preferred replica. Trigerring preferred replica leader election
2017-11-06 20:45:04,857 [WARN] kafka.controller.KafkaController [kafka-scheduler-0:] - [Controller 5]: Partition [ClientQosCombined,375] failed to complete preferred replica leader election. Leader is 10}}

I've also attached the logs and output from broker 10. Any idea what's wrong here? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)