[jira] [Created] (KAFKA-3966) Consumer briefly ignores partitions on backlogs

["Kanak Biscuitwala (JIRA)" <ji...@apache.org>]

Kanak Biscuitwala created KAFKA-3966:
----------------------------------------

             Summary: Consumer briefly ignores partitions on backlogs
                 Key: KAFKA-3966
                 URL: https://issues.apache.org/jira/browse/KAFKA-3966
             Project: Kafka
          Issue Type: Bug
            Reporter: Kanak Biscuitwala
         Attachments: screenshot-1.png

Setup:
Kafka 0.10.0.0
9 brokers
36 partitions
12 members in the consumer group
5 producers, partitioning data roughly evenly
max.poll.records = 10
~1500 events/sec
~500 bytes/message
KafkaConsumer API

In the following setup, 3 partitions are assigned to each consumer (and 12 are served by each broker). I've noticed that the third of the three partitions tends to be ignored in favor of the first two on each consumer, even though the third partition has data available. Only until the first two partitions are fully caught up does the consumer return back messages from the third. This causes a frustrating imbalance in which the same partitions always fall behind.

As a side note, this is true for all of our topics, regardless of partitioning strategy. The problem goes away if there are exactly as many consumers as partitions.

I can attach a screenshot showing the same partitions falling behind (verified that they're each assigned to different nodes), if that is helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)