[jira] [Updated] (KAFKA-3966) KafkaConsumer briefly ignores partitions on backlogs

["Kanak Biscuitwala (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kanak Biscuitwala updated KAFKA-3966:
-------------------------------------
    Summary: KafkaConsumer briefly ignores partitions on backlogs  (was: Consumer briefly ignores partitions on backlogs)

> KafkaConsumer briefly ignores partitions on backlogs
> ----------------------------------------------------
>
>                 Key: KAFKA-3966
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3966
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.10.0.0
>            Reporter: Kanak Biscuitwala
>         Attachments: screenshot-1.png
>
>
> Setup:
> Kafka 0.10.0.0
> 9 brokers
> 36 partitions
> 12 members in the consumer group
> 5 producers, partitioning data roughly evenly
> max.poll.records = 10
> ~1500 events/sec
> ~500 bytes/message
> KafkaConsumer API
> In the following setup, 3 partitions are assigned to each consumer (and 12 are served by each broker). I've noticed that the third of the three partitions tends to be ignored in favor of the first two on each consumer, even though the third partition has data available. Only until the first two partitions are fully caught up does the consumer return back messages from the third. This causes a frustrating imbalance in which the same partitions always fall behind.
> As a side note, this is true for all of our topics, regardless of partitioning strategy. The problem goes away if there are exactly as many consumers as partitions.
> I can attach a screenshot showing the same partitions falling behind (verified that they're each assigned to different nodes), if that is helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)