You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "redbrick9 (Jira)" <ji...@apache.org> on 2019/09/19 01:20:00 UTC
[jira] [Commented] (KAFKA-7026) Sticky assignor could assign a
partition to multiple consumers (KIP-341)
[ https://issues.apache.org/jira/browse/KAFKA-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932965#comment-16932965 ]
redbrick9 commented on KAFKA-7026:
----------------------------------
We are testing our application based on kafka, when we are using iptables to drop/reject packets from consumers to broker for the network issues' similation. We ran into the similiar problem.
A few partitions were assigned to the more than one consumers.
We are using kafka 2.3.0 and StickyAssignor assignment strategy. It looks good to us to use another assignment strategy.
It is easy to reproduce. Just adding rules to reject/drop packets from consumer then delete the rules. We can see the following output.
{code:java}
// code placeholder
{code}
bin/kafka-consumer-groups.sh --bootstrap-server 172.93.57.21:9092 --group test-group --describe GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID test-group test-topic 12 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1 test-group test-topic 2 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1 test-group test-topic 5 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1 test-group test-topic 15 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1 test-group test-topic 3 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1 test-group test-topic 8 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1 test-group test-topic 7 - 0 - consumer-1-f94c41a7-a64c-4f22-bfa2-86b369f22682 /172.93.57.22 consumer-1 test-group test-topic 5 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1 test-group test-topic 4 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1 test-group test-topic 11 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1 test-group test-topic 0 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1 test-group test-topic 3 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1 test-group test-topic 10 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1 test-group test-topic 6 - 0 - consumer-1-75e1a353-cd34-4e49-b88f-2e8da3117320 /172.93.57.23 consumer-1 test-group test-topic 12 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1 test-group test-topic 2 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1 test-group test-topic 14 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1 test-group test-topic 1 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1 test-group test-topic 13 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1 test-group test-topic 7 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1 test-group test-topic 9 - 0 - consumer-1-a5684c45-859c-4718-aa27-32fcf8f9c64b /172.93.57.21 consumer-1
> Sticky assignor could assign a partition to multiple consumers (KIP-341)
> ------------------------------------------------------------------------
>
> Key: KAFKA-7026
> URL: https://issues.apache.org/jira/browse/KAFKA-7026
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Reporter: Vahid Hashemian
> Assignee: Vahid Hashemian
> Priority: Major
> Labels: kip
> Fix For: 2.3.0
>
>
> In the following scenario sticky assignor assigns a topic partition to two consumers in the group:
> # Create a topic {{test}} with a single partition
> # Start consumer {{c1}} in group {{sticky-group}} ({{c1}} becomes group leader and gets {{test-0}})
> # Start consumer {{c2}} in group {{sticky-group}} ({{c1}} holds onto {{test-0}}, {{c2}} does not get any partition)
> # Pause {{c1}} (e.g. using Java debugger) ({{c2}} becomes leader and takes over {{test-0}}, {{c1}} leaves the group)
> # Resume {{c1}}
> At this point both {{c1}} and {{c2}} will have {{test-0}} assigned to them.
>
> The reason is {{c1}} still has kept its previous assignment ({{test-0}}) from the last assignment it received from the leader (itself) and did not get the next round of assignments (when {{c2}} became leader) because it was paused. Both {{c1}} and {{c2}} enter the rebalance supplying {{test-0}} as their existing assignment. The sticky assignor code does not currently check and avoid this duplication.
>
> Note: This issue was originally reported on [StackOverflow|https://stackoverflow.com/questions/50761842/kafka-stickyassignor-breaking-delivery-to-single-consumer-in-the-group].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)