You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Steven Aerts (JIRA)" <ji...@apache.org> on 2018/07/03 06:07:00 UTC
[jira] [Commented] (KAFKA-7026) Sticky assignor could assign a
partition to multiple consumers
[ https://issues.apache.org/jira/browse/KAFKA-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530876#comment-16530876 ]
Steven Aerts commented on KAFKA-7026:
-------------------------------------
Just fyi,
An extra example of a double assignment with a {{StickyAssignor}}:
{code}
CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT
consumer-1-fc091420 host41 consumer-1 20 topic(2,13,34,40,49,63,85,91,146,147,151,161,162,179,194,217,257,269,330,334)
consumer-2-e48a0ca6 host144 consumer-2 19 topic(15,17,36,57,66,69,87,111,127,168,178,216,235,275,282,292,300,313,333)
consumer-2-e2db3969 host247 consumer-2 19 topic(20,33,43,50,77,82,107,126,130,135,139,173,188,209,224,242,265,291,307)
consumer-1-fc3ab033 host144 consumer-1 20 topic(6,30,31,48,59,62,100,102,103,115,140,192,199,201,213,231,245,309,321,345)
consumer-2-8e6c6445 host81 consumer-2 20 topic(0,11,32,44,64,67,71,101,119,123,132,143,183,205,244,249,267,318,341,347)
consumer-2-a2093927 host255 consumer-2 20 topic(10,16,42,74,90,106,124,153,157,220,240,256,258,259,289,305,306,328,349,358)
consumer-1-87baca65 host72 consumer-1 20 topic(28,37,38,53,56,97,113,121,128,137,145,160,189,218,268,276,277,302,343,350)
consumer-2-17a88623 host26 consumer-2 20 topic(1,18,19,41,46,61,83,95,99,110,117,133,186,193,197,214,234,255,264,348)
consumer-1-e0386920 host81 consumer-1 20 topic(22,24,79,94,122,144,169,172,177,187,198,215,228,251,285,294,295,304,310,338)
consumer-1-1833b87f host255 consumer-1 20 topic(3,8,9,29,58,73,98,105,109,142,158,190,191,202,210,273,312,319,323,335)
consumer-2-48439f14 host72 consumer-2 20 topic(14,45,80,84,89,104,114,150,185,200,207,212,239,248,254,290,296,324,325,351)
consumer-2-2dd9999f host41 consumer-2 20 topic(47,65,118,149,184,196,223,225,227,238,241,250,253,263,279,283,298,301,332,355)
consumer-1-26ba25c1 host96 consumer-1 20 topic(5,23,39,51,70,78,125,129,175,182,219,221,230,247,252,270,286,316,317,352)
consumer-2-c595b608 host96 consumer-2 20 topic(7,21,52,55,72,134,154,155,156,163,167,203,226,233,236,266,287,303,322,339)
consumer-1-ad352cbc host164 consumer-1 20 topic(27,75,81,136,195,208,232,246,271,278,280,281,314,320,327,329,336,342,344,354)
consumer-2-4a8643ef host164 consumer-2 20 topic(68,93,131,138,148,159,164,165,170,176,204,222,229,260,288,299,315,331,346,356)
consumer-1-a3f65f96 host26 consumer-1 20 topic(4,12,35,86,108,112,116,120,181,211,237,243,261,262,272,297,311,337,353,359)
consumer-1-583ae694 host247 consumer-1 20 topic(25,54,60,76,88,92,96,141,152,171,174,180,206,274,284,293,308,326,340,357)
consumer-1-230f463c host216 consumer-1 19 topic(20,26,56,78,145,171,179,195,213,223,261,271,293,298,304,321,335,352,356)
consumer-2-4842910d host216 consumer-2 19 topic(40,52,69,87,99,101,135,143,150,166,207,210,226,228,255,259,299,337,349)
{code}
The double assignment is the last assignment (I reordered the list).
What is interesting that here, we have two consumers in the same process. And that the other consumer of that process did not loose its subscription.
> Sticky assignor could assign a partition to multiple consumers
> --------------------------------------------------------------
>
> Key: KAFKA-7026
> URL: https://issues.apache.org/jira/browse/KAFKA-7026
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Reporter: Vahid Hashemian
> Assignee: Vahid Hashemian
> Priority: Major
> Fix For: 2.1.0
>
>
> In the following scenario sticky assignor assigns a topic partition to two consumers in the group:
> # Create a topic {{test}} with a single partition
> # Start consumer {{c1}} in group {{sticky-group}} ({{c1}} becomes group leader and gets {{test-0}})
> # Start consumer {{c2}} in group {{sticky-group}} ({{c1}} holds onto {{test-0}}, {{c2}} does not get any partition)
> # Pause {{c1}} (e.g. using Java debugger) ({{c2}} becomes leader and takes over {{test-0}}, {{c1}} leaves the group)
> # Resume {{c1}}
> At this point both {{c1}} and {{c2}} will have {{test-0}} assigned to them.
>
> The reason is {{c1}} still has kept its previous assignment ({{test-0}}) from the last assignment it received from the leader (itself) and did not get the next round of assignments (when {{c2}} became leader) because it was paused. Both {{c1}} and {{c2}} enter the rebalance supplying {{test-0}} as their existing assignment. The sticky assignor code does not currently check and avoid this duplication.
>
> Note: This issue was originally reported on [StackOverflow|https://stackoverflow.com/questions/50761842/kafka-stickyassignor-breaking-delivery-to-single-consumer-in-the-group].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)