You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Navneeth Krishnan <re...@gmail.com> on 2020/12/14 19:29:01 UTC
Kafka Streams - Source Partition Assignment Issue
Hi All,
How does kafka streams partition assignment work for sources? I have a
stream application reading from a topic which has 24 partitions. There are
6 application containers with 4 stream tasks in each container running but
only 2 instances are assigned with partitions and even within the two, one
has 21 partitions and the other has 3 partitions assigned. There is
something wrong in the way the partitions are assigned. Is there a way to
evenly distribute partitions?
*Group Assignment:*
userapp user-data 5 1716495057 1716504638
9581
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
userapp user-data 13 1735721509 1735730754
9245
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
userapp user-data 9 1729975595 1729984597
9002
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
userapp user-data 17 1736457935 1736466997
9062
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
userapp user-data 0 1917953948 1917963103
9155
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
userapp user-data 11 1719082482 1719085979
3497
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp user-data 19 1755067810 1755071386
3576
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp user-data 7 1729598367 1729601982
3615
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp user-data 1 1949943357 1949947252
3895
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp user-data 15 1751889619 1751893116
3497
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp user-data 3 1752352594 1752356255
3661
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp user-data 6 1693612741 1693621514
8773
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
userapp user-data 2 1923482321 1923490826
8505
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
userapp user-data 14 1677960791 1677969039
8248
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
userapp user-data 10 1722133802 1722142388
8586
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
userapp user-data 18 1739152927 1739161360
8433
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
userapp user-data 20 1715428183 1715431991
3808
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
userapp user-data 8 1738854860 1738858541
3681
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
userapp user-data 4 1704061352 1704065142
3790
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
userapp user-data 12 1743370130 1743373727
3597
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
userapp user-data 16 1696027581 1696031399
3818
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
/172.21.82.140
userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
userapp user-data 21 1722643635 1722647766
4131
userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer-19fdbf9f-fb65-45fb-a020-aa16e9da76c5
/172.21.64.86
userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer
userapp user-data 23 1675792619 1675800530
7911
userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer-e98dae60-89b9-44b0-a366-d533b288fcf8
/172.21.64.86
userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer
userapp user-data 22 1735482901 1735490720
7819
userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer-3553c1a0-def4-453d-a4fb-a6c4f08ad3f2
/172.21.64.86
userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer
Regards,
Navneeth
Re: Kafka Streams - Source Partition Assignment Issue
Posted by Navneeth Krishnan <re...@gmail.com>.
In addition to the assignment issue I see the state of the application
changes every few minutes from Running to Rebalancing.
Other than the default configs only the below two extra config parameters
are set. Currently there are 10 instances and they are constantly
rebalancing.
fetch-max-wait: 3000
session.timeout.ms: 60000
2020-12-14T23:34:54.715-08:00
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48-StreamThread-4] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:54.715-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-3] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:54.715-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-3] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:54.721-08:00
[userapp-77c70232-bd1e-4d7f-9543-2d5b6ccf2781-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-77c70232-bd1e-4d7f-9543-2d5b6ccf2781] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:54.721-08:00
[userapp-77c70232-bd1e-4d7f-9543-2d5b6ccf2781-StreamThread-4] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:54.733-08:00
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a] State transition from
RUNNING to REBALANCING
*2020-12-14T23:34:54.733-08:00*
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO State
transition from RUNNING to REBALANCING
*2020-12-14T23:34:57.898-08:00
*[userapp-edc066cc-2a51-4c92-988c-0b03f1015398-StreamThread-1]
INFO org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-edc066cc-2a51-4c92-988c-0b03f1015398] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.898-08:00
[userapp-edc066cc-2a51-4c92-988c-0b03f1015398-StreamThread-1] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:57.899-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.899-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-4] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:57.899-08:00
[userapp-0f2af06e-e8f2-4dc2-b852-0a6484c7e4e9-StreamThread-3] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-0f2af06e-e8f2-4dc2-b852-0a6484c7e4e9] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.899-08:00
[userapp-0f2af06e-e8f2-4dc2-b852-0a6484c7e4e9-StreamThread-3] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48-StreamThread-3] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48-StreamThread-3] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-bd83fccf-8cc7-4f80-bd94-42cffd65faf5-StreamThread-2] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-bd83fccf-8cc7-4f80-bd94-42cffd65faf5] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-bd83fccf-8cc7-4f80-bd94-42cffd65faf5-StreamThread-2] INFO State
transition from RUNNING to REBALANCING
Thanks
On Mon, Dec 14, 2020 at 4:07 PM Navneeth Krishnan <re...@gmail.com>
wrote:
> Thanks Guozhang for the suggestion.
>
> We are using kafka 2.3.0 and the app.id is set to the same value.
>
> Bouncing off instances work for a small period of time but when auto
> scaling happens it skews it up again. This is becoming a big bottleneck for
> us since only one node is now trying to handle more than 60% of traffic and
> we are getting into huge lag.
>
> Regards,
> Navneeth
>
> On Mon, Dec 14, 2020 at 2:49 PM Guozhang Wang <wa...@gmail.com> wrote:
>
>> Kafka Streams should evenly distribute the partitions, but there are some
>> issues in old versions of Kafka that you may be observing. To verify if it
>> is a transient issue or it is permanently, I'd suggest you try:
>>
>> 1) bounce the instances that have no partitions assigned (not bounce them
>> all together, but in rolling manner), and see if after the rebalance they
>> are evenly distributed.
>>
>> 2) check if the app.id are set to the same value.
>>
>> Also which version of Kafka are you using?
>>
>>
>> Guozhang
>>
>>
>>
>>
>> On Mon, Dec 14, 2020 at 11:29 AM Navneeth Krishnan <
>> reachnavneeth2@gmail.com>
>> wrote:
>>
>> > Hi All,
>> >
>> > How does kafka streams partition assignment work for sources? I have a
>> > stream application reading from a topic which has 24 partitions. There
>> are
>> > 6 application containers with 4 stream tasks in each container running
>> but
>> > only 2 instances are assigned with partitions and even within the two,
>> one
>> > has 21 partitions and the other has 3 partitions assigned. There is
>> > something wrong in the way the partitions are assigned. Is there a way
>> to
>> > evenly distribute partitions?
>> >
>> > *Group Assignment:*
>> > userapp user-data 5 1716495057
>> 1716504638
>> > 9581
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp user-data 13 1735721509
>> 1735730754
>> > 9245
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp user-data 9 1729975595
>> 1729984597
>> > 9002
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp user-data 17 1736457935
>> 1736466997
>> > 9062
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp user-data 0 1917953948
>> 1917963103
>> > 9155
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp user-data 11 1719082482
>> 1719085979
>> > 3497
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 19 1755067810
>> 1755071386
>> > 3576
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 7 1729598367
>> 1729601982
>> > 3615
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 1 1949943357
>> 1949947252
>> > 3895
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 15 1751889619
>> 1751893116
>> > 3497
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 3 1752352594
>> 1752356255
>> > 3661
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 6 1693612741
>> 1693621514
>> > 8773
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp user-data 2 1923482321
>> 1923490826
>> > 8505
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp user-data 14 1677960791
>> 1677969039
>> > 8248
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp user-data 10 1722133802
>> 1722142388
>> > 8586
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp user-data 18 1739152927
>> 1739161360
>> > 8433
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp user-data 20 1715428183
>> 1715431991
>> > 3808
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp user-data 8 1738854860
>> 1738858541
>> > 3681
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp user-data 4 1704061352
>> 1704065142
>> > 3790
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp user-data 12 1743370130
>> 1743373727
>> > 3597
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp user-data 16 1696027581
>> 1696031399
>> > 3818
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp user-data 21 1722643635
>> 1722647766
>> > 4131
>> >
>> >
>> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer-19fdbf9f-fb65-45fb-a020-aa16e9da76c5
>> > /172.21.64.86
>> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer
>> > userapp user-data 23 1675792619
>> 1675800530
>> > 7911
>> >
>> >
>> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer-e98dae60-89b9-44b0-a366-d533b288fcf8
>> > /172.21.64.86
>> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer
>> > userapp user-data 22 1735482901
>> 1735490720
>> > 7819
>> >
>> >
>> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer-3553c1a0-def4-453d-a4fb-a6c4f08ad3f2
>> > /172.21.64.86
>> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer
>> >
>> > Regards,
>> > Navneeth
>> >
>>
>>
>> --
>> -- Guozhang
>>
>
Re: Kafka Streams - Source Partition Assignment Issue
Posted by Navneeth Krishnan <re...@gmail.com>.
Thanks Guozhang for the suggestion.
We are using kafka 2.3.0 and the app.id is set to the same value.
Bouncing off instances work for a small period of time but when auto
scaling happens it skews it up again. This is becoming a big bottleneck for
us since only one node is now trying to handle more than 60% of traffic and
we are getting into huge lag.
Regards,
Navneeth
On Mon, Dec 14, 2020 at 2:49 PM Guozhang Wang <wa...@gmail.com> wrote:
> Kafka Streams should evenly distribute the partitions, but there are some
> issues in old versions of Kafka that you may be observing. To verify if it
> is a transient issue or it is permanently, I'd suggest you try:
>
> 1) bounce the instances that have no partitions assigned (not bounce them
> all together, but in rolling manner), and see if after the rebalance they
> are evenly distributed.
>
> 2) check if the app.id are set to the same value.
>
> Also which version of Kafka are you using?
>
>
> Guozhang
>
>
>
>
> On Mon, Dec 14, 2020 at 11:29 AM Navneeth Krishnan <
> reachnavneeth2@gmail.com>
> wrote:
>
> > Hi All,
> >
> > How does kafka streams partition assignment work for sources? I have a
> > stream application reading from a topic which has 24 partitions. There
> are
> > 6 application containers with 4 stream tasks in each container running
> but
> > only 2 instances are assigned with partitions and even within the two,
> one
> > has 21 partitions and the other has 3 partitions assigned. There is
> > something wrong in the way the partitions are assigned. Is there a way to
> > evenly distribute partitions?
> >
> > *Group Assignment:*
> > userapp user-data 5 1716495057 1716504638
> > 9581
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp user-data 13 1735721509 1735730754
> > 9245
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp user-data 9 1729975595 1729984597
> > 9002
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp user-data 17 1736457935 1736466997
> > 9062
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp user-data 0 1917953948 1917963103
> > 9155
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp user-data 11 1719082482 1719085979
> > 3497
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 19 1755067810 1755071386
> > 3576
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 7 1729598367 1729601982
> > 3615
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 1 1949943357 1949947252
> > 3895
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 15 1751889619 1751893116
> > 3497
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 3 1752352594 1752356255
> > 3661
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 6 1693612741 1693621514
> > 8773
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp user-data 2 1923482321 1923490826
> > 8505
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp user-data 14 1677960791 1677969039
> > 8248
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp user-data 10 1722133802 1722142388
> > 8586
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp user-data 18 1739152927 1739161360
> > 8433
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp user-data 20 1715428183 1715431991
> > 3808
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp user-data 8 1738854860 1738858541
> > 3681
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp user-data 4 1704061352 1704065142
> > 3790
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp user-data 12 1743370130 1743373727
> > 3597
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp user-data 16 1696027581 1696031399
> > 3818
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp user-data 21 1722643635 1722647766
> > 4131
> >
> >
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer-19fdbf9f-fb65-45fb-a020-aa16e9da76c5
> > /172.21.64.86
> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer
> > userapp user-data 23 1675792619 1675800530
> > 7911
> >
> >
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer-e98dae60-89b9-44b0-a366-d533b288fcf8
> > /172.21.64.86
> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer
> > userapp user-data 22 1735482901 1735490720
> > 7819
> >
> >
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer-3553c1a0-def4-453d-a4fb-a6c4f08ad3f2
> > /172.21.64.86
> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer
> >
> > Regards,
> > Navneeth
> >
>
>
> --
> -- Guozhang
>
Re: Kafka Streams - Source Partition Assignment Issue
Posted by Guozhang Wang <wa...@gmail.com>.
Kafka Streams should evenly distribute the partitions, but there are some
issues in old versions of Kafka that you may be observing. To verify if it
is a transient issue or it is permanently, I'd suggest you try:
1) bounce the instances that have no partitions assigned (not bounce them
all together, but in rolling manner), and see if after the rebalance they
are evenly distributed.
2) check if the app.id are set to the same value.
Also which version of Kafka are you using?
Guozhang
On Mon, Dec 14, 2020 at 11:29 AM Navneeth Krishnan <re...@gmail.com>
wrote:
> Hi All,
>
> How does kafka streams partition assignment work for sources? I have a
> stream application reading from a topic which has 24 partitions. There are
> 6 application containers with 4 stream tasks in each container running but
> only 2 instances are assigned with partitions and even within the two, one
> has 21 partitions and the other has 3 partitions assigned. There is
> something wrong in the way the partitions are assigned. Is there a way to
> evenly distribute partitions?
>
> *Group Assignment:*
> userapp user-data 5 1716495057 1716504638
> 9581
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> userapp user-data 13 1735721509 1735730754
> 9245
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> userapp user-data 9 1729975595 1729984597
> 9002
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> userapp user-data 17 1736457935 1736466997
> 9062
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> userapp user-data 0 1917953948 1917963103
> 9155
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> userapp user-data 11 1719082482 1719085979
> 3497
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> userapp user-data 19 1755067810 1755071386
> 3576
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> userapp user-data 7 1729598367 1729601982
> 3615
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> userapp user-data 1 1949943357 1949947252
> 3895
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> userapp user-data 15 1751889619 1751893116
> 3497
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> userapp user-data 3 1752352594 1752356255
> 3661
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> userapp user-data 6 1693612741 1693621514
> 8773
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> userapp user-data 2 1923482321 1923490826
> 8505
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> userapp user-data 14 1677960791 1677969039
> 8248
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> userapp user-data 10 1722133802 1722142388
> 8586
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> userapp user-data 18 1739152927 1739161360
> 8433
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> userapp user-data 20 1715428183 1715431991
> 3808
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> userapp user-data 8 1738854860 1738858541
> 3681
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> userapp user-data 4 1704061352 1704065142
> 3790
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> userapp user-data 12 1743370130 1743373727
> 3597
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> userapp user-data 16 1696027581 1696031399
> 3818
>
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> /172.21.82.140
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> userapp user-data 21 1722643635 1722647766
> 4131
>
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer-19fdbf9f-fb65-45fb-a020-aa16e9da76c5
> /172.21.64.86
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer
> userapp user-data 23 1675792619 1675800530
> 7911
>
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer-e98dae60-89b9-44b0-a366-d533b288fcf8
> /172.21.64.86
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer
> userapp user-data 22 1735482901 1735490720
> 7819
>
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer-3553c1a0-def4-453d-a4fb-a6c4f08ad3f2
> /172.21.64.86
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer
>
> Regards,
> Navneeth
>
--
-- Guozhang