You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Laven Sukumaran <la...@autonomic.ai> on 2017/10/24 14:21:17 UTC
Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior
Hi all,
We're upgrading a Kafka streams application from 0.10.2.1 to 0.11.0.1 and
our application is running against a Kafka cluster with version 0.10.2.1.
When we first attempted to upgrade our application to Kafka 0.11.0.1, we
observed that when we deleted the PVCs for the service, and restarted, the
new Kafka stream thread task stickiness behavior (
https://issues.apache.org/jira/browse/KAFKA-4677) had unintentional
side-effects. Specifically, in scenarios where PVC recreation is necessary,
it appears that the new task stickiness semantics caused the task assignor
to prematurely decide that there was only ever going to be one instance of
our app in the consumer groups, and therefore all stream threads would be
launched on the first instance in the consumer group. Later, when the
second and third instance of our app were launched, the task assignor did
not take these new instances into account, and none of the partitions were
assigned to the second and third instance of our app.
We worked around this by deleting the PVCs, and when pod one was running
(running as defined by k8s), we would repeatedly kubectl delete the pod
until instance 2 and 3 were successfully running as well. Then, we would
allow the reprocessing to take place. This caused the task assignor to take
into account all pod instances, before making assignments which 'stuck'.
I saw with the 0.11.0.0 change that a new broker config was introduced "
group.initial.rebalance.delay.ms" however from what I understand this would
require upgrading our Kafka cluster to 0.11.0.0 which is not something we
will be able to do in the near future.
*"The group.initial.rebalance.delay.ms
<http://group.initial.rebalance.delay.ms/> config specifies the amount of
time, in milliseconds, that the GroupCoordinator will delay the initial
consumer rebalance. The rebalance is further delayed by the value
of group.initial.rebalance.delay.ms
<http://group.initial.rebalance.delay.ms/> as each new member joins the
consumer group, up to a maximum of the value set by max.poll.interval.ms
<http://max.poll.interval.ms/>. The net benefit is that this can reduce the
overall startup time for Kafka Streams applications that have more than one
thread. The default value for group.initial.rebalance.delay.ms
<http://group.initial.rebalance.delay.ms/> is 3000 milliseconds.In practice
this means that if you are starting up your Kafka Streams app from a cold
start, when the first member joins the group there will be at least a 3
second delay before it is assigned any tasks. If any other members join the
group within the initial 3 seconds, then there will be a further 3 second
delay. Once no new members have joined the group within the 3 second delay,
or max.poll.interval.ms <http://max.poll.interval.ms/> is reached, then the
group rebalance can complete and all current members will be assigned
tasks. The benefit of this approach, particularly for Kafka Streams
applications, is that we can now delay the assignment and re-assignment of
potentially expensive tasks as new members join. So we can avoid the
situation where one instance is assigned all tasks, begins
restoring/processing, only to shortly after be rebalanced, and then have to
start again with half of the tasks and so on."*
Similar to the "group.initial.rebalance.delay.ms", are there any
configuration values we can tweak to defer sticky task assignment to ensure
all instances are running first.
Thanks,
Laven
Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior
Posted by Eric Lalonde <er...@autonomic.ai>.
Hi Guozhang, I will talk to the team about prioritizing the recreation of this scenario and gathering logs.
- Eric
> On Nov 5, 2017, at 9:39 AM, Guozhang Wang <wa...@gmail.com> wrote:
>
> Hello Laven / Eric,
>
> I personally cannot think of any more bug fixes than KAFKA-5152 that is
> trying to solve a relative issue, and we would love to know what did you
> observe that in 0.11.0.1 this issue still exists: i.e. the first instance
> is stuck with the rebalance state, as we have moved the restoration process
> completely out of the rebalance phase in 0.11.0.1. So as Matthias
> mentioned, if you still saw this issue in 0.11.0.1 it may indicate another
> issue.
>
> Again, we are super interested to learn your scenario, and see if it is
> indeed the case.
>
> Guozhang
>
> On Sat, Nov 4, 2017 at 7:26 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Streams is backward compatible from 0.10.1 on.
>>
>> So 1.0.0 does not work with 0.10.0 but 0.10.1, 0.10.2, and 0.11.0.
>>
>> Compare:
>> https://docs.confluent.io/current/streams/upgrade-guide.html#compatibility
>>
>>
>> -Matthias
>>
>> On 10/25/17 7:54 AM, Eric Lalonde wrote:
>>>
>>>> On Oct 24, 2017, at 8:53 PM, Matthias J. Sax <ma...@confluent.io>
>> wrote:
>>>>
>>>> Might be worth a try with 1.0.0 RC3 -- even if I doubt that much
>> changes.
>>>>
>>>> Can you provide debug logs for your Kafka streams applications as well
>>>> as brokers? This would help to dig into this.
>>>>
>>>
>>> I searched, but it wasn’t clear to me — would a Kstream 1.0.0 RC3 client
>> be backwards compatible with a broker from the 0.10.x line?
>>>
>>>
>>
>>
>
>
> --
> -- Guozhang
Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Laven / Eric,
I personally cannot think of any more bug fixes than KAFKA-5152 that is
trying to solve a relative issue, and we would love to know what did you
observe that in 0.11.0.1 this issue still exists: i.e. the first instance
is stuck with the rebalance state, as we have moved the restoration process
completely out of the rebalance phase in 0.11.0.1. So as Matthias
mentioned, if you still saw this issue in 0.11.0.1 it may indicate another
issue.
Again, we are super interested to learn your scenario, and see if it is
indeed the case.
Guozhang
On Sat, Nov 4, 2017 at 7:26 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Streams is backward compatible from 0.10.1 on.
>
> So 1.0.0 does not work with 0.10.0 but 0.10.1, 0.10.2, and 0.11.0.
>
> Compare:
> https://docs.confluent.io/current/streams/upgrade-guide.html#compatibility
>
>
> -Matthias
>
> On 10/25/17 7:54 AM, Eric Lalonde wrote:
> >
> >> On Oct 24, 2017, at 8:53 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
> >>
> >> Might be worth a try with 1.0.0 RC3 -- even if I doubt that much
> changes.
> >>
> >> Can you provide debug logs for your Kafka streams applications as well
> >> as brokers? This would help to dig into this.
> >>
> >
> > I searched, but it wasn’t clear to me — would a Kstream 1.0.0 RC3 client
> be backwards compatible with a broker from the 0.10.x line?
> >
> >
>
>
--
-- Guozhang
Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Streams is backward compatible from 0.10.1 on.
So 1.0.0 does not work with 0.10.0 but 0.10.1, 0.10.2, and 0.11.0.
Compare:
https://docs.confluent.io/current/streams/upgrade-guide.html#compatibility
-Matthias
On 10/25/17 7:54 AM, Eric Lalonde wrote:
>
>> On Oct 24, 2017, at 8:53 PM, Matthias J. Sax <ma...@confluent.io> wrote:
>>
>> Might be worth a try with 1.0.0 RC3 -- even if I doubt that much changes.
>>
>> Can you provide debug logs for your Kafka streams applications as well
>> as brokers? This would help to dig into this.
>>
>
> I searched, but it wasn’t clear to me — would a Kstream 1.0.0 RC3 client be backwards compatible with a broker from the 0.10.x line?
>
>
Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior
Posted by Eric Lalonde <er...@lalonde.com>.
> On Oct 24, 2017, at 8:53 PM, Matthias J. Sax <ma...@confluent.io> wrote:
>
> Might be worth a try with 1.0.0 RC3 -- even if I doubt that much changes.
>
> Can you provide debug logs for your Kafka streams applications as well
> as brokers? This would help to dig into this.
>
I searched, but it wasn’t clear to me — would a Kstream 1.0.0 RC3 client be backwards compatible with a broker from the 0.10.x line?
Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Might be worth a try with 1.0.0 RC3 -- even if I doubt that much changes.
Can you provide debug logs for your Kafka streams applications as well
as brokers? This would help to dig into this.
-Matthias
On 10/24/17 5:53 PM, Ted Yu wrote:
> Eric:
> I wonder if it is possible to load up 1.0.0 RC3 on test cluster and see
> what the new behavior is ?
>
> Thanks
>
> On Tue, Oct 24, 2017 at 5:41 PM, Eric Lalonde <er...@autonomic.ai> wrote:
>
>>
>>>>> Could it be, that the first KafkaStreams instance was still in status
>>>>> "rebalancing" when you started the second/third container? If yes, this
>>>>> might explain what you observed: if the first instance is in status
>>>>> "rebalancing" it would miss that new instanced are joining the group.
>>>>> (We fixed this in upcoming 1.0.0 release).
>>
>>
>> Hi Matthias,
>>
>> I appreciate your time. Your characterization above of the rebalancing is
>> exactly the behavior we are seeing. Unfortunately, the issue still occurs
>> in 0.11.0.1, so I suspect the behavior you are describing is not fixed by
>> commits expressed in KAFKA-5152, or not completely.
>>
>> - Eric
>>
>>
>
Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior
Posted by Ted Yu <yu...@gmail.com>.
Eric:
I wonder if it is possible to load up 1.0.0 RC3 on test cluster and see
what the new behavior is ?
Thanks
On Tue, Oct 24, 2017 at 5:41 PM, Eric Lalonde <er...@autonomic.ai> wrote:
>
> >>> Could it be, that the first KafkaStreams instance was still in status
> >>> "rebalancing" when you started the second/third container? If yes, this
> >>> might explain what you observed: if the first instance is in status
> >>> "rebalancing" it would miss that new instanced are joining the group.
> >>> (We fixed this in upcoming 1.0.0 release).
>
>
> Hi Matthias,
>
> I appreciate your time. Your characterization above of the rebalancing is
> exactly the behavior we are seeing. Unfortunately, the issue still occurs
> in 0.11.0.1, so I suspect the behavior you are describing is not fixed by
> commits expressed in KAFKA-5152, or not completely.
>
> - Eric
>
>
Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior
Posted by Eric Lalonde <er...@autonomic.ai>.
>>> Could it be, that the first KafkaStreams instance was still in status
>>> "rebalancing" when you started the second/third container? If yes, this
>>> might explain what you observed: if the first instance is in status
>>> "rebalancing" it would miss that new instanced are joining the group.
>>> (We fixed this in upcoming 1.0.0 release).
Hi Matthias,
I appreciate your time. Your characterization above of the rebalancing is exactly the behavior we are seeing. Unfortunately, the issue still occurs in 0.11.0.1, so I suspect the behavior you are describing is not fixed by commits expressed in KAFKA-5152, or not completely.
- Eric
Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior
Posted by "Matthias J. Sax" <ma...@confluent.io>.
We had multiple Jira. I guess this one is the fix you are looking for:
https://issues.apache.org/jira/browse/KAFKA-5152
-Matthias
On 10/24/17 3:21 PM, Eric Lalonde wrote:
>>
>> Could it be, that the first KafkaStreams instance was still in status
>> "rebalancing" when you started the second/third container? If yes, this
>> might explain what you observed: if the first instance is in status
>> "rebalancing" it would miss that new instanced are joining the group.
>> (We fixed this in upcoming 1.0.0 release).
>
> Is there a JIRA which tracks this fix?
>
> Thanks,
>
> - Eric
>
Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior
Posted by Eric Lalonde <er...@autonomic.ai>.
>
> Could it be, that the first KafkaStreams instance was still in status
> "rebalancing" when you started the second/third container? If yes, this
> might explain what you observed: if the first instance is in status
> "rebalancing" it would miss that new instanced are joining the group.
> (We fixed this in upcoming 1.0.0 release).
Is there a JIRA which tracks this fix?
Thanks,
- Eric
Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Hi,
the issue you describe, that on a "fresh" restart, all tasks are
assigned to the first thread is known, and the solution for it was to
introduce the new broker config you mentioned. Thus, there is no config
for 0.10.2.x brokers or Streams API to handle this case (that's why we
introduced the new config).
However, what I do not understand is, how you do your upgrade. From your
description, it seems that you stop the whole application, and afterward
restart all containers. However, Kafka Streams support rolling upgrades.
Thus, you could just stop-restart a single container, and after this
container did recreate all it's state and is running, you can move on to
the next one and so on, until all containers are upgraded to 0.11.0.1.
About the StickyAssignor: even for a fresh restart and after it did
assign all tasks to the first thread, it should move tasks to newly
starting thread. StickyAssignor put load balancing first and stickiness
second when deciding to place tasks. Thus, what you describe should not
happen and if it happens it's a bug. We would need DEBUG level logs to
dig into this.
Could it be, that the first KafkaStreams instance was still in status
"rebalancing" when you started the second/third container? If yes, this
might explain what you observed: if the first instance is in status
"rebalancing" it would miss that new instanced are joining the group.
(We fixed this in upcoming 1.0.0 release).
Hope this helps.
-Matthias
On 10/24/17 7:21 AM, Laven Sukumaran wrote:
> Hi all,
>
> We're upgrading a Kafka streams application from 0.10.2.1 to 0.11.0.1 and
> our application is running against a Kafka cluster with version 0.10.2.1.
>
> When we first attempted to upgrade our application to Kafka 0.11.0.1, we
> observed that when we deleted the PVCs for the service, and restarted, the
> new Kafka stream thread task stickiness behavior (
> https://issues.apache.org/jira/browse/KAFKA-4677) had unintentional
> side-effects. Specifically, in scenarios where PVC recreation is necessary,
> it appears that the new task stickiness semantics caused the task assignor
> to prematurely decide that there was only ever going to be one instance of
> our app in the consumer groups, and therefore all stream threads would be
> launched on the first instance in the consumer group. Later, when the
> second and third instance of our app were launched, the task assignor did
> not take these new instances into account, and none of the partitions were
> assigned to the second and third instance of our app.
>
> We worked around this by deleting the PVCs, and when pod one was running
> (running as defined by k8s), we would repeatedly kubectl delete the pod
> until instance 2 and 3 were successfully running as well. Then, we would
> allow the reprocessing to take place. This caused the task assignor to take
> into account all pod instances, before making assignments which 'stuck'.
>
>
> I saw with the 0.11.0.0 change that a new broker config was introduced "
> group.initial.rebalance.delay.ms" however from what I understand this would
> require upgrading our Kafka cluster to 0.11.0.0 which is not something we
> will be able to do in the near future.
>
>
>
> *"The group.initial.rebalance.delay.ms
> <http://group.initial.rebalance.delay.ms/> config specifies the amount of
> time, in milliseconds, that the GroupCoordinator will delay the initial
> consumer rebalance. The rebalance is further delayed by the value
> of group.initial.rebalance.delay.ms
> <http://group.initial.rebalance.delay.ms/> as each new member joins the
> consumer group, up to a maximum of the value set by max.poll.interval.ms
> <http://max.poll.interval.ms/>. The net benefit is that this can reduce the
> overall startup time for Kafka Streams applications that have more than one
> thread. The default value for group.initial.rebalance.delay.ms
> <http://group.initial.rebalance.delay.ms/> is 3000 milliseconds.In practice
> this means that if you are starting up your Kafka Streams app from a cold
> start, when the first member joins the group there will be at least a 3
> second delay before it is assigned any tasks. If any other members join the
> group within the initial 3 seconds, then there will be a further 3 second
> delay. Once no new members have joined the group within the 3 second delay,
> or max.poll.interval.ms <http://max.poll.interval.ms/> is reached, then the
> group rebalance can complete and all current members will be assigned
> tasks. The benefit of this approach, particularly for Kafka Streams
> applications, is that we can now delay the assignment and re-assignment of
> potentially expensive tasks as new members join. So we can avoid the
> situation where one instance is assigned all tasks, begins
> restoring/processing, only to shortly after be rebalanced, and then have to
> start again with half of the tasks and so on."*
>
> Similar to the "group.initial.rebalance.delay.ms", are there any
> configuration values we can tweak to defer sticky task assignment to ensure
> all instances are running first.
>
> Thanks,
> Laven
>