You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bryan Baugher <bj...@gmail.com> on 2017/06/12 21:48:07 UTC
Slow Consumer Group Startup
Hi everyone,
We are currently experiencing slow startup times for our consumer groups
(16-32 processes for a hundred or more partitions) in the range of minutes
(3-15 minutes), where little to no messages are consumed before suddenly
everything just starts working at full speed.
I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading to
Kafka 0.10.2.1. We also using the newer kafka consumer API and group
management on a simple Apache Storm topology. We don't make use of Storm's
kafka spout but instead wrote a simple one ourselves.
Using the kafka AdminClient I was able to poll for consumer group summary
information. What I've found is that the group seems to sit
in PreparingRebalance state for minutes before finally becoming Stable
which then everything starts processing quickly. I've also enabled debug
logging around the consumer's coordinator classes but didn't see anything
to indicate the issue.
I'm hoping that just upgrading to 0.10 or tweaking how we use our consumer
in Apache Storm is the problem but are there any pointers on things I
should look at or try?
Bryan
Re: Slow Consumer Group Startup
Posted by Eno Thereska <en...@gmail.com>.
Hi Bryan,
So this must be something else since KIP-134 is not in 0.10.2.1, but in the new release 0.11 that hasn't come out yet.
Eno
> On 14 Jun 2017, at 21:35, Bryan Baugher <bj...@gmail.com> wrote:
>
> It does seem like we are in a similar situation described in the KIP (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance).
> For some historical reason we had set our session.timeout.ms to a high
> value (5 minutes) which corresponds with the amount of the group would
> wait. Lowering helps new consumers join faster but now I'm seeing group
> fluctuations so I'll have to continue to dig into why thats happening. Also
> worth noting the broker's rebalance timeout seems to switch to
> max.poll.interval.ms in 0.10.1.
>
> On Wed, Jun 14, 2017 at 10:53 AM Bryan Baugher <bj...@gmail.com> wrote:
>
>> While I do have some logs its not trivial to share since the logs are
>> across 16 JVMs and a few different hosts.
>>
>> On Wed, Jun 14, 2017 at 10:34 AM Eno Thereska <en...@gmail.com>
>> wrote:
>>
>>> The delay in that KIP is just 3 seconds, not minutes though, right? Would
>>> you have any logs to share?
>>>
>>> Thanks
>>> Eno
>>>> On 14 Jun 2017, at 16:14, Bryan Baugher <bj...@gmail.com> wrote:
>>>>
>>>> Our consumer group isn't doing anything stateful and we've seen this
>>>> behavior for existing groups as well. It seems like timing could be an
>>>> issue, thanks for the information.
>>>>
>>>> On Tue, Jun 13, 2017 at 7:39 PM James Cheng <wu...@gmail.com>
>>> wrote:
>>>>
>>>>> Bryan,
>>>>>
>>>>> This sounds related to
>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance
>>>>> and https://issues.apache.org/jira/browse/KAFKA-4925.
>>>>>
>>>>> -James
>>>>>
>>>>>> On Jun 13, 2017, at 7:02 AM, Bryan Baugher <bj...@gmail.com> wrote:
>>>>>>
>>>>>> The topics already exist prior to starting any of the consumers
>>>>>>
>>>>>> On Mon, Jun 12, 2017 at 9:35 PM J Pai <ja...@gmail.com>
>>> wrote:
>>>>>>
>>>>>>> When are the topics on which these consumer groups consume, created?
>>>>>>>
>>>>>>> -Jaikiran
>>>>>>> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bj...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> We are currently experiencing slow startup times for our consumer
>>> groups
>>>>>>> (16-32 processes for a hundred or more partitions) in the range of
>>>>> minutes
>>>>>>> (3-15 minutes), where little to no messages are consumed before
>>> suddenly
>>>>>>> everything just starts working at full speed.
>>>>>>>
>>>>>>> I'm currently using Kafka 0.9.0.1 but we are in the middle of
>>> upgrading
>>>>> to
>>>>>>> Kafka 0.10.2.1. We also using the newer kafka consumer API and group
>>>>>>> management on a simple Apache Storm topology. We don't make use of
>>>>> Storm's
>>>>>>> kafka spout but instead wrote a simple one ourselves.
>>>>>>>
>>>>>>> Using the kafka AdminClient I was able to poll for consumer group
>>>>> summary
>>>>>>> information. What I've found is that the group seems to sit
>>>>>>> in PreparingRebalance state for minutes before finally becoming
>>> Stable
>>>>>>> which then everything starts processing quickly. I've also enabled
>>> debug
>>>>>>> logging around the consumer's coordinator classes but didn't see
>>>>> anything
>>>>>>> to indicate the issue.
>>>>>>>
>>>>>>> I'm hoping that just upgrading to 0.10 or tweaking how we use our
>>>>> consumer
>>>>>>> in Apache Storm is the problem but are there any pointers on things I
>>>>>>> should look at or try?
>>>>>>>
>>>>>>> Bryan
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
Re: Slow Consumer Group Startup
Posted by Bryan Baugher <bj...@gmail.com>.
It does seem like we are in a similar situation described in the KIP (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance).
For some historical reason we had set our session.timeout.ms to a high
value (5 minutes) which corresponds with the amount of the group would
wait. Lowering helps new consumers join faster but now I'm seeing group
fluctuations so I'll have to continue to dig into why thats happening. Also
worth noting the broker's rebalance timeout seems to switch to
max.poll.interval.ms in 0.10.1.
On Wed, Jun 14, 2017 at 10:53 AM Bryan Baugher <bj...@gmail.com> wrote:
> While I do have some logs its not trivial to share since the logs are
> across 16 JVMs and a few different hosts.
>
> On Wed, Jun 14, 2017 at 10:34 AM Eno Thereska <en...@gmail.com>
> wrote:
>
>> The delay in that KIP is just 3 seconds, not minutes though, right? Would
>> you have any logs to share?
>>
>> Thanks
>> Eno
>> > On 14 Jun 2017, at 16:14, Bryan Baugher <bj...@gmail.com> wrote:
>> >
>> > Our consumer group isn't doing anything stateful and we've seen this
>> > behavior for existing groups as well. It seems like timing could be an
>> > issue, thanks for the information.
>> >
>> > On Tue, Jun 13, 2017 at 7:39 PM James Cheng <wu...@gmail.com>
>> wrote:
>> >
>> >> Bryan,
>> >>
>> >> This sounds related to
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance
>> >> and https://issues.apache.org/jira/browse/KAFKA-4925.
>> >>
>> >> -James
>> >>
>> >>> On Jun 13, 2017, at 7:02 AM, Bryan Baugher <bj...@gmail.com> wrote:
>> >>>
>> >>> The topics already exist prior to starting any of the consumers
>> >>>
>> >>> On Mon, Jun 12, 2017 at 9:35 PM J Pai <ja...@gmail.com>
>> wrote:
>> >>>
>> >>>> When are the topics on which these consumer groups consume, created?
>> >>>>
>> >>>> -Jaikiran
>> >>>> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bj...@gmail.com> wrote:
>> >>>>
>> >>>> Hi everyone,
>> >>>>
>> >>>> We are currently experiencing slow startup times for our consumer
>> groups
>> >>>> (16-32 processes for a hundred or more partitions) in the range of
>> >> minutes
>> >>>> (3-15 minutes), where little to no messages are consumed before
>> suddenly
>> >>>> everything just starts working at full speed.
>> >>>>
>> >>>> I'm currently using Kafka 0.9.0.1 but we are in the middle of
>> upgrading
>> >> to
>> >>>> Kafka 0.10.2.1. We also using the newer kafka consumer API and group
>> >>>> management on a simple Apache Storm topology. We don't make use of
>> >> Storm's
>> >>>> kafka spout but instead wrote a simple one ourselves.
>> >>>>
>> >>>> Using the kafka AdminClient I was able to poll for consumer group
>> >> summary
>> >>>> information. What I've found is that the group seems to sit
>> >>>> in PreparingRebalance state for minutes before finally becoming
>> Stable
>> >>>> which then everything starts processing quickly. I've also enabled
>> debug
>> >>>> logging around the consumer's coordinator classes but didn't see
>> >> anything
>> >>>> to indicate the issue.
>> >>>>
>> >>>> I'm hoping that just upgrading to 0.10 or tweaking how we use our
>> >> consumer
>> >>>> in Apache Storm is the problem but are there any pointers on things I
>> >>>> should look at or try?
>> >>>>
>> >>>> Bryan
>> >>>>
>> >>>>
>> >>
>> >>
>>
>>
Re: Slow Consumer Group Startup
Posted by Bryan Baugher <bj...@gmail.com>.
While I do have some logs its not trivial to share since the logs are
across 16 JVMs and a few different hosts.
On Wed, Jun 14, 2017 at 10:34 AM Eno Thereska <en...@gmail.com>
wrote:
> The delay in that KIP is just 3 seconds, not minutes though, right? Would
> you have any logs to share?
>
> Thanks
> Eno
> > On 14 Jun 2017, at 16:14, Bryan Baugher <bj...@gmail.com> wrote:
> >
> > Our consumer group isn't doing anything stateful and we've seen this
> > behavior for existing groups as well. It seems like timing could be an
> > issue, thanks for the information.
> >
> > On Tue, Jun 13, 2017 at 7:39 PM James Cheng <wu...@gmail.com>
> wrote:
> >
> >> Bryan,
> >>
> >> This sounds related to
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance
> >> and https://issues.apache.org/jira/browse/KAFKA-4925.
> >>
> >> -James
> >>
> >>> On Jun 13, 2017, at 7:02 AM, Bryan Baugher <bj...@gmail.com> wrote:
> >>>
> >>> The topics already exist prior to starting any of the consumers
> >>>
> >>> On Mon, Jun 12, 2017 at 9:35 PM J Pai <ja...@gmail.com>
> wrote:
> >>>
> >>>> When are the topics on which these consumer groups consume, created?
> >>>>
> >>>> -Jaikiran
> >>>> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bj...@gmail.com> wrote:
> >>>>
> >>>> Hi everyone,
> >>>>
> >>>> We are currently experiencing slow startup times for our consumer
> groups
> >>>> (16-32 processes for a hundred or more partitions) in the range of
> >> minutes
> >>>> (3-15 minutes), where little to no messages are consumed before
> suddenly
> >>>> everything just starts working at full speed.
> >>>>
> >>>> I'm currently using Kafka 0.9.0.1 but we are in the middle of
> upgrading
> >> to
> >>>> Kafka 0.10.2.1. We also using the newer kafka consumer API and group
> >>>> management on a simple Apache Storm topology. We don't make use of
> >> Storm's
> >>>> kafka spout but instead wrote a simple one ourselves.
> >>>>
> >>>> Using the kafka AdminClient I was able to poll for consumer group
> >> summary
> >>>> information. What I've found is that the group seems to sit
> >>>> in PreparingRebalance state for minutes before finally becoming Stable
> >>>> which then everything starts processing quickly. I've also enabled
> debug
> >>>> logging around the consumer's coordinator classes but didn't see
> >> anything
> >>>> to indicate the issue.
> >>>>
> >>>> I'm hoping that just upgrading to 0.10 or tweaking how we use our
> >> consumer
> >>>> in Apache Storm is the problem but are there any pointers on things I
> >>>> should look at or try?
> >>>>
> >>>> Bryan
> >>>>
> >>>>
> >>
> >>
>
>
Re: Slow Consumer Group Startup
Posted by Eno Thereska <en...@gmail.com>.
The delay in that KIP is just 3 seconds, not minutes though, right? Would you have any logs to share?
Thanks
Eno
> On 14 Jun 2017, at 16:14, Bryan Baugher <bj...@gmail.com> wrote:
>
> Our consumer group isn't doing anything stateful and we've seen this
> behavior for existing groups as well. It seems like timing could be an
> issue, thanks for the information.
>
> On Tue, Jun 13, 2017 at 7:39 PM James Cheng <wu...@gmail.com> wrote:
>
>> Bryan,
>>
>> This sounds related to
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance
>> and https://issues.apache.org/jira/browse/KAFKA-4925.
>>
>> -James
>>
>>> On Jun 13, 2017, at 7:02 AM, Bryan Baugher <bj...@gmail.com> wrote:
>>>
>>> The topics already exist prior to starting any of the consumers
>>>
>>> On Mon, Jun 12, 2017 at 9:35 PM J Pai <ja...@gmail.com> wrote:
>>>
>>>> When are the topics on which these consumer groups consume, created?
>>>>
>>>> -Jaikiran
>>>> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bj...@gmail.com> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> We are currently experiencing slow startup times for our consumer groups
>>>> (16-32 processes for a hundred or more partitions) in the range of
>> minutes
>>>> (3-15 minutes), where little to no messages are consumed before suddenly
>>>> everything just starts working at full speed.
>>>>
>>>> I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading
>> to
>>>> Kafka 0.10.2.1. We also using the newer kafka consumer API and group
>>>> management on a simple Apache Storm topology. We don't make use of
>> Storm's
>>>> kafka spout but instead wrote a simple one ourselves.
>>>>
>>>> Using the kafka AdminClient I was able to poll for consumer group
>> summary
>>>> information. What I've found is that the group seems to sit
>>>> in PreparingRebalance state for minutes before finally becoming Stable
>>>> which then everything starts processing quickly. I've also enabled debug
>>>> logging around the consumer's coordinator classes but didn't see
>> anything
>>>> to indicate the issue.
>>>>
>>>> I'm hoping that just upgrading to 0.10 or tweaking how we use our
>> consumer
>>>> in Apache Storm is the problem but are there any pointers on things I
>>>> should look at or try?
>>>>
>>>> Bryan
>>>>
>>>>
>>
>>
Re: Slow Consumer Group Startup
Posted by Bryan Baugher <bj...@gmail.com>.
Our consumer group isn't doing anything stateful and we've seen this
behavior for existing groups as well. It seems like timing could be an
issue, thanks for the information.
On Tue, Jun 13, 2017 at 7:39 PM James Cheng <wu...@gmail.com> wrote:
> Bryan,
>
> This sounds related to
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance
> and https://issues.apache.org/jira/browse/KAFKA-4925.
>
> -James
>
> > On Jun 13, 2017, at 7:02 AM, Bryan Baugher <bj...@gmail.com> wrote:
> >
> > The topics already exist prior to starting any of the consumers
> >
> > On Mon, Jun 12, 2017 at 9:35 PM J Pai <ja...@gmail.com> wrote:
> >
> >> When are the topics on which these consumer groups consume, created?
> >>
> >> -Jaikiran
> >> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bj...@gmail.com> wrote:
> >>
> >> Hi everyone,
> >>
> >> We are currently experiencing slow startup times for our consumer groups
> >> (16-32 processes for a hundred or more partitions) in the range of
> minutes
> >> (3-15 minutes), where little to no messages are consumed before suddenly
> >> everything just starts working at full speed.
> >>
> >> I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading
> to
> >> Kafka 0.10.2.1. We also using the newer kafka consumer API and group
> >> management on a simple Apache Storm topology. We don't make use of
> Storm's
> >> kafka spout but instead wrote a simple one ourselves.
> >>
> >> Using the kafka AdminClient I was able to poll for consumer group
> summary
> >> information. What I've found is that the group seems to sit
> >> in PreparingRebalance state for minutes before finally becoming Stable
> >> which then everything starts processing quickly. I've also enabled debug
> >> logging around the consumer's coordinator classes but didn't see
> anything
> >> to indicate the issue.
> >>
> >> I'm hoping that just upgrading to 0.10 or tweaking how we use our
> consumer
> >> in Apache Storm is the problem but are there any pointers on things I
> >> should look at or try?
> >>
> >> Bryan
> >>
> >>
>
>
Re: Slow Consumer Group Startup
Posted by James Cheng <wu...@gmail.com>.
Bryan,
This sounds related to https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance and https://issues.apache.org/jira/browse/KAFKA-4925.
-James
> On Jun 13, 2017, at 7:02 AM, Bryan Baugher <bj...@gmail.com> wrote:
>
> The topics already exist prior to starting any of the consumers
>
> On Mon, Jun 12, 2017 at 9:35 PM J Pai <ja...@gmail.com> wrote:
>
>> When are the topics on which these consumer groups consume, created?
>>
>> -Jaikiran
>> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bj...@gmail.com> wrote:
>>
>> Hi everyone,
>>
>> We are currently experiencing slow startup times for our consumer groups
>> (16-32 processes for a hundred or more partitions) in the range of minutes
>> (3-15 minutes), where little to no messages are consumed before suddenly
>> everything just starts working at full speed.
>>
>> I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading to
>> Kafka 0.10.2.1. We also using the newer kafka consumer API and group
>> management on a simple Apache Storm topology. We don't make use of Storm's
>> kafka spout but instead wrote a simple one ourselves.
>>
>> Using the kafka AdminClient I was able to poll for consumer group summary
>> information. What I've found is that the group seems to sit
>> in PreparingRebalance state for minutes before finally becoming Stable
>> which then everything starts processing quickly. I've also enabled debug
>> logging around the consumer's coordinator classes but didn't see anything
>> to indicate the issue.
>>
>> I'm hoping that just upgrading to 0.10 or tweaking how we use our consumer
>> in Apache Storm is the problem but are there any pointers on things I
>> should look at or try?
>>
>> Bryan
>>
>>
Re: Slow Consumer Group Startup
Posted by Bryan Baugher <bj...@gmail.com>.
The topics already exist prior to starting any of the consumers
On Mon, Jun 12, 2017 at 9:35 PM J Pai <ja...@gmail.com> wrote:
> When are the topics on which these consumer groups consume, created?
>
> -Jaikiran
> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bj...@gmail.com> wrote:
>
> Hi everyone,
>
> We are currently experiencing slow startup times for our consumer groups
> (16-32 processes for a hundred or more partitions) in the range of minutes
> (3-15 minutes), where little to no messages are consumed before suddenly
> everything just starts working at full speed.
>
> I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading to
> Kafka 0.10.2.1. We also using the newer kafka consumer API and group
> management on a simple Apache Storm topology. We don't make use of Storm's
> kafka spout but instead wrote a simple one ourselves.
>
> Using the kafka AdminClient I was able to poll for consumer group summary
> information. What I've found is that the group seems to sit
> in PreparingRebalance state for minutes before finally becoming Stable
> which then everything starts processing quickly. I've also enabled debug
> logging around the consumer's coordinator classes but didn't see anything
> to indicate the issue.
>
> I'm hoping that just upgrading to 0.10 or tweaking how we use our consumer
> in Apache Storm is the problem but are there any pointers on things I
> should look at or try?
>
> Bryan
>
>
Re: Slow Consumer Group Startup
Posted by J Pai <ja...@gmail.com>.
When are the topics on which these consumer groups consume, created?
-Jaikiran
On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bj...@gmail.com> wrote:
Hi everyone,
We are currently experiencing slow startup times for our consumer groups
(16-32 processes for a hundred or more partitions) in the range of minutes
(3-15 minutes), where little to no messages are consumed before suddenly
everything just starts working at full speed.
I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading to
Kafka 0.10.2.1. We also using the newer kafka consumer API and group
management on a simple Apache Storm topology. We don't make use of Storm's
kafka spout but instead wrote a simple one ourselves.
Using the kafka AdminClient I was able to poll for consumer group summary
information. What I've found is that the group seems to sit
in PreparingRebalance state for minutes before finally becoming Stable
which then everything starts processing quickly. I've also enabled debug
logging around the consumer's coordinator classes but didn't see anything
to indicate the issue.
I'm hoping that just upgrading to 0.10 or tweaking how we use our consumer
in Apache Storm is the problem but are there any pointers on things I
should look at or try?
Bryan