You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Yu, Libo " <li...@citi.com> on 2014/02/06 18:05:51 UTC
Consumer's behavior when brokers are temporarily not available.
Hi folks,
This is what we experienced recently:
Some zookeeper's issue made broker unavailable for a short period of time.
On the consumer side, this triggered rebalance and rebalanced failed after
four tries.
So while should we expect while the broker is not up? Should consumer keep
trying to rebalance or wait for the brokers to be brought back? Thanks.
Regards,
Libo
RE: Consumer's behavior when brokers are temporarily not available.
Posted by "Yu, Libo " <li...@citi.com>.
Thanks, Neha. We are planning to migrate to 0.8.1.
Regards,
Libo
-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Friday, February 07, 2014 10:26 AM
To: users@kafka.apache.org
Subject: RE: Consumer's behavior when brokers are temporarily not available.
Yes bouncing the process will allow you to consume again. Also would you mind giving 0.8 final a try? It is much more stable compared to 0.8 beta.
Thanks,
Neha
On Feb 7, 2014 6:49 AM, "Yu, Libo" <li...@citi.com> wrote:
> We are using 0.8 beta1. Our zookeeper had some issue which in turn
> triggered consumer rebalance.
> In this case, after maximum number of retries, the rebalance failed,
> should we bounce our process in order to consume again?
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, February 07, 2014 12:03 AM
> To: users@kafka.apache.org
> Subject: Re: Consumer's behavior when brokers are temporarily not
> available.
>
> Which version are you using? In 0.8, broker failure won't trigger
> consumer rebalances. Only changes in #partitions and consumers will
> trigger rebalances.
>
> Thanks,
>
> Jun
>
>
> On Thu, Feb 6, 2014 at 10:46 AM, Yu, Libo <li...@citi.com> wrote:
>
> > While the broker is not available (caused by zookeeper issue), the
> > rebalance will fail. Should rebalance succeed in this case? Thanks.
> >
> >
> > Regards,
> >
> > Libo
> >
> >
> > -----Original Message-----
> > From: Guozhang Wang [mailto:wangguoz@gmail.com]
> > Sent: Thursday, February 06, 2014 12:49 PM
> > To: users@kafka.apache.org
> > Subject: Re: Consumer's behavior when brokers are temporarily not
> > available.
> >
> > Neha is right, though it is a little interesting that consumers did
> > not achieve a concensus after four retries have exhasuted, for just
> > an event of broker-failure.
> >
> > Could you check your consumer log searching for "begin rebalance"
> > and check what caused these rebalances to fail?
> >
> > Guozhang
> >
> >
> > On Thu, Feb 6, 2014 at 9:12 AM, Neha Narkhede
> > <neha.narkhede@gmail.com
> > >wrote:
> >
> > > The consumer only retries "rebalance.max.retries" times. Once it
> > > runs out of the retries, it needs to be restarted to consume again.
> > >
> > >
> > >
> > >
> > > On Thu, Feb 6, 2014 at 9:05 AM, Yu, Libo <li...@citi.com> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > This is what we experienced recently:
> > > > Some zookeeper's issue made broker unavailable for a short
> > > > period of
> > > time.
> > > > On the consumer side, this triggered rebalance and rebalanced
> > > > failed
> > > after
> > > > four tries.
> > > > So while should we expect while the broker is not up? Should
> > > > consumer
> > > keep
> > > > trying to rebalance or wait for the brokers to be brought back?
> > Thanks.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Libo
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
RE: Consumer's behavior when brokers are temporarily not available.
Posted by Neha Narkhede <ne...@gmail.com>.
Yes bouncing the process will allow you to consume again. Also would you
mind giving 0.8 final a try? It is much more stable compared to 0.8 beta.
Thanks,
Neha
On Feb 7, 2014 6:49 AM, "Yu, Libo" <li...@citi.com> wrote:
> We are using 0.8 beta1. Our zookeeper had some issue which in turn
> triggered consumer rebalance.
> In this case, after maximum number of retries, the rebalance failed,
> should we bounce our process
> in order to consume again?
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, February 07, 2014 12:03 AM
> To: users@kafka.apache.org
> Subject: Re: Consumer's behavior when brokers are temporarily not
> available.
>
> Which version are you using? In 0.8, broker failure won't trigger consumer
> rebalances. Only changes in #partitions and consumers will trigger
> rebalances.
>
> Thanks,
>
> Jun
>
>
> On Thu, Feb 6, 2014 at 10:46 AM, Yu, Libo <li...@citi.com> wrote:
>
> > While the broker is not available (caused by zookeeper issue), the
> > rebalance will fail. Should rebalance succeed in this case? Thanks.
> >
> >
> > Regards,
> >
> > Libo
> >
> >
> > -----Original Message-----
> > From: Guozhang Wang [mailto:wangguoz@gmail.com]
> > Sent: Thursday, February 06, 2014 12:49 PM
> > To: users@kafka.apache.org
> > Subject: Re: Consumer's behavior when brokers are temporarily not
> > available.
> >
> > Neha is right, though it is a little interesting that consumers did
> > not achieve a concensus after four retries have exhasuted, for just an
> > event of broker-failure.
> >
> > Could you check your consumer log searching for "begin rebalance" and
> > check what caused these rebalances to fail?
> >
> > Guozhang
> >
> >
> > On Thu, Feb 6, 2014 at 9:12 AM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > The consumer only retries "rebalance.max.retries" times. Once it
> > > runs out of the retries, it needs to be restarted to consume again.
> > >
> > >
> > >
> > >
> > > On Thu, Feb 6, 2014 at 9:05 AM, Yu, Libo <li...@citi.com> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > This is what we experienced recently:
> > > > Some zookeeper's issue made broker unavailable for a short period
> > > > of
> > > time.
> > > > On the consumer side, this triggered rebalance and rebalanced
> > > > failed
> > > after
> > > > four tries.
> > > > So while should we expect while the broker is not up? Should
> > > > consumer
> > > keep
> > > > trying to rebalance or wait for the brokers to be brought back?
> > Thanks.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Libo
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
RE: Consumer's behavior when brokers are temporarily not available.
Posted by "Yu, Libo " <li...@citi.com>.
We are using 0.8 beta1. Our zookeeper had some issue which in turn triggered consumer rebalance.
In this case, after maximum number of retries, the rebalance failed, should we bounce our process
in order to consume again?
Regards,
Libo
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Friday, February 07, 2014 12:03 AM
To: users@kafka.apache.org
Subject: Re: Consumer's behavior when brokers are temporarily not available.
Which version are you using? In 0.8, broker failure won't trigger consumer rebalances. Only changes in #partitions and consumers will trigger rebalances.
Thanks,
Jun
On Thu, Feb 6, 2014 at 10:46 AM, Yu, Libo <li...@citi.com> wrote:
> While the broker is not available (caused by zookeeper issue), the
> rebalance will fail. Should rebalance succeed in this case? Thanks.
>
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Guozhang Wang [mailto:wangguoz@gmail.com]
> Sent: Thursday, February 06, 2014 12:49 PM
> To: users@kafka.apache.org
> Subject: Re: Consumer's behavior when brokers are temporarily not
> available.
>
> Neha is right, though it is a little interesting that consumers did
> not achieve a concensus after four retries have exhasuted, for just an
> event of broker-failure.
>
> Could you check your consumer log searching for "begin rebalance" and
> check what caused these rebalances to fail?
>
> Guozhang
>
>
> On Thu, Feb 6, 2014 at 9:12 AM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > The consumer only retries "rebalance.max.retries" times. Once it
> > runs out of the retries, it needs to be restarted to consume again.
> >
> >
> >
> >
> > On Thu, Feb 6, 2014 at 9:05 AM, Yu, Libo <li...@citi.com> wrote:
> >
> > > Hi folks,
> > >
> > > This is what we experienced recently:
> > > Some zookeeper's issue made broker unavailable for a short period
> > > of
> > time.
> > > On the consumer side, this triggered rebalance and rebalanced
> > > failed
> > after
> > > four tries.
> > > So while should we expect while the broker is not up? Should
> > > consumer
> > keep
> > > trying to rebalance or wait for the brokers to be brought back?
> Thanks.
> > >
> > >
> > >
> > >
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Consumer's behavior when brokers are temporarily not available.
Posted by Jun Rao <ju...@gmail.com>.
Which version are you using? In 0.8, broker failure won't trigger consumer
rebalances. Only changes in #partitions and consumers will trigger
rebalances.
Thanks,
Jun
On Thu, Feb 6, 2014 at 10:46 AM, Yu, Libo <li...@citi.com> wrote:
> While the broker is not available (caused by zookeeper issue), the
> rebalance will fail. Should
> rebalance succeed in this case? Thanks.
>
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Guozhang Wang [mailto:wangguoz@gmail.com]
> Sent: Thursday, February 06, 2014 12:49 PM
> To: users@kafka.apache.org
> Subject: Re: Consumer's behavior when brokers are temporarily not
> available.
>
> Neha is right, though it is a little interesting that consumers did not
> achieve a concensus after four retries have exhasuted, for just an event of
> broker-failure.
>
> Could you check your consumer log searching for "begin rebalance" and
> check what caused these rebalances to fail?
>
> Guozhang
>
>
> On Thu, Feb 6, 2014 at 9:12 AM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > The consumer only retries "rebalance.max.retries" times. Once it runs
> > out of the retries, it needs to be restarted to consume again.
> >
> >
> >
> >
> > On Thu, Feb 6, 2014 at 9:05 AM, Yu, Libo <li...@citi.com> wrote:
> >
> > > Hi folks,
> > >
> > > This is what we experienced recently:
> > > Some zookeeper's issue made broker unavailable for a short period of
> > time.
> > > On the consumer side, this triggered rebalance and rebalanced failed
> > after
> > > four tries.
> > > So while should we expect while the broker is not up? Should
> > > consumer
> > keep
> > > trying to rebalance or wait for the brokers to be brought back?
> Thanks.
> > >
> > >
> > >
> > >
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> >
>
>
>
> --
> -- Guozhang
>
RE: Consumer's behavior when brokers are temporarily not available.
Posted by "Yu, Libo " <li...@citi.com>.
While the broker is not available (caused by zookeeper issue), the rebalance will fail. Should
rebalance succeed in this case? Thanks.
Regards,
Libo
-----Original Message-----
From: Guozhang Wang [mailto:wangguoz@gmail.com]
Sent: Thursday, February 06, 2014 12:49 PM
To: users@kafka.apache.org
Subject: Re: Consumer's behavior when brokers are temporarily not available.
Neha is right, though it is a little interesting that consumers did not achieve a concensus after four retries have exhasuted, for just an event of broker-failure.
Could you check your consumer log searching for "begin rebalance" and check what caused these rebalances to fail?
Guozhang
On Thu, Feb 6, 2014 at 9:12 AM, Neha Narkhede <ne...@gmail.com>wrote:
> The consumer only retries "rebalance.max.retries" times. Once it runs
> out of the retries, it needs to be restarted to consume again.
>
>
>
>
> On Thu, Feb 6, 2014 at 9:05 AM, Yu, Libo <li...@citi.com> wrote:
>
> > Hi folks,
> >
> > This is what we experienced recently:
> > Some zookeeper's issue made broker unavailable for a short period of
> time.
> > On the consumer side, this triggered rebalance and rebalanced failed
> after
> > four tries.
> > So while should we expect while the broker is not up? Should
> > consumer
> keep
> > trying to rebalance or wait for the brokers to be brought back? Thanks.
> >
> >
> >
> >
> >
> > Regards,
> >
> > Libo
> >
> >
>
--
-- Guozhang
Re: Consumer's behavior when brokers are temporarily not available.
Posted by Guozhang Wang <wa...@gmail.com>.
Neha is right, though it is a little interesting that consumers did not
achieve a concensus after four retries have exhasuted, for just an event of
broker-failure.
Could you check your consumer log searching for "begin rebalance" and check
what caused these rebalances to fail?
Guozhang
On Thu, Feb 6, 2014 at 9:12 AM, Neha Narkhede <ne...@gmail.com>wrote:
> The consumer only retries "rebalance.max.retries" times. Once it runs out
> of the retries, it needs to be restarted to consume again.
>
>
>
>
> On Thu, Feb 6, 2014 at 9:05 AM, Yu, Libo <li...@citi.com> wrote:
>
> > Hi folks,
> >
> > This is what we experienced recently:
> > Some zookeeper's issue made broker unavailable for a short period of
> time.
> > On the consumer side, this triggered rebalance and rebalanced failed
> after
> > four tries.
> > So while should we expect while the broker is not up? Should consumer
> keep
> > trying to rebalance or wait for the brokers to be brought back? Thanks.
> >
> >
> >
> >
> >
> > Regards,
> >
> > Libo
> >
> >
>
--
-- Guozhang
RE: Consumer's behavior when brokers are temporarily not available.
Posted by "Yu, Libo " <li...@citi.com>.
"Restarted", do you mean restart the process/recreate a consumer from scratch in order to consume again?
This is quite different from the answer I got previously from here. I have more questions here.
Is "rebalance.backoff.ms" the time interval between any consecutive retries before "rebalance.max.retries" is reached?
This is what I was told previously.
After "rebalance.max.retries" number of retries, the consumer would back off for "rebalance.backoff.ms" before retrying
again. Please clarify. Thanks.
Regards,
Libo
-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Thursday, February 06, 2014 12:12 PM
To: users@kafka.apache.org
Subject: Re: Consumer's behavior when brokers are temporarily not available.
The consumer only retries "rebalance.max.retries" times. Once it runs out of the retries, it needs to be restarted to consume again.
On Thu, Feb 6, 2014 at 9:05 AM, Yu, Libo <li...@citi.com> wrote:
> Hi folks,
>
> This is what we experienced recently:
> Some zookeeper's issue made broker unavailable for a short period of time.
> On the consumer side, this triggered rebalance and rebalanced failed
> after four tries.
> So while should we expect while the broker is not up? Should consumer
> keep trying to rebalance or wait for the brokers to be brought back? Thanks.
>
>
>
>
>
> Regards,
>
> Libo
>
>
Re: Consumer's behavior when brokers are temporarily not available.
Posted by Neha Narkhede <ne...@gmail.com>.
The consumer only retries "rebalance.max.retries" times. Once it runs out
of the retries, it needs to be restarted to consume again.
On Thu, Feb 6, 2014 at 9:05 AM, Yu, Libo <li...@citi.com> wrote:
> Hi folks,
>
> This is what we experienced recently:
> Some zookeeper's issue made broker unavailable for a short period of time.
> On the consumer side, this triggered rebalance and rebalanced failed after
> four tries.
> So while should we expect while the broker is not up? Should consumer keep
> trying to rebalance or wait for the brokers to be brought back? Thanks.
>
>
>
>
>
> Regards,
>
> Libo
>
>