You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Liam Clarke-Hutchinson <li...@adscale.co.nz> on 2021/04/12 14:19:19 UTC
Kafka Definitive guide v2 states auto.leader.rebalance.enable = true
is not recommended
Hi all,
This question arose elsewhere, and I'm also going to fire it off to
O'Reilly in the hopes that they'll clarify, but on page 180 of the
Definitive Guide v2
<https://assets.confluent.io/m/2849a76e39cda2bd/original/20201119-EB-Kafka_The_Definitive_Guide-Preview-Chapters_1_thru_6.pdf>
it
states:
*Kafka brokers do not automatically take partition leadership back (unless
auto leader rebalance is enabled, but this configuration is not
recommended)*
The original commenter raised the point that this defaults to true, and it
sounds like a good idea to have auto leader rebalancing.
So I'm curious, in anyone's war stories or experiences, has this property
being enabled been harmful? From the context that the paragraph was written
in, I'm assuming the writers were perhaps intending to emphasise the Cruise
Control or Confluents self-balancing-cluster / auto-balancing features were
preferable, but in my very brief Google didn't see any advice to set
auto.leader.rebalance.enabled to false to use those tools.
So yeah, just curious if this rings any bells.
Cheers,
Liam Clarke-Hutchinson
Re: Kafka Definitive guide v2 states auto.leader.rebalance.enable =
true is not recommended
Posted by Alexandre Dupriez <al...@gmail.com>.
Hi Liam,
The property you referred to corresponds to partition leadership, not
ownership from consumers. See
https://issues.apache.org/jira/browse/KAFKA-4084 for a discussion
about why post-incident leader rebalance can sometimes impact
foreground traffic.
Thanks,
Alexandre
Le lun. 12 avr. 2021 à 15:37, Liam Clarke-Hutchinson
<li...@adscale.co.nz> a écrit :
>
> Ah, thanks Todd :)
>
> Was it causing issues back in the day of consumer rebalances always being
> stop the world? I was wondering if the statement had perhaps predated the
> cooperative / stick assignors we're able to run now.
>
> Cheers,
>
> Liam Clarke-Hutchinson
>
>
>
> On Tue, Apr 13, 2021 at 2:34 AM Todd Palino <tp...@gmail.com> wrote:
>
> > As a note, that part of the second edition has not been updated yet. This
> > setting used to cause significant problems, but more recent updates to the
> > controller code have made the auto leader rebalancing usable.
> >
> > -Todd
> >
> > On Mon, Apr 12, 2021 at 10:20 AM Liam Clarke-Hutchinson <
> > liam.clarke@adscale.co.nz> wrote:
> >
> > > Hi all,
> > >
> > > This question arose elsewhere, and I'm also going to fire it off to
> > > O'Reilly in the hopes that they'll clarify, but on page 180 of the
> > > Definitive Guide v2
> > > <
> > >
> > https://assets.confluent.io/m/2849a76e39cda2bd/original/20201119-EB-Kafka_The_Definitive_Guide-Preview-Chapters_1_thru_6.pdf
> > > >
> > > it
> > > states:
> > >
> > > *Kafka brokers do not automatically take partition leadership back
> > (unless
> > > auto leader rebalance is enabled, but this configuration is not
> > > recommended)*
> > >
> > > The original commenter raised the point that this defaults to true, and
> > it
> > > sounds like a good idea to have auto leader rebalancing.
> > >
> > > So I'm curious, in anyone's war stories or experiences, has this property
> > > being enabled been harmful? From the context that the paragraph was
> > written
> > > in, I'm assuming the writers were perhaps intending to emphasise the
> > Cruise
> > > Control or Confluents self-balancing-cluster / auto-balancing features
> > were
> > > preferable, but in my very brief Google didn't see any advice to set
> > > auto.leader.rebalance.enabled to false to use those tools.
> > >
> > > So yeah, just curious if this rings any bells.
> > >
> > > Cheers,
> > >
> > > Liam Clarke-Hutchinson
> > >
> > --
> > *Todd Palino*
> > Senior Staff Engineer, Site Reliability
> > Capacity Engineering
> >
> >
> >
> > linkedin.com/in/toddpalino
> >
Re: Kafka Definitive guide v2 states auto.leader.rebalance.enable =
true is not recommended
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
Ah, thanks Todd :)
Was it causing issues back in the day of consumer rebalances always being
stop the world? I was wondering if the statement had perhaps predated the
cooperative / stick assignors we're able to run now.
Cheers,
Liam Clarke-Hutchinson
On Tue, Apr 13, 2021 at 2:34 AM Todd Palino <tp...@gmail.com> wrote:
> As a note, that part of the second edition has not been updated yet. This
> setting used to cause significant problems, but more recent updates to the
> controller code have made the auto leader rebalancing usable.
>
> -Todd
>
> On Mon, Apr 12, 2021 at 10:20 AM Liam Clarke-Hutchinson <
> liam.clarke@adscale.co.nz> wrote:
>
> > Hi all,
> >
> > This question arose elsewhere, and I'm also going to fire it off to
> > O'Reilly in the hopes that they'll clarify, but on page 180 of the
> > Definitive Guide v2
> > <
> >
> https://assets.confluent.io/m/2849a76e39cda2bd/original/20201119-EB-Kafka_The_Definitive_Guide-Preview-Chapters_1_thru_6.pdf
> > >
> > it
> > states:
> >
> > *Kafka brokers do not automatically take partition leadership back
> (unless
> > auto leader rebalance is enabled, but this configuration is not
> > recommended)*
> >
> > The original commenter raised the point that this defaults to true, and
> it
> > sounds like a good idea to have auto leader rebalancing.
> >
> > So I'm curious, in anyone's war stories or experiences, has this property
> > being enabled been harmful? From the context that the paragraph was
> written
> > in, I'm assuming the writers were perhaps intending to emphasise the
> Cruise
> > Control or Confluents self-balancing-cluster / auto-balancing features
> were
> > preferable, but in my very brief Google didn't see any advice to set
> > auto.leader.rebalance.enabled to false to use those tools.
> >
> > So yeah, just curious if this rings any bells.
> >
> > Cheers,
> >
> > Liam Clarke-Hutchinson
> >
> --
> *Todd Palino*
> Senior Staff Engineer, Site Reliability
> Capacity Engineering
>
>
>
> linkedin.com/in/toddpalino
>
Re: Kafka Definitive guide v2 states auto.leader.rebalance.enable =
true is not recommended
Posted by Todd Palino <tp...@gmail.com>.
As a note, that part of the second edition has not been updated yet. This
setting used to cause significant problems, but more recent updates to the
controller code have made the auto leader rebalancing usable.
-Todd
On Mon, Apr 12, 2021 at 10:20 AM Liam Clarke-Hutchinson <
liam.clarke@adscale.co.nz> wrote:
> Hi all,
>
> This question arose elsewhere, and I'm also going to fire it off to
> O'Reilly in the hopes that they'll clarify, but on page 180 of the
> Definitive Guide v2
> <
> https://assets.confluent.io/m/2849a76e39cda2bd/original/20201119-EB-Kafka_The_Definitive_Guide-Preview-Chapters_1_thru_6.pdf
> >
> it
> states:
>
> *Kafka brokers do not automatically take partition leadership back (unless
> auto leader rebalance is enabled, but this configuration is not
> recommended)*
>
> The original commenter raised the point that this defaults to true, and it
> sounds like a good idea to have auto leader rebalancing.
>
> So I'm curious, in anyone's war stories or experiences, has this property
> being enabled been harmful? From the context that the paragraph was written
> in, I'm assuming the writers were perhaps intending to emphasise the Cruise
> Control or Confluents self-balancing-cluster / auto-balancing features were
> preferable, but in my very brief Google didn't see any advice to set
> auto.leader.rebalance.enabled to false to use those tools.
>
> So yeah, just curious if this rings any bells.
>
> Cheers,
>
> Liam Clarke-Hutchinson
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Capacity Engineering
linkedin.com/in/toddpalino