LeaderElectionNotNeededException

[Liam Clarke-Hutchinson <li...@adscale.co.nz>]

Kia ora,

I am rather bemused by this one, I'll admit. Kafka version 2.4.0. We've
been migrating topic-partitions from old broker machines to new brokers,
have redistributed replicas evenly across the new brokers, removed the old
brokers, and tried to run a preferred leader election.

And when I did that, I got a "LeaderElectionNotNeededException", which I
really disagree with, as two brokers out of six have 41% of partition
leaders.

I wonder if it's rack awareness messing with us or something? We are
running two brokers in three AZs/racks with correctly configured rack.ids,
but what are the origins of a "LeaderElectionNotNeededException"? And for
the love of God, where is the parameter in the admin client for "I
significantly disagree".

Kind regards,

Liam Clarke-Hutchinson

Re: LeaderElectionNotNeededException

[Liam Clarke-Hutchinson <li...@adscale.co.nz>]

Hi Tom,

You're right, I had had a complete brain fade on this.

Turns out that the script I wrote had a logical error - while trying to
ensure every zone had a replica, it went through zones in this same order,
thus unintentionally ensuring that the two brokers in zone A were quite
often the first replica in the list, hence preferred leader for 80% of the
partitions...

...and I think this incident has very much sold my team on Cruise Control.

Thanks for the reply :)

Liam Clarke


On Tue, 20 Oct. 2020, 11:49 pm Tom Bentley, <tb...@redhat.com> wrote:

> Hi Liam,
>
> I think you have a misunderstanding of what preferred leader election does.
> All it does is ensure that the "preferred leader" (the first in the list of
> replicas for a partition) becomes the actual leader if it can (that is, if
> it's in the ISR) and if the current leader is already the preferred leader
> you get ElectionNotNeededException. It *doesn't* change the brokers which
> are replicating a partition, or change which broker is the preferred
> leader. So if the broker assignments are unbalanced, or ignore rack
> constraints it doesn't help.
>
> Obviously you can use kafka-reassign-partitions.sh to change how partitions
> are assigned to brokers, including which is the preferred leader, in order
> to ensure the cluster is better balanced. But it's up to you to figure out
> how to assign replicas to the brokers to achieve that (i.e. for each
> partition come up with the list of brokers which should have a replica,
> with the preferred one being first in the list). As you note with the
> racks, in general there are several criteria to consider and finding a good
> assignment is an exercise in constrained optimization. So people often use
> external tools to find "good" assignments according to their criteria, and
> often to manage the reassignment process too. There are several open source
> tools available, such as Cruise Control.
>
> Hope this helps,
>
> Tom
>
> On Fri, Oct 16, 2020 at 7:05 AM Liam Clarke-Hutchinson <
> liam.clarke@adscale.co.nz> wrote:
>
> > And to follow up on this, we rolled one of the 41% brokers, partition
> > leadership was redistributed to other brokers, but a preferred leader
> > election immediately led to the exact same distribution.
> >
> > What's worse, is both of the 41% leadership brokers are in the same rack.
> >
> > Very keen for advice, or in the worst case, recommendations on how to
> > manually elect leaders for a topic-partition to get us through the
> weekend.
> >
> > Kind regards,
> >
> > Liam
> >
> > On Fri, 16 Oct. 2020, 6:22 pm Liam Clarke-Hutchinson, <
> > liam.clarke@adscale.co.nz> wrote:
> >
> > > Kia ora,
> > >
> > > I am rather bemused by this one, I'll admit. Kafka version 2.4.0. We've
> > > been migrating topic-partitions from old broker machines to new
> brokers,
> > > have redistributed replicas evenly across the new brokers, removed the
> > old
> > > brokers, and tried to run a preferred leader election.
> > >
> > > And when I did that, I got a "LeaderElectionNotNeededException", which
> I
> > > really disagree with, as two brokers out of six have 41% of partition
> > > leaders.
> > >
> > > I wonder if it's rack awareness messing with us or something? We are
> > > running two brokers in three AZs/racks with correctly configured
> > rack.ids,
> > > but what are the origins of a "LeaderElectionNotNeededException"? And
> for
> > > the love of God, where is the parameter in the admin client for "I
> > > significantly disagree".
> > >
> > > Kind regards,
> > >
> > > Liam Clarke-Hutchinson
> > >
> >
>

Re: LeaderElectionNotNeededException

[Tom Bentley <tb...@redhat.com>]

Hi Liam,

I think you have a misunderstanding of what preferred leader election does.
All it does is ensure that the "preferred leader" (the first in the list of
replicas for a partition) becomes the actual leader if it can (that is, if
it's in the ISR) and if the current leader is already the preferred leader
you get ElectionNotNeededException. It *doesn't* change the brokers which
are replicating a partition, or change which broker is the preferred
leader. So if the broker assignments are unbalanced, or ignore rack
constraints it doesn't help.

Obviously you can use kafka-reassign-partitions.sh to change how partitions
are assigned to brokers, including which is the preferred leader, in order
to ensure the cluster is better balanced. But it's up to you to figure out
how to assign replicas to the brokers to achieve that (i.e. for each
partition come up with the list of brokers which should have a replica,
with the preferred one being first in the list). As you note with the
racks, in general there are several criteria to consider and finding a good
assignment is an exercise in constrained optimization. So people often use
external tools to find "good" assignments according to their criteria, and
often to manage the reassignment process too. There are several open source
tools available, such as Cruise Control.

Hope this helps,

Tom

On Fri, Oct 16, 2020 at 7:05 AM Liam Clarke-Hutchinson <
liam.clarke@adscale.co.nz> wrote:

> And to follow up on this, we rolled one of the 41% brokers, partition
> leadership was redistributed to other brokers, but a preferred leader
> election immediately led to the exact same distribution.
>
> What's worse, is both of the 41% leadership brokers are in the same rack.
>
> Very keen for advice, or in the worst case, recommendations on how to
> manually elect leaders for a topic-partition to get us through the weekend.
>
> Kind regards,
>
> Liam
>
> On Fri, 16 Oct. 2020, 6:22 pm Liam Clarke-Hutchinson, <
> liam.clarke@adscale.co.nz> wrote:
>
> > Kia ora,
> >
> > I am rather bemused by this one, I'll admit. Kafka version 2.4.0. We've
> > been migrating topic-partitions from old broker machines to new brokers,
> > have redistributed replicas evenly across the new brokers, removed the
> old
> > brokers, and tried to run a preferred leader election.
> >
> > And when I did that, I got a "LeaderElectionNotNeededException", which I
> > really disagree with, as two brokers out of six have 41% of partition
> > leaders.
> >
> > I wonder if it's rack awareness messing with us or something? We are
> > running two brokers in three AZs/racks with correctly configured
> rack.ids,
> > but what are the origins of a "LeaderElectionNotNeededException"? And for
> > the love of God, where is the parameter in the admin client for "I
> > significantly disagree".
> >
> > Kind regards,
> >
> > Liam Clarke-Hutchinson
> >
>

Re: LeaderElectionNotNeededException

[Liam Clarke-Hutchinson <li...@adscale.co.nz>]

And to follow up on this, we rolled one of the 41% brokers, partition
leadership was redistributed to other brokers, but a preferred leader
election immediately led to the exact same distribution.

What's worse, is both of the 41% leadership brokers are in the same rack.

Very keen for advice, or in the worst case, recommendations on how to
manually elect leaders for a topic-partition to get us through the weekend.

Kind regards,

Liam

On Fri, 16 Oct. 2020, 6:22 pm Liam Clarke-Hutchinson, <
liam.clarke@adscale.co.nz> wrote:

> Kia ora,
>
> I am rather bemused by this one, I'll admit. Kafka version 2.4.0. We've
> been migrating topic-partitions from old broker machines to new brokers,
> have redistributed replicas evenly across the new brokers, removed the old
> brokers, and tried to run a preferred leader election.
>
> And when I did that, I got a "LeaderElectionNotNeededException", which I
> really disagree with, as two brokers out of six have 41% of partition
> leaders.
>
> I wonder if it's rack awareness messing with us or something? We are
> running two brokers in three AZs/racks with correctly configured rack.ids,
> but what are the origins of a "LeaderElectionNotNeededException"? And for
> the love of God, where is the parameter in the admin client for "I
> significantly disagree".
>
> Kind regards,
>
> Liam Clarke-Hutchinson
>