You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Łukasz Antoniak <lu...@gmail.com> on 2020/04/14 19:30:45 UTC

Preferred Partition Leaders

Hi Everyone,

Recently I came across Kafka setup where two data centers are close to each
other, but the company could not find a suitable place for the third one.
As a result third DC is little further, lower network throughput, but still
within range of decent network latency, qualifying for stretch cluster. Let
us assume that client applications are being deployed only on two "primary"
DCs. My idea was to minimize network traffic between DC3 and other data
centers (ideally only to replication).

For Kafka consumer, we can configure rack-awareness, so that consumers will
read data from closest replica (replica.selector.class).
Kafka producers have to send data to partition leaders. There is no way to
tell that we prefer replica leaders to be running in DC1 and DC2. Kafka
will also try to evenly balance leaders across brokers
(auto.leader.rebalance.enable).

Does it sound like a good feature to make the choice of partition leaders
pluggable? Basically, users would be given list of topic-partitions with
ISRs and rack they are running, and could reshuffle them according to
custom logic.

Comments appreciated.

Kind regards,
Lukasz

Re: Preferred Partition Leaders

Posted by Sönke Liebau <so...@opencore.com.INVALID>.
Hi Lukasz,

that sounds like a worthwhile addition to me! Just a few thoughts that
occurred to me while reading your mail
"Preferred Leader" as a term is somewhat taken in the Kafka namespace. It
refers to the concept that the first leader from the list of replicas is
the preferred leader and will be chosen as the leader for that partition if
possible. Quite similar to what you are proposing - but without the logic
behind it, as currently that preferred leader is chosen mostly at random.

If your need for this is urgent, you could fairly easily use the
AdminClient to generate a new partition assignment that takes into account
the rack ids of the brokers already, and Kafka would try to honor that
preferred leader for your partitions. But this would only work
retrospectively, not for new topics, for those a random distribution would
again be chosen.

Regarding your idea, I agree that replica and leader assignment is a topic
that is in need of some love. However, by "just" adding the rack id to this
logic we run the risk of making potential future work for things like load
or size based replica assignment to brokers / disks harder. I'm not saying
we need to do it now, but I think we should consider what it might look
like to ensure that your solution can lay to groundwork for later work to
build on.

Best regards,
Sönke




On Tue, 14 Apr 2020 at 23:32, Michael K. Edwards <m....@gmail.com>
wrote:

> #nailedit
>
> On Tue, Apr 14, 2020 at 2:05 PM Jamie <ja...@aol.co.uk> wrote:
>
> > Hi All,
> >
> > There is a KIP which might be of interest to you:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120736982
> -
> > it sounds like you want to blacklist brokers in DC3?
> >
> > Thanks,
> >
> > Jamie
> >
> > -----Original Message-----
> > From: Michael K. Edwards <m....@gmail.com>
> > To: dev@kafka.apache.org
> > Sent: Tue, 14 Apr 2020 20:50
> > Subject: Re: Preferred Partition Leaders
> >
> > I have clients with a similar need relating to disaster recovery.  (Three
> > replicas per partition within a data center / AWS AZ/region, fourth
> replica
> > elsewhere, ineligible to become the partition leader without manual
> > intervention.)
> >
> > On Tue, Apr 14, 2020 at 12:31 PM Łukasz Antoniak <
> > lukasz.antoniak@gmail.com>
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > Recently I came across Kafka setup where two data centers are close to
> > each
> > > other, but the company could not find a suitable place for the third
> one.
> > > As a result third DC is little further, lower network throughput, but
> > still
> > > within range of decent network latency, qualifying for stretch cluster.
> > Let
> > > us assume that client applications are being deployed only on two
> > "primary"
> > > DCs. My idea was to minimize network traffic between DC3 and other data
> > > centers (ideally only to replication).
> > >
> > > For Kafka consumer, we can configure rack-awareness, so that consumers
> > will
> > > read data from closest replica (replica.selector.class).
> > > Kafka producers have to send data to partition leaders. There is no way
> > to
> > > tell that we prefer replica leaders to be running in DC1 and DC2. Kafka
> > > will also try to evenly balance leaders across brokers
> > > (auto.leader.rebalance.enable).
> > >
> > > Does it sound like a good feature to make the choice of partition
> leaders
> > > pluggable? Basically, users would be given list of topic-partitions
> with
> > > ISRs and rack they are running, and could reshuffle them according to
> > > custom logic.
> > >
> > > Comments appreciated.
> > >
> > > Kind regards,
> > > Lukasz
> > >
> >
>


-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: Preferred Partition Leaders

Posted by "Michael K. Edwards" <m....@gmail.com>.
#nailedit

On Tue, Apr 14, 2020 at 2:05 PM Jamie <ja...@aol.co.uk> wrote:

> Hi All,
>
> There is a KIP which might be of interest to you:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120736982 -
> it sounds like you want to blacklist brokers in DC3?
>
> Thanks,
>
> Jamie
>
> -----Original Message-----
> From: Michael K. Edwards <m....@gmail.com>
> To: dev@kafka.apache.org
> Sent: Tue, 14 Apr 2020 20:50
> Subject: Re: Preferred Partition Leaders
>
> I have clients with a similar need relating to disaster recovery.  (Three
> replicas per partition within a data center / AWS AZ/region, fourth replica
> elsewhere, ineligible to become the partition leader without manual
> intervention.)
>
> On Tue, Apr 14, 2020 at 12:31 PM Łukasz Antoniak <
> lukasz.antoniak@gmail.com>
> wrote:
>
> > Hi Everyone,
> >
> > Recently I came across Kafka setup where two data centers are close to
> each
> > other, but the company could not find a suitable place for the third one.
> > As a result third DC is little further, lower network throughput, but
> still
> > within range of decent network latency, qualifying for stretch cluster.
> Let
> > us assume that client applications are being deployed only on two
> "primary"
> > DCs. My idea was to minimize network traffic between DC3 and other data
> > centers (ideally only to replication).
> >
> > For Kafka consumer, we can configure rack-awareness, so that consumers
> will
> > read data from closest replica (replica.selector.class).
> > Kafka producers have to send data to partition leaders. There is no way
> to
> > tell that we prefer replica leaders to be running in DC1 and DC2. Kafka
> > will also try to evenly balance leaders across brokers
> > (auto.leader.rebalance.enable).
> >
> > Does it sound like a good feature to make the choice of partition leaders
> > pluggable? Basically, users would be given list of topic-partitions with
> > ISRs and rack they are running, and could reshuffle them according to
> > custom logic.
> >
> > Comments appreciated.
> >
> > Kind regards,
> > Lukasz
> >
>

Re: Preferred Partition Leaders

Posted by Jamie <ja...@aol.co.uk.INVALID>.
Hi All, 
There is a KIP which might be of interest to you: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120736982 - it sounds like you want to blacklist brokers in DC3? 

Thanks, 
Jamie

-----Original Message-----
From: Michael K. Edwards <m....@gmail.com>
To: dev@kafka.apache.org
Sent: Tue, 14 Apr 2020 20:50
Subject: Re: Preferred Partition Leaders

I have clients with a similar need relating to disaster recovery.  (Three
replicas per partition within a data center / AWS AZ/region, fourth replica
elsewhere, ineligible to become the partition leader without manual
intervention.)

On Tue, Apr 14, 2020 at 12:31 PM Łukasz Antoniak <lu...@gmail.com>
wrote:

> Hi Everyone,
>
> Recently I came across Kafka setup where two data centers are close to each
> other, but the company could not find a suitable place for the third one.
> As a result third DC is little further, lower network throughput, but still
> within range of decent network latency, qualifying for stretch cluster. Let
> us assume that client applications are being deployed only on two "primary"
> DCs. My idea was to minimize network traffic between DC3 and other data
> centers (ideally only to replication).
>
> For Kafka consumer, we can configure rack-awareness, so that consumers will
> read data from closest replica (replica.selector.class).
> Kafka producers have to send data to partition leaders. There is no way to
> tell that we prefer replica leaders to be running in DC1 and DC2. Kafka
> will also try to evenly balance leaders across brokers
> (auto.leader.rebalance.enable).
>
> Does it sound like a good feature to make the choice of partition leaders
> pluggable? Basically, users would be given list of topic-partitions with
> ISRs and rack they are running, and could reshuffle them according to
> custom logic.
>
> Comments appreciated.
>
> Kind regards,
> Lukasz
>

Re: Preferred Partition Leaders

Posted by "Michael K. Edwards" <m....@gmail.com>.
I have clients with a similar need relating to disaster recovery.  (Three
replicas per partition within a data center / AWS AZ/region, fourth replica
elsewhere, ineligible to become the partition leader without manual
intervention.)

On Tue, Apr 14, 2020 at 12:31 PM Łukasz Antoniak <lu...@gmail.com>
wrote:

> Hi Everyone,
>
> Recently I came across Kafka setup where two data centers are close to each
> other, but the company could not find a suitable place for the third one.
> As a result third DC is little further, lower network throughput, but still
> within range of decent network latency, qualifying for stretch cluster. Let
> us assume that client applications are being deployed only on two "primary"
> DCs. My idea was to minimize network traffic between DC3 and other data
> centers (ideally only to replication).
>
> For Kafka consumer, we can configure rack-awareness, so that consumers will
> read data from closest replica (replica.selector.class).
> Kafka producers have to send data to partition leaders. There is no way to
> tell that we prefer replica leaders to be running in DC1 and DC2. Kafka
> will also try to evenly balance leaders across brokers
> (auto.leader.rebalance.enable).
>
> Does it sound like a good feature to make the choice of partition leaders
> pluggable? Basically, users would be given list of topic-partitions with
> ISRs and rack they are running, and could reshuffle them according to
> custom logic.
>
> Comments appreciated.
>
> Kind regards,
> Lukasz
>