You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Viktor Somogyi-Vass <vi...@cloudera.com.INVALID> on 2022/10/28 14:14:20 UTC
[DISCUSS] KIP-879: Multi-level Rack Awareness
Hey all,
I'd like to propose a new broker side replica assignment strategy and an
interface that generalizes replica assignment on brokers and makes them
pluggable.
Briefly, the motivation for the new replica assignment strategy is that
more and more of our customers would want to run their clusters in a
stretched environment, where for instance a cluster is running over
multiple regions (and multiple racks inside a region). Since this seems
like a more common need, we'd like to contribute back our implementation
and also make a generalized interface, so that new strategies that people
may come up with could be served better.
I welcome any feedback on this KIP.
The link:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness
Best to all,
Viktor
Re: [DISCUSS] KIP-879: Multi-level Rack Awareness
Posted by Viktor Somogyi-Vass <vi...@cloudera.com.INVALID>.
Hi Ziming,
Thanks for the reply. It may not be a public interface, yes, but it is
annotated with the @InterfaceStability.Unstable annotation from which I
presumed it'll be public at some point. Therefore in my KIP I proposed to
take it public as we've seen significant interest from our users towards
stretch clusters and this kind of replica placement I described. In fact in
our latest release we've already implemented something like this and we'd
like to contribute it back if the community also feels the need for it.
May I ask why don't you want to make this public?
Thanks,
Viktor
On Thu, Dec 8, 2022 at 3:06 PM Andrew Otto <ot...@wikimedia.org> wrote:
> FWIW, the Wikimedia Foundation would find this change really helpful. We
> are going to soon experiment with a stretched Kafka cluster, and it would
> be nice to be able to target datacenter AND racks for replica placement.
>
> On Thu, Dec 8, 2022 at 3:37 AM ziming deng <de...@gmail.com>
> wrote:
>
> > Hi Viktor,
> >
> > As far as I know, we haven't make ReplicaPlacer a public interface, and
> we
> > have no plan to make it public. I think you can submit a discussion or
> > create a JIRA ticket directly without KIP if you have ideas on improving
> > it, right?
> >
> > --
> > Best,
> > Ziming
> >
> > > On Nov 29, 2022, at 21:52, Viktor Somogyi-Vass <
> > viktor.somogyi@cloudera.com.INVALID> wrote:
> > >
> > > Hi All,
> > >
> > > I'd like to bump this. I've also updated the KIP to incorporate the new
> > > KRaft changes (ReplicaPlacer). Luckily my proposals were quite similar
> to
> > > that, so mostly I've made some minor rewording, naming changes, etc.
> > >
> > > Again, the brief summary of the KIP:
> > > - expose replica placement strategies with a new config
> > > - create an admin API and protocol to expose replica placement
> > > functionality (mainly for the reassignment tool)
> > > - create a new multi-level rack awareness strategy which improves
> > > availability on stretch clusters
> > >
> > > I'm happy for any feedback.
> > >
> > > Best,
> > > Viktor
> > >
> > > On Fri, Oct 28, 2022 at 4:14 PM Viktor Somogyi-Vass <
> > > viktor.somogyi@cloudera.com> wrote:
> > >
> > >> Hey all,
> > >>
> > >> I'd like to propose a new broker side replica assignment strategy and
> an
> > >> interface that generalizes replica assignment on brokers and makes
> them
> > >> pluggable.
> > >>
> > >> Briefly, the motivation for the new replica assignment strategy is
> that
> > >> more and more of our customers would want to run their clusters in a
> > >> stretched environment, where for instance a cluster is running over
> > >> multiple regions (and multiple racks inside a region). Since this
> seems
> > >> like a more common need, we'd like to contribute back our
> implementation
> > >> and also make a generalized interface, so that new strategies that
> > people
> > >> may come up with could be served better.
> > >>
> > >> I welcome any feedback on this KIP.
> > >>
> > >> The link:
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness
> > >>
> > >> Best to all,
> > >> Viktor
> > >>
> >
> >
>
Re: [DISCUSS] KIP-879: Multi-level Rack Awareness
Posted by Andrew Otto <ot...@wikimedia.org>.
FWIW, the Wikimedia Foundation would find this change really helpful. We
are going to soon experiment with a stretched Kafka cluster, and it would
be nice to be able to target datacenter AND racks for replica placement.
On Thu, Dec 8, 2022 at 3:37 AM ziming deng <de...@gmail.com> wrote:
> Hi Viktor,
>
> As far as I know, we haven't make ReplicaPlacer a public interface, and we
> have no plan to make it public. I think you can submit a discussion or
> create a JIRA ticket directly without KIP if you have ideas on improving
> it, right?
>
> --
> Best,
> Ziming
>
> > On Nov 29, 2022, at 21:52, Viktor Somogyi-Vass <
> viktor.somogyi@cloudera.com.INVALID> wrote:
> >
> > Hi All,
> >
> > I'd like to bump this. I've also updated the KIP to incorporate the new
> > KRaft changes (ReplicaPlacer). Luckily my proposals were quite similar to
> > that, so mostly I've made some minor rewording, naming changes, etc.
> >
> > Again, the brief summary of the KIP:
> > - expose replica placement strategies with a new config
> > - create an admin API and protocol to expose replica placement
> > functionality (mainly for the reassignment tool)
> > - create a new multi-level rack awareness strategy which improves
> > availability on stretch clusters
> >
> > I'm happy for any feedback.
> >
> > Best,
> > Viktor
> >
> > On Fri, Oct 28, 2022 at 4:14 PM Viktor Somogyi-Vass <
> > viktor.somogyi@cloudera.com> wrote:
> >
> >> Hey all,
> >>
> >> I'd like to propose a new broker side replica assignment strategy and an
> >> interface that generalizes replica assignment on brokers and makes them
> >> pluggable.
> >>
> >> Briefly, the motivation for the new replica assignment strategy is that
> >> more and more of our customers would want to run their clusters in a
> >> stretched environment, where for instance a cluster is running over
> >> multiple regions (and multiple racks inside a region). Since this seems
> >> like a more common need, we'd like to contribute back our implementation
> >> and also make a generalized interface, so that new strategies that
> people
> >> may come up with could be served better.
> >>
> >> I welcome any feedback on this KIP.
> >>
> >> The link:
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness
> >>
> >> Best to all,
> >> Viktor
> >>
>
>
Re: [DISCUSS] KIP-879: Multi-level Rack Awareness
Posted by ziming deng <de...@gmail.com>.
Hi Viktor,
As far as I know, we haven't make ReplicaPlacer a public interface, and we have no plan to make it public. I think you can submit a discussion or create a JIRA ticket directly without KIP if you have ideas on improving it, right?
--
Best,
Ziming
> On Nov 29, 2022, at 21:52, Viktor Somogyi-Vass <vi...@cloudera.com.INVALID> wrote:
>
> Hi All,
>
> I'd like to bump this. I've also updated the KIP to incorporate the new
> KRaft changes (ReplicaPlacer). Luckily my proposals were quite similar to
> that, so mostly I've made some minor rewording, naming changes, etc.
>
> Again, the brief summary of the KIP:
> - expose replica placement strategies with a new config
> - create an admin API and protocol to expose replica placement
> functionality (mainly for the reassignment tool)
> - create a new multi-level rack awareness strategy which improves
> availability on stretch clusters
>
> I'm happy for any feedback.
>
> Best,
> Viktor
>
> On Fri, Oct 28, 2022 at 4:14 PM Viktor Somogyi-Vass <
> viktor.somogyi@cloudera.com> wrote:
>
>> Hey all,
>>
>> I'd like to propose a new broker side replica assignment strategy and an
>> interface that generalizes replica assignment on brokers and makes them
>> pluggable.
>>
>> Briefly, the motivation for the new replica assignment strategy is that
>> more and more of our customers would want to run their clusters in a
>> stretched environment, where for instance a cluster is running over
>> multiple regions (and multiple racks inside a region). Since this seems
>> like a more common need, we'd like to contribute back our implementation
>> and also make a generalized interface, so that new strategies that people
>> may come up with could be served better.
>>
>> I welcome any feedback on this KIP.
>>
>> The link:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness
>>
>> Best to all,
>> Viktor
>>
Re: [DISCUSS] KIP-879: Multi-level Rack Awareness
Posted by Viktor Somogyi-Vass <vi...@cloudera.com.INVALID>.
Hi All,
I'd like to bump this. I've also updated the KIP to incorporate the new
KRaft changes (ReplicaPlacer). Luckily my proposals were quite similar to
that, so mostly I've made some minor rewording, naming changes, etc.
Again, the brief summary of the KIP:
- expose replica placement strategies with a new config
- create an admin API and protocol to expose replica placement
functionality (mainly for the reassignment tool)
- create a new multi-level rack awareness strategy which improves
availability on stretch clusters
I'm happy for any feedback.
Best,
Viktor
On Fri, Oct 28, 2022 at 4:14 PM Viktor Somogyi-Vass <
viktor.somogyi@cloudera.com> wrote:
> Hey all,
>
> I'd like to propose a new broker side replica assignment strategy and an
> interface that generalizes replica assignment on brokers and makes them
> pluggable.
>
> Briefly, the motivation for the new replica assignment strategy is that
> more and more of our customers would want to run their clusters in a
> stretched environment, where for instance a cluster is running over
> multiple regions (and multiple racks inside a region). Since this seems
> like a more common need, we'd like to contribute back our implementation
> and also make a generalized interface, so that new strategies that people
> may come up with could be served better.
>
> I welcome any feedback on this KIP.
>
> The link:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness
>
> Best to all,
> Viktor
>