You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Rahul Agarwal (Flipkart Data Group)" <ra...@flipkart.com.INVALID> on 2019/04/05 07:07:57 UTC
Kafka replication throttling on broker level
Hi,
We are using Kafka 1.0.2 and wanted to rate limit when a broker is replaced
and it syncs up with other brokers in the cluster. I followed the below doc
for the same:
https://kafka.apache.org/10/documentation.html#rep-throttle
According to this
document, leader.replication.throttled.rate,
follower.replication.throttled.rate
can be applied on broker level which I did but it didn't do anything. I had
to apply leader.replication.throttled.replicas,
follower.replication.throttled.replicas too *which this document doesn't
mention anywhere(that replica config needs to be applied too for rate
config to work)*. We don't want to apply replica config per topic level
basis as when new topics are created, we will have to apply the same config
for them too. Is there any way to do the same?
Also, can you please update the documentation too?
Thanks,
Rahul
Re: Kafka replication throttling on broker level
Posted by "Rahul Agarwal (Flipkart Data Group)" <ra...@flipkart.com.INVALID>.
Hi,
My Jira Id is rahul.mnit
Please add me to the contributors list.
Thanks,
Rahul
On Sat, Apr 6, 2019 at 2:58 AM Jun Rao <ju...@confluent.io> wrote:
> Hi, Rahul,
>
> If the ticket is unassigned, feel free to assign it to yourself. If you
> can't do that, let us know your jira id and we can add you to the
> contributors list.
>
> Thanks,
>
> Jun
>
>
> On Fri, Apr 5, 2019 at 11:45 AM Rahul Agarwal (Flipkart Data Group)
> <ra...@flipkart.com.invalid> wrote:
>
> > Hi,
> >
> > We actually need this feature for our system. This ticket is currently
> > unassigned. Can this ticket be assigned to me? I will send a pull request
> > for the feature.
> >
> > Thanks,
> > Rahul
> >
> > On Fri, Apr 5, 2019 at 9:59 PM Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Rahul,
> > >
> > > Those 2 configs are not supported at the broker level right now. This
> is
> > > being tracked at https://issues.apache.org/jira/browse/KAFKA-7983.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Apr 5, 2019 at 9:17 AM Rahul Agarwal (Flipkart Data Group)
> > > <ra...@flipkart.com.invalid> wrote:
> > >
> > > > Hi,
> > > >
> > > > We are using Kafka 1.0.2 and wanted to rate limit when a broker is
> > > replaced
> > > > and it syncs up with other brokers in the cluster. I followed the
> below
> > > doc
> > > > for the same:
> > > > https://kafka.apache.org/10/documentation.html#rep-throttle
> > > >
> > > > According to this
> > > > document, leader.replication.throttled.rate,
> > > > follower.replication.throttled.rate
> > > > can be applied on broker level which I did but it didn't do
> anything. I
> > > had
> > > > to apply leader.replication.throttled.replicas,
> > > > follower.replication.throttled.replicas too *which this document
> > doesn't
> > > > mention anywhere(that replica config needs to be applied too for rate
> > > > config to work)*. We don't want to apply replica config per topic
> level
> > > > basis as when new topics are created, we will have to apply the same
> > > config
> > > > for them too. Is there any way to do the same?
> > > >
> > > > Also, can you please update the documentation too?
> > > >
> > > > Thanks,
> > > > Rahul
> > > >
> > >
> >
>
Re: Kafka replication throttling on broker level
Posted by Jun Rao <ju...@confluent.io>.
Hi, Rahul,
If the ticket is unassigned, feel free to assign it to yourself. If you
can't do that, let us know your jira id and we can add you to the
contributors list.
Thanks,
Jun
On Fri, Apr 5, 2019 at 11:45 AM Rahul Agarwal (Flipkart Data Group)
<ra...@flipkart.com.invalid> wrote:
> Hi,
>
> We actually need this feature for our system. This ticket is currently
> unassigned. Can this ticket be assigned to me? I will send a pull request
> for the feature.
>
> Thanks,
> Rahul
>
> On Fri, Apr 5, 2019 at 9:59 PM Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Rahul,
> >
> > Those 2 configs are not supported at the broker level right now. This is
> > being tracked at https://issues.apache.org/jira/browse/KAFKA-7983.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Apr 5, 2019 at 9:17 AM Rahul Agarwal (Flipkart Data Group)
> > <ra...@flipkart.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > We are using Kafka 1.0.2 and wanted to rate limit when a broker is
> > replaced
> > > and it syncs up with other brokers in the cluster. I followed the below
> > doc
> > > for the same:
> > > https://kafka.apache.org/10/documentation.html#rep-throttle
> > >
> > > According to this
> > > document, leader.replication.throttled.rate,
> > > follower.replication.throttled.rate
> > > can be applied on broker level which I did but it didn't do anything. I
> > had
> > > to apply leader.replication.throttled.replicas,
> > > follower.replication.throttled.replicas too *which this document
> doesn't
> > > mention anywhere(that replica config needs to be applied too for rate
> > > config to work)*. We don't want to apply replica config per topic level
> > > basis as when new topics are created, we will have to apply the same
> > config
> > > for them too. Is there any way to do the same?
> > >
> > > Also, can you please update the documentation too?
> > >
> > > Thanks,
> > > Rahul
> > >
> >
>
Re: Kafka replication throttling on broker level
Posted by "Rahul Agarwal (Flipkart Data Group)" <ra...@flipkart.com.INVALID>.
Hi,
We actually need this feature for our system. This ticket is currently
unassigned. Can this ticket be assigned to me? I will send a pull request
for the feature.
Thanks,
Rahul
On Fri, Apr 5, 2019 at 9:59 PM Jun Rao <ju...@confluent.io> wrote:
> Hi, Rahul,
>
> Those 2 configs are not supported at the broker level right now. This is
> being tracked at https://issues.apache.org/jira/browse/KAFKA-7983.
>
> Thanks,
>
> Jun
>
> On Fri, Apr 5, 2019 at 9:17 AM Rahul Agarwal (Flipkart Data Group)
> <ra...@flipkart.com.invalid> wrote:
>
> > Hi,
> >
> > We are using Kafka 1.0.2 and wanted to rate limit when a broker is
> replaced
> > and it syncs up with other brokers in the cluster. I followed the below
> doc
> > for the same:
> > https://kafka.apache.org/10/documentation.html#rep-throttle
> >
> > According to this
> > document, leader.replication.throttled.rate,
> > follower.replication.throttled.rate
> > can be applied on broker level which I did but it didn't do anything. I
> had
> > to apply leader.replication.throttled.replicas,
> > follower.replication.throttled.replicas too *which this document doesn't
> > mention anywhere(that replica config needs to be applied too for rate
> > config to work)*. We don't want to apply replica config per topic level
> > basis as when new topics are created, we will have to apply the same
> config
> > for them too. Is there any way to do the same?
> >
> > Also, can you please update the documentation too?
> >
> > Thanks,
> > Rahul
> >
>
Re: Kafka replication throttling on broker level
Posted by Jun Rao <ju...@confluent.io>.
Hi, Rahul,
Those 2 configs are not supported at the broker level right now. This is
being tracked at https://issues.apache.org/jira/browse/KAFKA-7983.
Thanks,
Jun
On Fri, Apr 5, 2019 at 9:17 AM Rahul Agarwal (Flipkart Data Group)
<ra...@flipkart.com.invalid> wrote:
> Hi,
>
> We are using Kafka 1.0.2 and wanted to rate limit when a broker is replaced
> and it syncs up with other brokers in the cluster. I followed the below doc
> for the same:
> https://kafka.apache.org/10/documentation.html#rep-throttle
>
> According to this
> document, leader.replication.throttled.rate,
> follower.replication.throttled.rate
> can be applied on broker level which I did but it didn't do anything. I had
> to apply leader.replication.throttled.replicas,
> follower.replication.throttled.replicas too *which this document doesn't
> mention anywhere(that replica config needs to be applied too for rate
> config to work)*. We don't want to apply replica config per topic level
> basis as when new topics are created, we will have to apply the same config
> for them too. Is there any way to do the same?
>
> Also, can you please update the documentation too?
>
> Thanks,
> Rahul
>