You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Harry Fallows <ha...@protonmail.com.INVALID> on 2024/06/06 16:59:33 UTC
[DISCUSS] KIP-1051 Statically configured log replication throttling
Hi everyone,
I would like to propose a change to allow the static configuration of leader and follower replication throttling rates.
These configurations are very useful for preventing client traffic from getting throttled by replication traffic during events that cause a spike in replication. Currently they are only configurable dynamically, which means they are only really useful for throttling replication traffic during planned events. By allowing these configurations to be set statically, they can be used to prevent client traffic throttling during unplanned events.
KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1051%3A+Statically+configured+log+replication+throttling
Best regards,
Harry Fallows
Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
Posted by Divij Vaidya <di...@gmail.com>.
Thank you for the KIP. Overall, the proposal looks good to me.
I am adding some additional comments here:
1. What type of constraints do we want to add on static configuration? I am
assuming we should probably define some sane lower limit so that basic
replication (empty fetch requests) can still proceed normally. I am
suggesting this because users may mis-configure this and block simple fetch
requests which act as "heartbeat" in Kafka. This will lead to clusters
becoming unstable in weird ways and would be very hard to debug.
2. There is a problem with existing broker-level throttling that it will
throttle reserved topics as well (please correct me if I am wrong here).
This is because the reserved topics (such as __metadata, __consumer_offsets
etc.) are treated the same as regular topics from a throttling perspective.
Having a static configuration would increase the probability of landing in
a situation where the cluster becomes unstable since replication for
reserved topics is getting throttled. This is not a blocker for this KIP
because this is not a new risk, it already exists with dynamic
configuration, but perhaps worth calling out as a risk in our documentation
since the probability of risk will increase now. This is also the reason I
do not want users to set throttling below a certain threshold (and hence,
introduce a lower bound).
--
Divij Vaidya
On Thu, Dec 26, 2024 at 3:53 PM Harry Fallows
<ha...@protonmail.com.invalid> wrote:
> Hi Kamal,
>
> Apologies for the very long delay, but I have now updated the KIP to
> include the metrics and the kafka-reassign-partitions.sh clarification.
>
> Thanks,
> Harry
>
> On Thursday, 18 July 2024 at 12:16, Kamal Chandraprakash <
> kamal.chandraprakash@gmail.com> wrote:
>
> > Hi Harry,
> >
> > Thanks for the updates!
> >
> > Yes, the proposed metric looks good.
> >
> > If the user runs the kafka-reassign-partitions script with throttle set,
> > then the static throttle gets overwritten
> > until the reassignment gets completed. Can you clarify this on the KIP?
> >
> > --
> > Kamal
> >
> >
> >
> > On Sun, Jul 14, 2024 at 9:59 PM Harry Fallows
> > harryfallows@protonmail.com.invalid wrote:
> >
> > > Hi Kamal,
> > >
> > > Thank you for reading KIP-1051!
> > >
> > > Yes, it's true that it can impact regular replication traffic. However,
> > > network throughput is bounded so regardless of whether we allow it as a
> > > config in Kafka or not, there is always a chance that replication
> traffic
> > > will get throttled. Having it as a config will at least ensure that the
> > > entire bandwidth is not taken up by replication traffic.
> > >
> > > I agree, the nature of the leader replication throttling is dependent
> on
> > > how many followers there are, however, I don't think it's dependent on
> the
> > > partition assignment strategy or the number of brokers; it should only
> be
> > > dependent on the replication factor. I think it's key to point out here
> > > that these configurations do not need to be "optimised" for use cases
> with
> > > different replication factors, they just need to be set to match the
> > > infrastructure that they are deployed in. For example if you have a
> maximum
> > > network bandwidth of 200MB/s and a replication factor of 3, you may set
> > > follower.replication.throttled.replicas to 150MB/s, to reserve some
> > > bandwidth for other traffic (e.g. producing and consuming). In this
> case,
> > > if you start with all replicas in sync, I don't think it's possible
> for the
> > > follower throttling to be the sole cause of a replica falling out of
> sync.
> > > It may be the case that it takes longer for an out-of-sync replica to
> > > become in sync, but in that case the replication throttling just
> serves to
> > > mitigate other traffic from getting throttled (e.g. producer traffic
> to a
> > > different partition). Even so, it is possible that misconfiguring these
> > > values could cause issues, so the potential consequences should be
> clearly
> > > documented.
> > >
> > > I think the concern about producing spikes causing ISR issues is only
> an
> > > issue if these values are poorly configured. I think in general if
> these
> > > values are always configured as >=
> > > (replicationFactor/(replicationFactor+1))*maxBandwidth (e.g. like the
> above
> > > example: 3/(3+1) * 200 = 150), then even if 100% of the non-replication
> > > traffic is producer traffic, all followers should be able to stay in
> sync.
> > >
> > > I like the idea of emitting a metric for when a quota is breached,
> what do
> > > you think about having it as a gauge for number of partitions that are
> > > currently leader of follower throttled (similar to the URP metric)?
> > >
> > > Kind regards,
> > > Harry
> > >
> > > On Thursday, 11 July 2024 at 19:02, Kamal Chandraprakash <
> > > kamal.chandraprakash@gmail.com> wrote:
> > >
> > > > Hi Harry Fallows,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > I went over both the KIP-1051 and KIP-1009. Assuming that the
> > > > leader.replication.throttled.replicas
> > > > and follower.replication.throttled.replicas are set to Wildcard (*)
> to
> > > > apply for all the partitions in the
> > > > broker. If we set a static value for leader and follower replication
> > > > throttled rate, then it might impact
> > > > the normal replication traffic.
> > > >
> > > > Throttling rate depends on the number of brokers in the cluster. If
> the
> > > > cluster contains 100+ brokers, then
> > > > the leader.replication.throttled.rate is shared across all the
> followers.
> > > > The number of followers reading
> > > > data from the leader depends on the partition assignment strategy.
> If the
> > > > leader replication throttle is breached,
> > > > then the follower might fail to catch-up with the leader.
> > > >
> > > > If there are sudden spikes in a specific set of topics/partitions in
> the
> > > > cluster, then the replicas might fail to join
> > > > the isr and can impact the cluster reliability. If we are going with
> this
> > > > proposal, then we may also have to emit
> > > > a metric to inform the administrator that the leader/follower
> replication
> > > > quota is breached.
> > > >
> > > > --
> > > > Kamal
> > > >
> > > > On Thu, Jul 4, 2024 at 8:10 PM Harry Fallows
> > > > harryfallows@protonmail.com.invalid wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Bumping this one last time before I call a vote. Please take a
> look if
> > > > > you're interested in replication throttling and/or static/dynamic
> > > > > config.
> > > > >
> > > > > Kind regards,
> > > > > Harry
> > > > >
> > > > > On Thursday, 13 June 2024 at 19:39, Harry Fallows <
> > > > > harryfallows@protonmail.com.INVALID> wrote:
> > > > >
> > > > > > Hi Hector,
> > > > > >
> > > > > > I did see your colleague's KIP, and I actually mentioned it in
> the
> > > > > > KIP
> > > > > > that I have written. As I see it, both of these KIPs move towards
> > > > > > more
> > > > > > easily configurable replication throttling and both should be
> > > > > > implemented.
> > > > > > KIP-1009 makes it easier to enable throttling and KIP-1051 makes
> it
> > > > > > easier
> > > > > > to apply a throttle rate. I did try to look at supporting
> KIP-1009
> > > > > > in the
> > > > > > discussion thread, however, I only subscribed to the mailing list
> > > > > > after it
> > > > > > was published and I couldn't figure out how to respond to it in
> Pony
> > > > > > mail.
> > > > > > I would be definitely be interested in partnering up to get both
> > > > > > changes
> > > > > > across the line, whether that be by combining them or supporting
> both
> > > > > > individually (I'm not sure which is best, this is my first
> > > > > > contribution!).
> > > > > >
> > > > > > I also see that KAFKA-10190 is mentioned in KIP-1009 as a related
> > > > > > ticket. Coincidentally, I raised a PR to address this bug a
> couple
> > > > > > of days
> > > > > > ago (https://github.com/apache/kafka/pull/16280). I think this
> is
> > > > > > also a
> > > > > > change that will move towards more easily configurable
> replication
> > > > > > throttling as it allows configuring the throttle rate across the
> > > > > > whole
> > > > > > cluster via a default value. As far as I understand, this change
> > > > > > does not
> > > > > > need a KIP though because it is a bugfix (the current behaviour
> of
> > > > > > ignoring
> > > > > > the default is unintentional).
> > > > > >
> > > > > > Let me know what you think.
> > > > > >
> > > > > > Kind regards,
> > > > > > Harry
> > > > > >
> > > > > > -------- Original Message --------
> > > > > > On 6/13/24 19:08, Hector Geraldino (BLOOMBERG/ 919 3RD A)
> > > > > > hgeraldino@bloomberg.net wrote:
> > > > > >
> > > > > > > Hi Harry,
> > > > > > >
> > > > > > > A colleague of mine opened KIP-1009: Add Broker-level Throttle
> > > > > > > Configurations, which aims to achieve the same goal (although
> from
> > > > > > > a
> > > > > > > different angle).
> > > > > > >
> > > > > > > Can you please take a look and see if this would work for the
> > > > > > > things
> > > > > > > you have in mind? Maybe we can partner and coalesce around
> either
> > > > > > > KIP and
> > > > > > > try to push it to the end line.
> > > > > > >
> > > > > > > KIP:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1009%3A+Add+Broker-level+Throttle+Configurations
> > >
> > > > > > > From: dev@kafka.apache.org At: 06/13/24 09:22:40 UTC-4:00To:
> > > > > > > dev@kafka.apache.org
> > > > > > > Subject: Re: [DISCUSS] KIP-1051 Statically configured log
> > > > > > > replication
> > > > > > > throttling
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Bumping this thread, as I haven't yet had any replies.
> > > > > > >
> > > > > > > Kind regards,
> > > > > > > Harry
> > > > > > >
> > > > > > > On Thursday, 6 June 2024 at 17:59, Harry Fallows
> > > > > > > harryfallows@protonmail.com.INVALID wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I would like to propose a change to allow the static
> > > > > > > > configuration
> > > > > > > > of leader
> > > > > > > > and follower replication throttling rates.
> > > > > > > >
> > > > > > > > These configurations are very useful for preventing client
> > > > > > > > traffic
> > > > > > > > from
> > > > > > > > getting throttled by replication traffic during events that
> > > > > > > > cause a
> > > > > > > > spike in
> > > > > > > > replication. Currently they are only configurable
> dynamically,
> > > > > > > > which
> > > > > > > > means they
> > > > > > > > are only really useful for throttling replication traffic
> during
> > > > > > > > planned
> > > > > > > > events. By allowing these configurations to be set
> statically,
> > > > > > > > they
> > > > > > > > can be used
> > > > > > > > to prevent client traffic throttling during unplanned events.
> > > > > > > >
> > > > > > > > KIP:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1051%3A+Statically+configu
> > >
> > > > > > > > red+log+replication+throttling
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Harry Fallows
>
Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
Posted by Harry Fallows <ha...@protonmail.com.INVALID>.
Hi Kamal,
Apologies for the very long delay, but I have now updated the KIP to include the metrics and the kafka-reassign-partitions.sh clarification.
Thanks,
Harry
On Thursday, 18 July 2024 at 12:16, Kamal Chandraprakash <ka...@gmail.com> wrote:
> Hi Harry,
>
> Thanks for the updates!
>
> Yes, the proposed metric looks good.
>
> If the user runs the kafka-reassign-partitions script with throttle set,
> then the static throttle gets overwritten
> until the reassignment gets completed. Can you clarify this on the KIP?
>
> --
> Kamal
>
>
>
> On Sun, Jul 14, 2024 at 9:59 PM Harry Fallows
> harryfallows@protonmail.com.invalid wrote:
>
> > Hi Kamal,
> >
> > Thank you for reading KIP-1051!
> >
> > Yes, it's true that it can impact regular replication traffic. However,
> > network throughput is bounded so regardless of whether we allow it as a
> > config in Kafka or not, there is always a chance that replication traffic
> > will get throttled. Having it as a config will at least ensure that the
> > entire bandwidth is not taken up by replication traffic.
> >
> > I agree, the nature of the leader replication throttling is dependent on
> > how many followers there are, however, I don't think it's dependent on the
> > partition assignment strategy or the number of brokers; it should only be
> > dependent on the replication factor. I think it's key to point out here
> > that these configurations do not need to be "optimised" for use cases with
> > different replication factors, they just need to be set to match the
> > infrastructure that they are deployed in. For example if you have a maximum
> > network bandwidth of 200MB/s and a replication factor of 3, you may set
> > follower.replication.throttled.replicas to 150MB/s, to reserve some
> > bandwidth for other traffic (e.g. producing and consuming). In this case,
> > if you start with all replicas in sync, I don't think it's possible for the
> > follower throttling to be the sole cause of a replica falling out of sync.
> > It may be the case that it takes longer for an out-of-sync replica to
> > become in sync, but in that case the replication throttling just serves to
> > mitigate other traffic from getting throttled (e.g. producer traffic to a
> > different partition). Even so, it is possible that misconfiguring these
> > values could cause issues, so the potential consequences should be clearly
> > documented.
> >
> > I think the concern about producing spikes causing ISR issues is only an
> > issue if these values are poorly configured. I think in general if these
> > values are always configured as >=
> > (replicationFactor/(replicationFactor+1))*maxBandwidth (e.g. like the above
> > example: 3/(3+1) * 200 = 150), then even if 100% of the non-replication
> > traffic is producer traffic, all followers should be able to stay in sync.
> >
> > I like the idea of emitting a metric for when a quota is breached, what do
> > you think about having it as a gauge for number of partitions that are
> > currently leader of follower throttled (similar to the URP metric)?
> >
> > Kind regards,
> > Harry
> >
> > On Thursday, 11 July 2024 at 19:02, Kamal Chandraprakash <
> > kamal.chandraprakash@gmail.com> wrote:
> >
> > > Hi Harry Fallows,
> > >
> > > Thanks for the KIP!
> > >
> > > I went over both the KIP-1051 and KIP-1009. Assuming that the
> > > leader.replication.throttled.replicas
> > > and follower.replication.throttled.replicas are set to Wildcard (*) to
> > > apply for all the partitions in the
> > > broker. If we set a static value for leader and follower replication
> > > throttled rate, then it might impact
> > > the normal replication traffic.
> > >
> > > Throttling rate depends on the number of brokers in the cluster. If the
> > > cluster contains 100+ brokers, then
> > > the leader.replication.throttled.rate is shared across all the followers.
> > > The number of followers reading
> > > data from the leader depends on the partition assignment strategy. If the
> > > leader replication throttle is breached,
> > > then the follower might fail to catch-up with the leader.
> > >
> > > If there are sudden spikes in a specific set of topics/partitions in the
> > > cluster, then the replicas might fail to join
> > > the isr and can impact the cluster reliability. If we are going with this
> > > proposal, then we may also have to emit
> > > a metric to inform the administrator that the leader/follower replication
> > > quota is breached.
> > >
> > > --
> > > Kamal
> > >
> > > On Thu, Jul 4, 2024 at 8:10 PM Harry Fallows
> > > harryfallows@protonmail.com.invalid wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Bumping this one last time before I call a vote. Please take a look if
> > > > you're interested in replication throttling and/or static/dynamic
> > > > config.
> > > >
> > > > Kind regards,
> > > > Harry
> > > >
> > > > On Thursday, 13 June 2024 at 19:39, Harry Fallows <
> > > > harryfallows@protonmail.com.INVALID> wrote:
> > > >
> > > > > Hi Hector,
> > > > >
> > > > > I did see your colleague's KIP, and I actually mentioned it in the
> > > > > KIP
> > > > > that I have written. As I see it, both of these KIPs move towards
> > > > > more
> > > > > easily configurable replication throttling and both should be
> > > > > implemented.
> > > > > KIP-1009 makes it easier to enable throttling and KIP-1051 makes it
> > > > > easier
> > > > > to apply a throttle rate. I did try to look at supporting KIP-1009
> > > > > in the
> > > > > discussion thread, however, I only subscribed to the mailing list
> > > > > after it
> > > > > was published and I couldn't figure out how to respond to it in Pony
> > > > > mail.
> > > > > I would be definitely be interested in partnering up to get both
> > > > > changes
> > > > > across the line, whether that be by combining them or supporting both
> > > > > individually (I'm not sure which is best, this is my first
> > > > > contribution!).
> > > > >
> > > > > I also see that KAFKA-10190 is mentioned in KIP-1009 as a related
> > > > > ticket. Coincidentally, I raised a PR to address this bug a couple
> > > > > of days
> > > > > ago (https://github.com/apache/kafka/pull/16280). I think this is
> > > > > also a
> > > > > change that will move towards more easily configurable replication
> > > > > throttling as it allows configuring the throttle rate across the
> > > > > whole
> > > > > cluster via a default value. As far as I understand, this change
> > > > > does not
> > > > > need a KIP though because it is a bugfix (the current behaviour of
> > > > > ignoring
> > > > > the default is unintentional).
> > > > >
> > > > > Let me know what you think.
> > > > >
> > > > > Kind regards,
> > > > > Harry
> > > > >
> > > > > -------- Original Message --------
> > > > > On 6/13/24 19:08, Hector Geraldino (BLOOMBERG/ 919 3RD A)
> > > > > hgeraldino@bloomberg.net wrote:
> > > > >
> > > > > > Hi Harry,
> > > > > >
> > > > > > A colleague of mine opened KIP-1009: Add Broker-level Throttle
> > > > > > Configurations, which aims to achieve the same goal (although from
> > > > > > a
> > > > > > different angle).
> > > > > >
> > > > > > Can you please take a look and see if this would work for the
> > > > > > things
> > > > > > you have in mind? Maybe we can partner and coalesce around either
> > > > > > KIP and
> > > > > > try to push it to the end line.
> > > > > >
> > > > > > KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1009%3A+Add+Broker-level+Throttle+Configurations
> >
> > > > > > From: dev@kafka.apache.org At: 06/13/24 09:22:40 UTC-4:00To:
> > > > > > dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-1051 Statically configured log
> > > > > > replication
> > > > > > throttling
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Bumping this thread, as I haven't yet had any replies.
> > > > > >
> > > > > > Kind regards,
> > > > > > Harry
> > > > > >
> > > > > > On Thursday, 6 June 2024 at 17:59, Harry Fallows
> > > > > > harryfallows@protonmail.com.INVALID wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I would like to propose a change to allow the static
> > > > > > > configuration
> > > > > > > of leader
> > > > > > > and follower replication throttling rates.
> > > > > > >
> > > > > > > These configurations are very useful for preventing client
> > > > > > > traffic
> > > > > > > from
> > > > > > > getting throttled by replication traffic during events that
> > > > > > > cause a
> > > > > > > spike in
> > > > > > > replication. Currently they are only configurable dynamically,
> > > > > > > which
> > > > > > > means they
> > > > > > > are only really useful for throttling replication traffic during
> > > > > > > planned
> > > > > > > events. By allowing these configurations to be set statically,
> > > > > > > they
> > > > > > > can be used
> > > > > > > to prevent client traffic throttling during unplanned events.
> > > > > > >
> > > > > > > KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1051%3A+Statically+configu
> >
> > > > > > > red+log+replication+throttling
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Harry Fallows
Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
Posted by Kamal Chandraprakash <ka...@gmail.com>.
Hi Harry,
Thanks for the updates!
Yes, the proposed metric looks good.
If the user runs the kafka-reassign-partitions script with throttle set,
then the static throttle gets overwritten
until the reassignment gets completed. Can you clarify this on the KIP?
--
Kamal
On Sun, Jul 14, 2024 at 9:59 PM Harry Fallows
<ha...@protonmail.com.invalid> wrote:
> Hi Kamal,
>
> Thank you for reading KIP-1051!
>
> Yes, it's true that it can impact regular replication traffic. However,
> network throughput is bounded so regardless of whether we allow it as a
> config in Kafka or not, there is always a chance that replication traffic
> will get throttled. Having it as a config will at least ensure that the
> entire bandwidth is not taken up by replication traffic.
>
> I agree, the nature of the leader replication throttling is dependent on
> how many followers there are, however, I don't think it's dependent on the
> partition assignment strategy or the number of brokers; it should only be
> dependent on the replication factor. I think it's key to point out here
> that these configurations do not need to be "optimised" for use cases with
> different replication factors, they just need to be set to match the
> infrastructure that they are deployed in. For example if you have a maximum
> network bandwidth of 200MB/s and a replication factor of 3, you may set
> follower.replication.throttled.replicas to 150MB/s, to reserve some
> bandwidth for other traffic (e.g. producing and consuming). In this case,
> if you start with all replicas in sync, I don't think it's possible for the
> follower throttling to be the sole cause of a replica falling out of sync.
> It may be the case that it takes longer for an out-of-sync replica to
> become in sync, but in that case the replication throttling just serves to
> mitigate other traffic from getting throttled (e.g. producer traffic to a
> different partition). Even so, it is possible that misconfiguring these
> values could cause issues, so the potential consequences should be clearly
> documented.
>
> I think the concern about producing spikes causing ISR issues is only an
> issue if these values are poorly configured. I think in general if these
> values are always configured as >=
> (replicationFactor/(replicationFactor+1))*maxBandwidth (e.g. like the above
> example: 3/(3+1) * 200 = 150), then even if 100% of the non-replication
> traffic is producer traffic, all followers should be able to stay in sync.
>
> I like the idea of emitting a metric for when a quota is breached, what do
> you think about having it as a gauge for number of partitions that are
> currently leader of follower throttled (similar to the URP metric)?
>
> Kind regards,
> Harry
>
> On Thursday, 11 July 2024 at 19:02, Kamal Chandraprakash <
> kamal.chandraprakash@gmail.com> wrote:
>
> > Hi Harry Fallows,
> >
> > Thanks for the KIP!
> >
> > I went over both the KIP-1051 and KIP-1009. Assuming that the
> > leader.replication.throttled.replicas
> > and follower.replication.throttled.replicas are set to Wildcard (*) to
> > apply for all the partitions in the
> > broker. If we set a static value for leader and follower replication
> > throttled rate, then it might impact
> > the normal replication traffic.
> >
> > Throttling rate depends on the number of brokers in the cluster. If the
> > cluster contains 100+ brokers, then
> > the leader.replication.throttled.rate is shared across all the followers.
> > The number of followers reading
> > data from the leader depends on the partition assignment strategy. If the
> > leader replication throttle is breached,
> > then the follower might fail to catch-up with the leader.
> >
> > If there are sudden spikes in a specific set of topics/partitions in the
> > cluster, then the replicas might fail to join
> > the isr and can impact the cluster reliability. If we are going with this
> > proposal, then we may also have to emit
> > a metric to inform the administrator that the leader/follower replication
> > quota is breached.
> >
> > --
> > Kamal
> >
> > On Thu, Jul 4, 2024 at 8:10 PM Harry Fallows
> > harryfallows@protonmail.com.invalid wrote:
> >
> > > Hi everyone,
> > >
> > > Bumping this one last time before I call a vote. Please take a look if
> > > you're interested in replication throttling and/or static/dynamic
> config.
> > >
> > > Kind regards,
> > > Harry
> > >
> > > On Thursday, 13 June 2024 at 19:39, Harry Fallows <
> > > harryfallows@protonmail.com.INVALID> wrote:
> > >
> > > > Hi Hector,
> > > >
> > > > I did see your colleague's KIP, and I actually mentioned it in the
> KIP
> > > > that I have written. As I see it, both of these KIPs move towards
> more
> > > > easily configurable replication throttling and both should be
> implemented.
> > > > KIP-1009 makes it easier to enable throttling and KIP-1051 makes it
> easier
> > > > to apply a throttle rate. I did try to look at supporting KIP-1009
> in the
> > > > discussion thread, however, I only subscribed to the mailing list
> after it
> > > > was published and I couldn't figure out how to respond to it in Pony
> mail.
> > > > I would be definitely be interested in partnering up to get both
> changes
> > > > across the line, whether that be by combining them or supporting both
> > > > individually (I'm not sure which is best, this is my first
> contribution!).
> > > >
> > > > I also see that KAFKA-10190 is mentioned in KIP-1009 as a related
> > > > ticket. Coincidentally, I raised a PR to address this bug a couple
> of days
> > > > ago (https://github.com/apache/kafka/pull/16280). I think this is
> also a
> > > > change that will move towards more easily configurable replication
> > > > throttling as it allows configuring the throttle rate across the
> whole
> > > > cluster via a default value. As far as I understand, this change
> does not
> > > > need a KIP though because it is a bugfix (the current behaviour of
> ignoring
> > > > the default is unintentional).
> > > >
> > > > Let me know what you think.
> > > >
> > > > Kind regards,
> > > > Harry
> > > >
> > > > -------- Original Message --------
> > > > On 6/13/24 19:08, Hector Geraldino (BLOOMBERG/ 919 3RD A)
> > > > hgeraldino@bloomberg.net wrote:
> > > >
> > > > > Hi Harry,
> > > > >
> > > > > A colleague of mine opened KIP-1009: Add Broker-level Throttle
> > > > > Configurations, which aims to achieve the same goal (although from
> a
> > > > > different angle).
> > > > >
> > > > > Can you please take a look and see if this would work for the
> things
> > > > > you have in mind? Maybe we can partner and coalesce around either
> KIP and
> > > > > try to push it to the end line.
> > > > >
> > > > > KIP:
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1009%3A+Add+Broker-level+Throttle+Configurations
> > > > >
> > > > > From: dev@kafka.apache.org At: 06/13/24 09:22:40 UTC-4:00To:
> > > > > dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-1051 Statically configured log
> replication
> > > > > throttling
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > Bumping this thread, as I haven't yet had any replies.
> > > > >
> > > > > Kind regards,
> > > > > Harry
> > > > >
> > > > > On Thursday, 6 June 2024 at 17:59, Harry Fallows
> > > > > harryfallows@protonmail.com.INVALID wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I would like to propose a change to allow the static
> configuration
> > > > > > of leader
> > > > > > and follower replication throttling rates.
> > > > > >
> > > > > > These configurations are very useful for preventing client
> traffic
> > > > > > from
> > > > > > getting throttled by replication traffic during events that
> cause a
> > > > > > spike in
> > > > > > replication. Currently they are only configurable dynamically,
> which
> > > > > > means they
> > > > > > are only really useful for throttling replication traffic during
> > > > > > planned
> > > > > > events. By allowing these configurations to be set statically,
> they
> > > > > > can be used
> > > > > > to prevent client traffic throttling during unplanned events.
> > > > > >
> > > > > > KIP:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1051%3A+Statically+configu
> > >
> > > > > > red+log+replication+throttling
> > > > > >
> > > > > > Best regards,
> > > > > > Harry Fallows
>
Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
Posted by Harry Fallows <ha...@protonmail.com.INVALID>.
Hi Kamal,
Thank you for reading KIP-1051!
Yes, it's true that it can impact regular replication traffic. However, network throughput is bounded so regardless of whether we allow it as a config in Kafka or not, there is always a chance that replication traffic will get throttled. Having it as a config will at least ensure that the entire bandwidth is not taken up by replication traffic.
I agree, the nature of the leader replication throttling is dependent on how many followers there are, however, I don't think it's dependent on the partition assignment strategy or the number of brokers; it should only be dependent on the replication factor. I think it's key to point out here that these configurations do not need to be "optimised" for use cases with different replication factors, they just need to be set to match the infrastructure that they are deployed in. For example if you have a maximum network bandwidth of 200MB/s and a replication factor of 3, you may set follower.replication.throttled.replicas to 150MB/s, to reserve some bandwidth for other traffic (e.g. producing and consuming). In this case, if you start with all replicas in sync, I don't think it's possible for the follower throttling to be the sole cause of a replica falling out of sync. It may be the case that it takes longer for an out-of-sync replica to become in sync, but in that case the replication throttling just serves to mitigate other traffic from getting throttled (e.g. producer traffic to a different partition). Even so, it is possible that misconfiguring these values could cause issues, so the potential consequences should be clearly documented.
I think the concern about producing spikes causing ISR issues is only an issue if these values are poorly configured. I think in general if these values are always configured as >= (replicationFactor/(replicationFactor+1))*maxBandwidth (e.g. like the above example: 3/(3+1) * 200 = 150), then even if 100% of the non-replication traffic is producer traffic, all followers should be able to stay in sync.
I like the idea of emitting a metric for when a quota is breached, what do you think about having it as a gauge for number of partitions that are currently leader of follower throttled (similar to the URP metric)?
Kind regards,
Harry
On Thursday, 11 July 2024 at 19:02, Kamal Chandraprakash <ka...@gmail.com> wrote:
> Hi Harry Fallows,
>
> Thanks for the KIP!
>
> I went over both the KIP-1051 and KIP-1009. Assuming that the
> leader.replication.throttled.replicas
> and follower.replication.throttled.replicas are set to Wildcard (*) to
> apply for all the partitions in the
> broker. If we set a static value for leader and follower replication
> throttled rate, then it might impact
> the normal replication traffic.
>
> Throttling rate depends on the number of brokers in the cluster. If the
> cluster contains 100+ brokers, then
> the leader.replication.throttled.rate is shared across all the followers.
> The number of followers reading
> data from the leader depends on the partition assignment strategy. If the
> leader replication throttle is breached,
> then the follower might fail to catch-up with the leader.
>
> If there are sudden spikes in a specific set of topics/partitions in the
> cluster, then the replicas might fail to join
> the isr and can impact the cluster reliability. If we are going with this
> proposal, then we may also have to emit
> a metric to inform the administrator that the leader/follower replication
> quota is breached.
>
> --
> Kamal
>
> On Thu, Jul 4, 2024 at 8:10 PM Harry Fallows
> harryfallows@protonmail.com.invalid wrote:
>
> > Hi everyone,
> >
> > Bumping this one last time before I call a vote. Please take a look if
> > you're interested in replication throttling and/or static/dynamic config.
> >
> > Kind regards,
> > Harry
> >
> > On Thursday, 13 June 2024 at 19:39, Harry Fallows <
> > harryfallows@protonmail.com.INVALID> wrote:
> >
> > > Hi Hector,
> > >
> > > I did see your colleague's KIP, and I actually mentioned it in the KIP
> > > that I have written. As I see it, both of these KIPs move towards more
> > > easily configurable replication throttling and both should be implemented.
> > > KIP-1009 makes it easier to enable throttling and KIP-1051 makes it easier
> > > to apply a throttle rate. I did try to look at supporting KIP-1009 in the
> > > discussion thread, however, I only subscribed to the mailing list after it
> > > was published and I couldn't figure out how to respond to it in Pony mail.
> > > I would be definitely be interested in partnering up to get both changes
> > > across the line, whether that be by combining them or supporting both
> > > individually (I'm not sure which is best, this is my first contribution!).
> > >
> > > I also see that KAFKA-10190 is mentioned in KIP-1009 as a related
> > > ticket. Coincidentally, I raised a PR to address this bug a couple of days
> > > ago (https://github.com/apache/kafka/pull/16280). I think this is also a
> > > change that will move towards more easily configurable replication
> > > throttling as it allows configuring the throttle rate across the whole
> > > cluster via a default value. As far as I understand, this change does not
> > > need a KIP though because it is a bugfix (the current behaviour of ignoring
> > > the default is unintentional).
> > >
> > > Let me know what you think.
> > >
> > > Kind regards,
> > > Harry
> > >
> > > -------- Original Message --------
> > > On 6/13/24 19:08, Hector Geraldino (BLOOMBERG/ 919 3RD A)
> > > hgeraldino@bloomberg.net wrote:
> > >
> > > > Hi Harry,
> > > >
> > > > A colleague of mine opened KIP-1009: Add Broker-level Throttle
> > > > Configurations, which aims to achieve the same goal (although from a
> > > > different angle).
> > > >
> > > > Can you please take a look and see if this would work for the things
> > > > you have in mind? Maybe we can partner and coalesce around either KIP and
> > > > try to push it to the end line.
> > > >
> > > > KIP:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1009%3A+Add+Broker-level+Throttle+Configurations
> > > >
> > > > From: dev@kafka.apache.org At: 06/13/24 09:22:40 UTC-4:00To:
> > > > dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-1051 Statically configured log replication
> > > > throttling
> > > >
> > > > Hi everyone,
> > > >
> > > > Bumping this thread, as I haven't yet had any replies.
> > > >
> > > > Kind regards,
> > > > Harry
> > > >
> > > > On Thursday, 6 June 2024 at 17:59, Harry Fallows
> > > > harryfallows@protonmail.com.INVALID wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I would like to propose a change to allow the static configuration
> > > > > of leader
> > > > > and follower replication throttling rates.
> > > > >
> > > > > These configurations are very useful for preventing client traffic
> > > > > from
> > > > > getting throttled by replication traffic during events that cause a
> > > > > spike in
> > > > > replication. Currently they are only configurable dynamically, which
> > > > > means they
> > > > > are only really useful for throttling replication traffic during
> > > > > planned
> > > > > events. By allowing these configurations to be set statically, they
> > > > > can be used
> > > > > to prevent client traffic throttling during unplanned events.
> > > > >
> > > > > KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1051%3A+Statically+configu
> >
> > > > > red+log+replication+throttling
> > > > >
> > > > > Best regards,
> > > > > Harry Fallows
Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
Posted by Kamal Chandraprakash <ka...@gmail.com>.
Hi Harry Fallows,
Thanks for the KIP!
I went over both the KIP-1051 and KIP-1009. Assuming that the
leader.replication.throttled.replicas
and follower.replication.throttled.replicas are set to Wildcard (*) to
apply for all the partitions in the
broker. If we set a static value for leader and follower replication
throttled rate, then it might impact
the normal replication traffic.
Throttling rate depends on the number of brokers in the cluster. If the
cluster contains 100+ brokers, then
the leader.replication.throttled.rate is shared across all the followers.
The number of followers reading
data from the leader depends on the partition assignment strategy. If the
leader replication throttle is breached,
then the follower might fail to catch-up with the leader.
If there are sudden spikes in a specific set of topics/partitions in the
cluster, then the replicas might fail to join
the isr and can impact the cluster reliability. If we are going with this
proposal, then we may also have to emit
a metric to inform the administrator that the leader/follower replication
quota is breached.
--
Kamal
On Thu, Jul 4, 2024 at 8:10 PM Harry Fallows
<ha...@protonmail.com.invalid> wrote:
> Hi everyone,
>
> Bumping this one last time before I call a vote. Please take a look if
> you're interested in replication throttling and/or static/dynamic config.
>
> Kind regards,
> Harry
>
> On Thursday, 13 June 2024 at 19:39, Harry Fallows <
> harryfallows@protonmail.com.INVALID> wrote:
>
> > Hi Hector,
> >
> > I did see your colleague's KIP, and I actually mentioned it in the KIP
> that I have written. As I see it, both of these KIPs move towards more
> easily configurable replication throttling and both should be implemented.
> KIP-1009 makes it easier to enable throttling and KIP-1051 makes it easier
> to apply a throttle rate. I did try to look at supporting KIP-1009 in the
> discussion thread, however, I only subscribed to the mailing list after it
> was published and I couldn't figure out how to respond to it in Pony mail.
> I would be definitely be interested in partnering up to get both changes
> across the line, whether that be by combining them or supporting both
> individually (I'm not sure which is best, this is my first contribution!).
> >
> > I also see that KAFKA-10190 is mentioned in KIP-1009 as a related
> ticket. Coincidentally, I raised a PR to address this bug a couple of days
> ago (https://github.com/apache/kafka/pull/16280). I think this is also a
> change that will move towards more easily configurable replication
> throttling as it allows configuring the throttle rate across the whole
> cluster via a default value. As far as I understand, this change does not
> need a KIP though because it is a bugfix (the current behaviour of ignoring
> the default is unintentional).
> >
> > Let me know what you think.
> >
> > Kind regards,
> > Harry
> >
> >
> > -------- Original Message --------
> > On 6/13/24 19:08, Hector Geraldino (BLOOMBERG/ 919 3RD A)
> hgeraldino@bloomberg.net wrote:
> >
> > > Hi Harry,
> > >
> > > A colleague of mine opened KIP-1009: Add Broker-level Throttle
> Configurations, which aims to achieve the same goal (although from a
> different angle).
> > >
> > > Can you please take a look and see if this would work for the things
> you have in mind? Maybe we can partner and coalesce around either KIP and
> try to push it to the end line.
> > >
> > > KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1009%3A+Add+Broker-level+Throttle+Configurations
> > >
> > > From: dev@kafka.apache.org At: 06/13/24 09:22:40 UTC-4:00To:
> dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-1051 Statically configured log replication
> throttling
> > >
> > > Hi everyone,
> > >
> > > Bumping this thread, as I haven't yet had any replies.
> > >
> > > Kind regards,
> > > Harry
> > >
> > > On Thursday, 6 June 2024 at 17:59, Harry Fallows
> > > harryfallows@protonmail.com.INVALID wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I would like to propose a change to allow the static configuration
> of leader
> > > > and follower replication throttling rates.
> > > >
> > > > These configurations are very useful for preventing client traffic
> from
> > > > getting throttled by replication traffic during events that cause a
> spike in
> > > > replication. Currently they are only configurable dynamically, which
> means they
> > > > are only really useful for throttling replication traffic during
> planned
> > > > events. By allowing these configurations to be set statically, they
> can be used
> > > > to prevent client traffic throttling during unplanned events.
> > > >
> > > > KIP:
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1051%3A+Statically+configu
> > > > red+log+replication+throttling
> > > >
> > > > Best regards,
> > > > Harry Fallows
>
Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
Posted by Harry Fallows <ha...@protonmail.com.INVALID>.
Hi everyone,
Bumping this one last time before I call a vote. Please take a look if you're interested in replication throttling and/or static/dynamic config.
Kind regards,
Harry
On Thursday, 13 June 2024 at 19:39, Harry Fallows <ha...@protonmail.com.INVALID> wrote:
> Hi Hector,
>
> I did see your colleague's KIP, and I actually mentioned it in the KIP that I have written. As I see it, both of these KIPs move towards more easily configurable replication throttling and both should be implemented. KIP-1009 makes it easier to enable throttling and KIP-1051 makes it easier to apply a throttle rate. I did try to look at supporting KIP-1009 in the discussion thread, however, I only subscribed to the mailing list after it was published and I couldn't figure out how to respond to it in Pony mail. I would be definitely be interested in partnering up to get both changes across the line, whether that be by combining them or supporting both individually (I'm not sure which is best, this is my first contribution!).
>
> I also see that KAFKA-10190 is mentioned in KIP-1009 as a related ticket. Coincidentally, I raised a PR to address this bug a couple of days ago (https://github.com/apache/kafka/pull/16280). I think this is also a change that will move towards more easily configurable replication throttling as it allows configuring the throttle rate across the whole cluster via a default value. As far as I understand, this change does not need a KIP though because it is a bugfix (the current behaviour of ignoring the default is unintentional).
>
> Let me know what you think.
>
> Kind regards,
> Harry
>
>
> -------- Original Message --------
> On 6/13/24 19:08, Hector Geraldino (BLOOMBERG/ 919 3RD A) hgeraldino@bloomberg.net wrote:
>
> > Hi Harry,
> >
> > A colleague of mine opened KIP-1009: Add Broker-level Throttle Configurations, which aims to achieve the same goal (although from a different angle).
> >
> > Can you please take a look and see if this would work for the things you have in mind? Maybe we can partner and coalesce around either KIP and try to push it to the end line.
> >
> > KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1009%3A+Add+Broker-level+Throttle+Configurations
> >
> > From: dev@kafka.apache.org At: 06/13/24 09:22:40 UTC-4:00To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
> >
> > Hi everyone,
> >
> > Bumping this thread, as I haven't yet had any replies.
> >
> > Kind regards,
> > Harry
> >
> > On Thursday, 6 June 2024 at 17:59, Harry Fallows
> > harryfallows@protonmail.com.INVALID wrote:
> >
> > > Hi everyone,
> > >
> > > I would like to propose a change to allow the static configuration of leader
> > > and follower replication throttling rates.
> > >
> > > These configurations are very useful for preventing client traffic from
> > > getting throttled by replication traffic during events that cause a spike in
> > > replication. Currently they are only configurable dynamically, which means they
> > > are only really useful for throttling replication traffic during planned
> > > events. By allowing these configurations to be set statically, they can be used
> > > to prevent client traffic throttling during unplanned events.
> > >
> > > KIP:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1051%3A+Statically+configu
> > > red+log+replication+throttling
> > >
> > > Best regards,
> > > Harry Fallows
Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
Posted by Harry Fallows <ha...@protonmail.com.INVALID>.
Hi Hector,
I did see your colleague's KIP, and I actually mentioned it in the KIP that I have written. As I see it, both of these KIPs move towards more easily configurable replication throttling and both should be implemented. KIP-1009 makes it easier to enable throttling and KIP-1051 makes it easier to apply a throttle rate. I did try to look at supporting KIP-1009 in the discussion thread, however, I only subscribed to the mailing list after it was published and I couldn't figure out how to respond to it in Pony mail. I would be definitely be interested in partnering up to get both changes across the line, whether that be by combining them or supporting both individually (I'm not sure which is best, this is my first contribution!).
I also see that KAFKA-10190 is mentioned in KIP-1009 as a related ticket. Coincidentally, I raised a PR to address this bug a couple of days ago (https://github.com/apache/kafka/pull/16280). I think this is also a change that will move towards more easily configurable replication throttling as it allows configuring the throttle rate across the whole cluster via a default value. As far as I understand, this change does not need a KIP though because it is a bugfix (the current behaviour of ignoring the default is unintentional).
Let me know what you think.
Kind regards,
Harry
-------- Original Message --------
On 6/13/24 19:08, Hector Geraldino (BLOOMBERG/ 919 3RD A) <hg...@bloomberg.net> wrote:
> Hi Harry,
>
> A colleague of mine opened KIP-1009: Add Broker-level Throttle Configurations, which aims to achieve the same goal (although from a different angle).
>
> Can you please take a look and see if this would work for the things you have in mind? Maybe we can partner and coalesce around either KIP and try to push it to the end line.
>
> KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1009%3A+Add+Broker-level+Throttle+Configurations
>
> From: dev@kafka.apache.org At: 06/13/24 09:22:40 UTC-4:00To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
>
> Hi everyone,
>
> Bumping this thread, as I haven't yet had any replies.
>
> Kind regards,
> Harry
>
> On Thursday, 6 June 2024 at 17:59, Harry Fallows
> <ha...@protonmail.com.INVALID> wrote:
>
> > Hi everyone,
> >
> > I would like to propose a change to allow the static configuration of leader
> and follower replication throttling rates.
> >
> > These configurations are very useful for preventing client traffic from
> getting throttled by replication traffic during events that cause a spike in
> replication. Currently they are only configurable dynamically, which means they
> are only really useful for throttling replication traffic during planned
> events. By allowing these configurations to be set statically, they can be used
> to prevent client traffic throttling during unplanned events.
> >
> > KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1051%3A+Statically+configu
> red+log+replication+throttling
> >
> > Best regards,
> > Harry Fallows
>
>
>
Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
Posted by "Hector Geraldino (BLOOMBERG/ 919 3RD A)" <hg...@bloomberg.net>.
Hi Harry,
A colleague of mine opened KIP-1009: Add Broker-level Throttle Configurations, which aims to achieve the same goal (although from a different angle).
Can you please take a look and see if this would work for the things you have in mind? Maybe we can partner and coalesce around either KIP and try to push it to the end line.
KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1009%3A+Add+Broker-level+Throttle+Configurations
From: dev@kafka.apache.org At: 06/13/24 09:22:40 UTC-4:00To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
Hi everyone,
Bumping this thread, as I haven't yet had any replies.
Kind regards,
Harry
On Thursday, 6 June 2024 at 17:59, Harry Fallows
<ha...@protonmail.com.INVALID> wrote:
> Hi everyone,
>
> I would like to propose a change to allow the static configuration of leader
and follower replication throttling rates.
>
> These configurations are very useful for preventing client traffic from
getting throttled by replication traffic during events that cause a spike in
replication. Currently they are only configurable dynamically, which means they
are only really useful for throttling replication traffic during planned
events. By allowing these configurations to be set statically, they can be used
to prevent client traffic throttling during unplanned events.
>
> KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1051%3A+Statically+configu
red+log+replication+throttling
>
> Best regards,
> Harry Fallows
Re: [DISCUSS] KIP-1051 Statically configured log replication throttling
Posted by Harry Fallows <ha...@protonmail.com.INVALID>.
Hi everyone,
Bumping this thread, as I haven't yet had any replies.
Kind regards,
Harry
On Thursday, 6 June 2024 at 17:59, Harry Fallows <ha...@protonmail.com.INVALID> wrote:
> Hi everyone,
>
> I would like to propose a change to allow the static configuration of leader and follower replication throttling rates.
>
> These configurations are very useful for preventing client traffic from getting throttled by replication traffic during events that cause a spike in replication. Currently they are only configurable dynamically, which means they are only really useful for throttling replication traffic during planned events. By allowing these configurations to be set statically, they can be used to prevent client traffic throttling during unplanned events.
>
> KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1051%3A+Statically+configured+log+replication+throttling
>
> Best regards,
> Harry Fallows