Consumer vs replication traffic priority

[Łukasz Mierzwa <l....@gmail.com>]

Hi,

looking at docs I see that Kafka seems to  support throttling of
consumer/replication traffic, but I can't find anything that would suggest
you can prioritize one traffic type over another.
The problem: if at some point consumers starts to be lagging they will
start consuming messages as fast as they can, if there's enough lag to
recover from and if there's enough affected consumers that traffic can
easily saturate network on the leader, which can in turn affect replicas of
the partitions on that leader.
Is there a way to avoid replicas falling out of sync with the leader under
such scenario? Is there a way to prefer replication traffic over consumer
traffic? Or is throttling the only way to achieve this? Throttling solves
this, but it requires setting limits that can change over time, so it's bit
more manual and requires more maintenance than priority based solution.

-- 
Łukasz Mierzwa

Re: Consumer vs replication traffic priority

[Sam Pegler <sa...@infectiousmedia.com>]

Have you checked the documentation on quotas?

https://kafka.apache.org/documentation/#design_quotas

This should enable you to do what you're after.

__

Sam Pegler

PRODUCTION ENGINEER

T. +44(0) 07 562 867 486

<http://www.infectiousmedia.com/>
3-7 Herbal Hill / London / EC1R 5EJ
www.infectiousmedia.com

This email and any attachments are confidential and may also be privileged.
If you
are not the intended recipient, please notify the sender immediately, and
do not
disclose the contents to another person, use it for any purpose, or store,
or copy
the information in any medium. Please also destroy and delete the message
from
your computer.


On 10 July 2017 at 19:27, Łukasz Mierzwa <l....@gmail.com> wrote:

> Hi,
>
> looking at docs I see that Kafka seems to  support throttling of
> consumer/replication traffic, but I can't find anything that would suggest
> you can prioritize one traffic type over another.
> The problem: if at some point consumers starts to be lagging they will
> start consuming messages as fast as they can, if there's enough lag to
> recover from and if there's enough affected consumers that traffic can
> easily saturate network on the leader, which can in turn affect replicas of
> the partitions on that leader.
> Is there a way to avoid replicas falling out of sync with the leader under
> such scenario? Is there a way to prefer replication traffic over consumer
> traffic? Or is throttling the only way to achieve this? Throttling solves
> this, but it requires setting limits that can change over time, so it's bit
> more manual and requires more maintenance than priority based solution.
>
> --
> Łukasz Mierzwa
>