KIP-922: Add the traffic metric of the partition dimension

[hudeqi <16...@bjtu.edu.cn>]

Hi all,

I have written a new small KIPto add the traffic metric of the partition dimension. The motivation is:

Currently, there are two metrics for measuring the traffic in topic dimensions: MessagesInPerSec, BytesInPerSec, but there are two problems:
1. It is difficult to intuitively reflect the problem of topic partition traffic inclination through these indicators, and it is impossible to clearly see which partition has the largest traffic and the traffic situation of each partition. But the partition dimension can solve this.
2. For the sudden increase in topic traffic (for example, this will lead some followers to out of Isr, which can be avoided by appropriately increasing the number of partitions.), the metrics of the partition dimension can help to provide guidance on whether to expand the partition.

On the whole, I think it is very meaningful to add traffic metrics of partition dimension, especially the issue of traffic skew.

Please take a look here in deatil: https://cwiki.apache.org/confluence/x/LQs0Dw

best,

hudeqi

Re: Re: KIP-922: Add the traffic metric of the partition dimension

[hudeqi <16...@bjtu.edu.cn>]

Hi Edoardo, 
thank you for your reply. However, using log end offset can only numerically reflect the inclination of a topic that has not been expanded partitions. If a topic that has been written for a long time is expanded, the log end offset of the newly expanded partitions will be smaller than the previous partitions, but the traffic of the newly expanded partitions may be larger.

best,
hudeqi

"Edoardo Comar" <edoardlists@gmail.com>写道：
> Hi hudeqi,
> 
> thanks for the KIP.
> 
> For the purpose of monitoring if partitions of a topic are used "fairly",
> the log end offset metric offers a good hint.
> Of course, it only expresses a message count, not bytes, but I find it
> sufficient and do not need the actual throughput per partition.
> Perhaps the KIP could detail a use case for which monitoring different LOE
> rates is not good enough.
> 
> Edoardo
> 
> On Sun, 23 Apr 2023 at 04:54, hudeqi <16...@bjtu.edu.cn> wrote:
> 
> > Hi all,
> >
> > I have written a new small KIPto add the traffic metric of the partition
> > dimension. The motivation is:
> >
> > Currently, there are two metrics for measuring the traffic in topic
> > dimensions: MessagesInPerSec, BytesInPerSec, but there are two problems:
> > 1. It is difficult to intuitively reflect the problem of topic partition
> > traffic inclination through these indicators, and it is impossible to
> > clearly see which partition has the largest traffic and the traffic
> > situation of each partition. But the partition dimension can solve this.
> > 2. For the sudden increase in topic traffic (for example, this will lead
> > some followers to out of Isr, which can be avoided by appropriately
> > increasing the number of partitions.), the metrics of the partition
> > dimension can help to provide guidance on whether to expand the partition.
> >
> > On the whole, I think it is very meaningful to add traffic metrics of
> > partition dimension, especially the issue of traffic skew.
> >
> > Please take a look here in deatil:
> > https://cwiki.apache.org/confluence/x/LQs0Dw
> >
> > best,
> >
> > hudeqi

Re: KIP-922: Add the traffic metric of the partition dimension

[Edoardo Comar <ed...@gmail.com>]

Hi hudeqi,

thanks for the KIP.

For the purpose of monitoring if partitions of a topic are used "fairly",
the log end offset metric offers a good hint.
Of course, it only expresses a message count, not bytes, but I find it
sufficient and do not need the actual throughput per partition.
Perhaps the KIP could detail a use case for which monitoring different LOE
rates is not good enough.

Edoardo

On Sun, 23 Apr 2023 at 04:54, hudeqi <16...@bjtu.edu.cn> wrote:

> Hi all,
>
> I have written a new small KIPto add the traffic metric of the partition
> dimension. The motivation is:
>
> Currently, there are two metrics for measuring the traffic in topic
> dimensions: MessagesInPerSec, BytesInPerSec, but there are two problems:
> 1. It is difficult to intuitively reflect the problem of topic partition
> traffic inclination through these indicators, and it is impossible to
> clearly see which partition has the largest traffic and the traffic
> situation of each partition. But the partition dimension can solve this.
> 2. For the sudden increase in topic traffic (for example, this will lead
> some followers to out of Isr, which can be avoided by appropriately
> increasing the number of partitions.), the metrics of the partition
> dimension can help to provide guidance on whether to expand the partition.
>
> On the whole, I think it is very meaningful to add traffic metrics of
> partition dimension, especially the issue of traffic skew.
>
> Please take a look here in deatil:
> https://cwiki.apache.org/confluence/x/LQs0Dw
>
> best,
>
> hudeqi