You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ivan Yurchenko <iv...@ivanyu.me> on 2026/06/22 11:13:50 UTC
[DISCUSS] KIP-1355: In-rack traffic observability for clients and brokers
Hi team,
I'd like to start the discussion for a new KIP-1355: In-rack traffic observability for clients and brokers [1] The KIP is small, just to add some additional metrics. Thanks to Chia-Ping Tsai for the idea, proposed as a follow-up to KIP-1123: Rack-aware partitioning for Kafka Producer [2].
Best regard,
Ivan Yurchenko
[1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-1355%3A+In-rack+traffic+observability+for+clients+and+brokers
[2] https://cwiki.apache.org/confluence/display/KAFKA/KIP-1123:+Rack-aware+partitioning+for+Kafka+Producer
Re: [DISCUSS] KIP-1355: In-rack traffic observability for clients and brokers
Posted by Ivan Yurchenko <iv...@ivanyu.me>.
Hi all,
Thank you for your comments.
chia_00:
I wanted to keep the changes to the scope of the KIP, but sure we can close this gap.
chia_01:
We have the total bytes bytes-consumed-rate and are adding bytes-consumed-in-rack-rate. So technically one can do `bytes-consumed-rate - bytes-consumed-in-rack-rate` to get what goes outside of the rack. Or do you mean to explicitly mark the traffic that the operator wanted to be in rack, but the client couldn't figure out the rack?
svv1:
It can be done, but we should be aware that it's asymmetrical: Produce request don't carry the rack information now.
svv2:
So basically replace "byte-in-rack-rate" with "byte-cross-rack-rate" and so on? If we scope it to "traffic that goes to a _known_ rack, different from ours", IMO the usefulness will be quite limited, because racks may be just not configured. To make it work, we need to give it the meaning of "traffic that goes to a _known_ rack, different from ours, or to an unknown rack".
Best,
Ivan
On Wed, Jul 15, 2026, at 17:01, Viktor Somogyi-Vass via dev wrote:
> Hi Ivan,
>
> My comments:
>
> svv1. Should we add a BytesOutRackAlignedPerSec that is similar to
> BytesOutPerSec but counts only the rack-aligned traffic? Since we have the
> rack IDs in fetch requests it may be easy to calculate it and might be
> higher value if client metrics aren't available.
>
> svv2. Operators I guess would mostly care about cross-rack traffic for cost
> reasons. Have you considered computing directly that instead or
> additionally? Also "CrossRack" would also resolve the naming complication
> that was proposed in AS1.
>
> Best,
> Viktor
>
> On Fri, Jul 3, 2026 at 11:19 PM Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > hi Ivan
> >
> > chia_00: Should we also add
> > "kafka.producer:type=producer-metrics,client-id=<id>" for the producer? The
> > proposed consumer metrics have both levels (client vs. client+topic), but
> > the producer currently only has the topic level.
> >
> > chia_01: Should we provide "unknown-rack-bytes" for the misconfiguration?
> >
> > Best,
> > Chia-Ping
> >
> > On 2026/06/22 11:13:50 Ivan Yurchenko wrote:
> > > Hi team,
> > >
> > > I'd like to start the discussion for a new KIP-1355: In-rack traffic
> > observability for clients and brokers [1] The KIP is small, just to add
> > some additional metrics. Thanks to Chia-Ping Tsai for the idea, proposed as
> > a follow-up to KIP-1123: Rack-aware partitioning for Kafka Producer [2].
> > >
> > > Best regard,
> > > Ivan Yurchenko
> > >
> > > [1]
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1355%3A+In-rack+traffic+observability+for+clients+and+brokers
> > > [2]
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1123:+Rack-aware+partitioning+for+Kafka+Producer
> >
>
Re: [DISCUSS] KIP-1355: In-rack traffic observability for clients and brokers
Posted by Viktor Somogyi-Vass via dev <de...@kafka.apache.org>.
Hi Ivan,
My comments:
svv1. Should we add a BytesOutRackAlignedPerSec that is similar to
BytesOutPerSec but counts only the rack-aligned traffic? Since we have the
rack IDs in fetch requests it may be easy to calculate it and might be
higher value if client metrics aren't available.
svv2. Operators I guess would mostly care about cross-rack traffic for cost
reasons. Have you considered computing directly that instead or
additionally? Also "CrossRack" would also resolve the naming complication
that was proposed in AS1.
Best,
Viktor
On Fri, Jul 3, 2026 at 11:19 PM Chia-Ping Tsai <ch...@apache.org> wrote:
> hi Ivan
>
> chia_00: Should we also add
> "kafka.producer:type=producer-metrics,client-id=<id>" for the producer? The
> proposed consumer metrics have both levels (client vs. client+topic), but
> the producer currently only has the topic level.
>
> chia_01: Should we provide "unknown-rack-bytes" for the misconfiguration?
>
> Best,
> Chia-Ping
>
> On 2026/06/22 11:13:50 Ivan Yurchenko wrote:
> > Hi team,
> >
> > I'd like to start the discussion for a new KIP-1355: In-rack traffic
> observability for clients and brokers [1] The KIP is small, just to add
> some additional metrics. Thanks to Chia-Ping Tsai for the idea, proposed as
> a follow-up to KIP-1123: Rack-aware partitioning for Kafka Producer [2].
> >
> > Best regard,
> > Ivan Yurchenko
> >
> > [1]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1355%3A+In-rack+traffic+observability+for+clients+and+brokers
> > [2]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1123:+Rack-aware+partitioning+for+Kafka+Producer
>
Re: [DISCUSS] KIP-1355: In-rack traffic observability for clients and brokers
Posted by Chia-Ping Tsai <ch...@apache.org>.
hi Ivan
chia_00: Should we also add "kafka.producer:type=producer-metrics,client-id=<id>" for the producer? The proposed consumer metrics have both levels (client vs. client+topic), but the producer currently only has the topic level.
chia_01: Should we provide "unknown-rack-bytes" for the misconfiguration?
Best,
Chia-Ping
On 2026/06/22 11:13:50 Ivan Yurchenko wrote:
> Hi team,
>
> I'd like to start the discussion for a new KIP-1355: In-rack traffic observability for clients and brokers [1] The KIP is small, just to add some additional metrics. Thanks to Chia-Ping Tsai for the idea, proposed as a follow-up to KIP-1123: Rack-aware partitioning for Kafka Producer [2].
>
> Best regard,
> Ivan Yurchenko
>
> [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-1355%3A+In-rack+traffic+observability+for+clients+and+brokers
> [2] https://cwiki.apache.org/confluence/display/KAFKA/KIP-1123:+Rack-aware+partitioning+for+Kafka+Producer
Re: [DISCUSS] KIP-1355: In-rack traffic observability for clients and brokers
Posted by Ivan Yurchenko <iv...@ivanyu.me>.
Hi Andrew,
Thank you for the reply.
AS1 totally makes sense, I was in search of a better name :)
Best,
Ivan
On Mon, Jun 22, 2026, at 19:36, Andrew Schofield wrote:
> Hi Ivan,
> Thanks for the KIP. Looks like a nice idea. Just one comment.
>
> AS1: I wonder whether "rack-aligned" rather than "in-rack" might be nicer. This avoids the various InIn, OutIn and so on in the metric names proposed.
>
> Thanks,
> Andrew
>
> On 2026/06/22 11:13:50 Ivan Yurchenko wrote:
> > Hi team,
> >
> > I'd like to start the discussion for a new KIP-1355: In-rack traffic observability for clients and brokers [1] The KIP is small, just to add some additional metrics. Thanks to Chia-Ping Tsai for the idea, proposed as a follow-up to KIP-1123: Rack-aware partitioning for Kafka Producer [2].
> >
> > Best regard,
> > Ivan Yurchenko
> >
> > [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-1355%3A+In-rack+traffic+observability+for+clients+and+brokers
> > [2] https://cwiki.apache.org/confluence/display/KAFKA/KIP-1123:+Rack-aware+partitioning+for+Kafka+Producer
>
Re: [DISCUSS] KIP-1355: In-rack traffic observability for clients and brokers
Posted by Andrew Schofield <sc...@apache.org>.
Hi Ivan,
Thanks for the KIP. Looks like a nice idea. Just one comment.
AS1: I wonder whether "rack-aligned" rather than "in-rack" might be nicer. This avoids the various InIn, OutIn and so on in the metric names proposed.
Thanks,
Andrew
On 2026/06/22 11:13:50 Ivan Yurchenko wrote:
> Hi team,
>
> I'd like to start the discussion for a new KIP-1355: In-rack traffic observability for clients and brokers [1] The KIP is small, just to add some additional metrics. Thanks to Chia-Ping Tsai for the idea, proposed as a follow-up to KIP-1123: Rack-aware partitioning for Kafka Producer [2].
>
> Best regard,
> Ivan Yurchenko
>
> [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-1355%3A+In-rack+traffic+observability+for+clients+and+brokers
> [2] https://cwiki.apache.org/confluence/display/KAFKA/KIP-1123:+Rack-aware+partitioning+for+Kafka+Producer