You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by lirop kaykov <li...@gmail.com> on 2023/02/24 15:46:36 UTC
Monitoring Kafka
Hi, I have a question about monitoring kafka.
When using jmx exporter and kafka exporter, you get key metrics like topic
metrics (bytes in per second, messages in per second etc).
Is there a way to get bytes/messages per second per message key?
Meaning, getting the distribution of messages by key.
If not, what is the best way to know how many messages with a specific key
got inside a topic?
Re: Monitoring Kafka
Posted by lirop kaykov <li...@gmail.com>.
I have a system which gathers logs from several clients.
The client logs are produced with a key specifying which client it is.
That way processing logs is guaranteed to be ordered by time per user.
Getting metrics per the topic is good and all, but getting metrics per key
allows you to analyze how many logs there are per client.
Of course one can claim the entity responsible for processing the logs can
expose those kinds of metrics.
But this only gives you the metrics on the already processed logs, instead
of merely reporting how much input per client got in.
I can create a different consumer group only for that purpose, but ideally
I would expect such information to be exposed by the broker itself.
Given the broker already calculates the message rate, doing the same per
key (while limiting the memory of course) wouldn't be unreasonable.
Re: Monitoring Kafka
Posted by Pere Urbón Bayes <pe...@gmail.com>.
Hi Iirop,
question:
> If not, what is the best way to know how many messages with a specific key
got inside a topic?
why do require this information? is this a business metric? or are you
looking at it from an operational point of view?
I'm not aware of any metric that will give you the per-key/partition
distribution, especially from the broker side.
-- Pere
On Fri, Feb 24, 2023 at 4:46 PM lirop kaykov <li...@gmail.com> wrote:
> Hi, I have a question about monitoring kafka.
> When using jmx exporter and kafka exporter, you get key metrics like topic
> metrics (bytes in per second, messages in per second etc).
> Is there a way to get bytes/messages per second per message key?
> Meaning, getting the distribution of messages by key.
> If not, what is the best way to know how many messages with a specific key
> got inside a topic?
>
--
Pere Urbon-Bayes
Software Architect
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/