You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 李海军 <li...@gmail.com> on 2019/08/27 08:07:35 UTC

Metrics of BytesRejectedPerSec keeping rising

Hi all,

We have a kafka cluster consisted of 6 brokers with the version of
kafka_2.11-0.10.2.1. These days we saw a continuous increasing on the
metrics of "BytesRejectedPerSec" on broker 1. I already knew that this
could happen when the message records were too large and raise a
"RecordTooLargeException" on the client side, which could be seen in the
client's log by using "Callback()" in the send method or just adapting the
log level to debug.

However, from the perspective of the server side, there are no logs about
the exceptions at all. Since there are too many producers involved in the
clusters and checking all the client logs is out of the question, how could
I find these certain topics causing "BytesRejectedPerSec" in the server
logs?

Thanks in advance.

Sinsecrely,
Haijun

Re: Metrics of BytesRejectedPerSec keeping rising

Posted by 李海军 <li...@gmail.com>.
Hi Liam,

Thank you for you kind reply. I will check it out to see if this helps.

There are amost 5K topics in our cluster with 6 nodes. We collect the
metrics by collectd (https://collectd.org/) and visualize with Grafana. If
all these per topic metrics are collected, will it affect the performance
of the cluster?

Haijun

Liam Clarke <li...@adscale.co.nz> 于2019年8月27日周二 下午4:15写道:

> Hi,
>
> The broker exposes per topic error rates, that might help?
>
> https://docs.confluent.io/current/kafka/monitoring.html#per-topic-metrics
>
> Kind regards,
>
> Liam Clarke
>
>
> On Tue, 27 Aug. 2019, 8:07 pm 李海军, <li...@gmail.com> wrote:
>
> > Hi all,
> >
> > We have a kafka cluster consisted of 6 brokers with the version of
> > kafka_2.11-0.10.2.1. These days we saw a continuous increasing on the
> > metrics of "BytesRejectedPerSec" on broker 1. I already knew that this
> > could happen when the message records were too large and raise a
> > "RecordTooLargeException" on the client side, which could be seen in the
> > client's log by using "Callback()" in the send method or just adapting
> the
> > log level to debug.
> >
> > However, from the perspective of the server side, there are no logs about
> > the exceptions at all. Since there are too many producers involved in the
> > clusters and checking all the client logs is out of the question, how
> could
> > I find these certain topics causing "BytesRejectedPerSec" in the server
> > logs?
> >
> > Thanks in advance.
> >
> > Sinsecrely,
> > Haijun
> >
>

Re: Metrics of BytesRejectedPerSec keeping rising

Posted by Liam Clarke <li...@adscale.co.nz>.
Hi,

The broker exposes per topic error rates, that might help?

https://docs.confluent.io/current/kafka/monitoring.html#per-topic-metrics

Kind regards,

Liam Clarke


On Tue, 27 Aug. 2019, 8:07 pm 李海军, <li...@gmail.com> wrote:

> Hi all,
>
> We have a kafka cluster consisted of 6 brokers with the version of
> kafka_2.11-0.10.2.1. These days we saw a continuous increasing on the
> metrics of "BytesRejectedPerSec" on broker 1. I already knew that this
> could happen when the message records were too large and raise a
> "RecordTooLargeException" on the client side, which could be seen in the
> client's log by using "Callback()" in the send method or just adapting the
> log level to debug.
>
> However, from the perspective of the server side, there are no logs about
> the exceptions at all. Since there are too many producers involved in the
> clusters and checking all the client logs is out of the question, how could
> I find these certain topics causing "BytesRejectedPerSec" in the server
> logs?
>
> Thanks in advance.
>
> Sinsecrely,
> Haijun
>