High CPU on kafka nodes

[Alexander Filipchik <af...@gmail.com>]

Hello!

I inherited Kafka cluster which runs on AWS (I3.4xl instances).
Each node ingests around 8k messages per second.
Instance stats:
Network in is around: 13 MB/s
Network out is around: 50 MB/s

Kafka stats show slightly different picture:
counter-bytes-in: 2.7 MB/s
counter-bytes-out: 2.9 MB/s

CPU is constantly at 80% (16 cores) .
I'm trying to figure out what can be causing such a high CPU and why kafka
JMX stats are so different from AWS network stats. GC is not looking bad,
the only fishy part is - there are *14000 connections *opened to each
server. Can it be causing high CPU? What is the best way to troubleshoot it?

Thank you,
Alex

Re: High CPU on kafka nodes

[Alexander Filipchik <af...@gmail.com>]

Consumers don't look too busy. Couple java apps and a spark cluster.
Java boxes are not busy, and spark is hard to judge as it does shuffling,
so network is high, but CPU is low.

But thank you for the hint, will check!

Alex

On Thu, Nov 29, 2018 at 10:31 PM Johan Rask <jo...@klipsk.se> wrote:

> What about CPU and load on your consumers? We had a similar problem with
> high cpu and high network traffic on our Kafka cluster and high cpu and
> load and traffic on our consumer machines. Thiis was caused by la to ow
> setting on kafka consumer fetch.max.wait.ms which was only 5 ms. Setting
> this to 100 made huge impact.
>
>
>
> Regards /johan
> fre 30 nov. 2018 kl. 06:59 skrev Suman B N <su...@gmail.com>:
>
> > +1
> > We too face similar issue sometime. I am interested to know more about
> it.
> >
> > On Thursday, November 29, 2018, Alexander Filipchik <
> afilipchik@gmail.com>
> > wrote:
> >
> > > Hello!
> > >
> > > I inherited Kafka cluster which runs on AWS (I3.4xl instances).
> > > Each node ingests around 8k messages per second.
> > > Instance stats:
> > > Network in is around: 13 MB/s
> > > Network out is around: 50 MB/s
> > >
> > > Kafka stats show slightly different picture:
> > > counter-bytes-in: 2.7 MB/s
> > > counter-bytes-out: 2.9 MB/s
> > >
> > > CPU is constantly at 80% (16 cores) .
> > > I'm trying to figure out what can be causing such a high CPU and why
> > kafka
> > > JMX stats are so different from AWS network stats. GC is not looking
> bad,
> > > the only fishy part is - there are *14000 connections *opened to each
> > > server. Can it be causing high CPU? What is the best way to
> troubleshoot
> > > it?
> > >
> > > Thank you,
> > > Alex
> > >
> >
> >
> > --
> > *Suman*
> > *OlaCabs*
> >
>

Re: High CPU on kafka nodes

[Johan Rask <jo...@klipsk.se>]

What about CPU and load on your consumers? We had a similar problem with
high cpu and high network traffic on our Kafka cluster and high cpu and
load and traffic on our consumer machines. Thiis was caused by la to ow
setting on kafka consumer fetch.max.wait.ms which was only 5 ms. Setting
this to 100 made huge impact.



Regards /johan
fre 30 nov. 2018 kl. 06:59 skrev Suman B N <su...@gmail.com>:

> +1
> We too face similar issue sometime. I am interested to know more about it.
>
> On Thursday, November 29, 2018, Alexander Filipchik <af...@gmail.com>
> wrote:
>
> > Hello!
> >
> > I inherited Kafka cluster which runs on AWS (I3.4xl instances).
> > Each node ingests around 8k messages per second.
> > Instance stats:
> > Network in is around: 13 MB/s
> > Network out is around: 50 MB/s
> >
> > Kafka stats show slightly different picture:
> > counter-bytes-in: 2.7 MB/s
> > counter-bytes-out: 2.9 MB/s
> >
> > CPU is constantly at 80% (16 cores) .
> > I'm trying to figure out what can be causing such a high CPU and why
> kafka
> > JMX stats are so different from AWS network stats. GC is not looking bad,
> > the only fishy part is - there are *14000 connections *opened to each
> > server. Can it be causing high CPU? What is the best way to troubleshoot
> > it?
> >
> > Thank you,
> > Alex
> >
>
>
> --
> *Suman*
> *OlaCabs*
>

Re: High CPU on kafka nodes

[Suman B N <su...@gmail.com>]

+1
We too face similar issue sometime. I am interested to know more about it.

On Thursday, November 29, 2018, Alexander Filipchik <af...@gmail.com>
wrote:

> Hello!
>
> I inherited Kafka cluster which runs on AWS (I3.4xl instances).
> Each node ingests around 8k messages per second.
> Instance stats:
> Network in is around: 13 MB/s
> Network out is around: 50 MB/s
>
> Kafka stats show slightly different picture:
> counter-bytes-in: 2.7 MB/s
> counter-bytes-out: 2.9 MB/s
>
> CPU is constantly at 80% (16 cores) .
> I'm trying to figure out what can be causing such a high CPU and why kafka
> JMX stats are so different from AWS network stats. GC is not looking bad,
> the only fishy part is - there are *14000 connections *opened to each
> server. Can it be causing high CPU? What is the best way to troubleshoot
> it?
>
> Thank you,
> Alex
>


-- 
*Suman*
*OlaCabs*