High CPU Usage on 0.8.2.1

[Craig Ching <cr...@gmail.com>]

Hi all,

We're experiencing high cpu usage on some of our development machines when
kafka should be idle.  These are single node kafka instances used only for
development, so far we haven't seen any problems on our production system.

It doesn't seem to happen until we deploy a storm topology and the more
topologies we deploy, the worse it gets.  And the higher the partitions for
the topics, the worse it gets.

For instance, on my system if I change the number of partitions from 3 to 1
for ~10 topics, I experience a decrease in CPU usage from ~89% to ~50%

I'll admit, I'm still getting up to speed on kafka, so maybe having 3
partitions on our development machines is crazy, but, still, even 1
partition causes this excessive CPU.

I have profiled using jvisualvm and the hotspot seems to be
RequestChannel.receiveRequest().  I've tinkered with the code a bit, but
haven't been able to affect any change.

Does anyone have any experience with such a scenario?  Is there anything I
can provide that would help troubleshoot this?

I appreciate any help!

Cheers,
Craig

Re: High CPU Usage on 0.8.2.1

[Craig Ching <cr...@gmail.com>]

So I found the culprit.  On the storm-kafka spout there is a setting:

topology.sleep.spout.wait.strategy.time.ms

this setting defaults to 1ms.  Changing it dramatically decreases CPU in
kafka and storm.  Does anyone here know anything about this?  Should this
be able to peg Kafka's CPU usage?  I'm also going to ask on the storm list
about this setting.

Cheers,
Craig

On Thu, Jul 16, 2015 at 8:41 AM, Craig Ching <cr...@gmail.com> wrote:

> Hi all,
>
> We're experiencing high cpu usage on some of our development machines when
> kafka should be idle.  These are single node kafka instances used only for
> development, so far we haven't seen any problems on our production system.
>
> It doesn't seem to happen until we deploy a storm topology and the more
> topologies we deploy, the worse it gets.  And the higher the partitions for
> the topics, the worse it gets.
>
> For instance, on my system if I change the number of partitions from 3 to
> 1 for ~10 topics, I experience a decrease in CPU usage from ~89% to ~50%
>
> I'll admit, I'm still getting up to speed on kafka, so maybe having 3
> partitions on our development machines is crazy, but, still, even 1
> partition causes this excessive CPU.
>
> I have profiled using jvisualvm and the hotspot seems to be
> RequestChannel.receiveRequest().  I've tinkered with the code a bit, but
> haven't been able to affect any change.
>
> Does anyone have any experience with such a scenario?  Is there anything I
> can provide that would help troubleshoot this?
>
> I appreciate any help!
>
> Cheers,
> Craig
>