High CPU Usage while processing tuples

[Indra Nath Bardhan <in...@gmail.com>]

We have been using Storm to port over our legacy ingestion framework to
Storm.

However, we have been seeing a very high CPU usage (~ 95%) everytime tuples
are being processed. This happens even under a very low input to Spout (200
per/s) The idle CPU usage remains well below ~10%.


*Below are my m/c configuration: 24 core box, and we see almost all cores
max'ed out during high CPU*
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                24
On-line CPU(s) list:   0-23
Thread(s) per core:    2
Core(s) per socket:    6
CPU socket(s):         2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 45

Both supervisor and workers run on the same m/c.

Running Java flight recorder indicates that it is spending a lot of time,
inside the BlockWaitingStrategy.waitFor() method. Below is a snapshot.Many
of the threads in "storm-user" also mention the same issue.

[image: Inline image 1]


Below are the configs with which I tried.

"topology.worker.childopts": "-Xmx2048m -Xms2048m",

"topology.executor.send.buffer.size": 1024,

"topology.executor.receive.buffer.size": 1024,

"topology.receiver.buffer.size": 8,

"topology.transfer.buffer.size": 32,

"topology.sleep.spout.wait.strategy.time.ms": 1000



   - Already tried increasing number of workers, but the same CPU starving
   occurs.
   - Also tried a high executor buffer size i.e., 262144 assuming that the
   executor buffers are getting filled up, due to which we have waitFor()
   calls.
   - Note that the bolts are processing intensive with a fair socket i/o,
   but we ideally do not expect the CPU to spike so much for the processing,
   and as indicated by the FlightRecorder graph almost 72% of method calls are
   for the BlockWaitingStrategy.waitFor() call.


Please suggest how we can fix this.

Thanks,
Indra