Huge Performance Problem - Slow Bolts

[Martin Illecker <mi...@apache.org>]

Hello,

I'm observing a huge performance problem with my topology.
The topology consists of 5 bolts and two of them are really slow about 60
ms and 30 ms of execute latency. (please see attached UI screenshot)

My topology configuration consists of one worker per node and 19 16-core
nodes, which is a total of 19 workers.
Every worker runs 3 executors, one for each of the three fast bolts, 5
executors for the 60 ms bolt, 3 executors for the 30ms bolt and 3 hidden
threads (acker, worker buffer receive, worker buffer transfer).
This leads to a total number of 14 threads per worker, which is the upper
limit for a 16-core nodes. In my case the CPU is the limiting factor.

With this configuration I could only measure 2000 to 3000 tuples per second
at the end of this pipeline. I have tried multiple MaxSpoutPending settings
but 2000 to 3000 tuples per second seem to be the maximum.
I think, this might be the upper limit for a total execution latency of
100ms?

How can I achieve x0,000 tuples for my topology? Or is this impossible with
these two slow bolts?
Is there anything I could try because horizontal scaling will not solve my
problem.

Thanks!

Best regards
Martin