Tuples are not delivered to bolts

[al...@ubs.com]

Hello all, 

 

I have some unclear Storm behavior and need your advice.

 

I have topology like this:

 

                                                   ,->  calculate  --.

      jms -> spout  ->  map  -->  calculate  - - > reduce -> jms

                     |                            `->   calculate --' 

                     |                             |
|

                 local                       local
fields   

                   or                             or
grouping  

                shuffle                 shuffle

 

I have parallelism hint set up > 1 for bolts/spouts.

 

Depends on type of tuple, emitted by spout, I have 'map' bolt executed
different amount of time.  99% of tuples are executed < 200 ms (light),
but 1% could take 40 seconds or more (heavy).

Working on performance optimization, I monitored amount of tuples
emitted by spout  and amount 'map' bolts executed in parallel and found
some strange pattern.

In case of 'heavy' request only one 'map bolt' could be executed, but
other tuples are queued in topology waiting while the 'heavy' one will
finish. Take a look on the attached picture. 

Queued tuples are light tuples.  I have made a thread dump at the moment
of such 'idle' and found that only one 'map bolt' thread is RUNNING,
while all other are in TIMED_WAITING, means Storm didn't deliver tuples
to free bolts.

 

Even worse, I have made the same test monitoring multiple workers, and
found completely the same picture. One worker has one thread executing
'heavy' request, all other workers are doing nothing even if I have
tuples to be processed. 

 

I'm in doubt, why does it happen and need your help.

 

Best regards,

Aleksey.