Disruptor Queue flush behaviour

["Thomas Cooper (PGR)" <t....@newcastle.ac.uk>]

Hi,


I have a question about the behaviour of the LMAX disruptor queues that the executor send/receive and the worker transfer queues use.


These queues batch tuples for processing (100 by default) and will wait until a full batch has arrived before passing them to the executor. However, they will also flush any tuples in the queue periodically (1 ms by default) to prevent the queue blocking for a long time while it waits for 100 tuples to turn up.


My question is about the implementation of the flush interval behaviour:


  1.  Does the flush interval thread run continuously, issuing a flush command every 1 ms and the queue just ignores it if it is already flushing. If 100 tuples turn up between the constant flush commands the queue issues them straight away.
  2.  Or does the flush interval timer only start when consumeBatchWhenAvailable is called on the disruptor queue and a full batch is not available? In which case the queue will wait for 1ms and return whatever is in the queue at the end of that interval or, if 100 tuples turn up within that 1ms, return the full batch.


From the code in storm-core/src/jvm/org/apache/storm/utils/DisruptorQueue.java it seems option 1 might be the case. However, the code in that class is quite complex and the interplay with the underlying LMAX library makes it hard to reason about.


Any help with the above would greatly appreciated, I am attempting to model the effect of these queues on topology performance and hopefully investigate a way to optimise the choice of batch size and flush interval.<http://localhost:7080/github.com/apache/storm@v1.2.2/-/blob/storm-core/src/jvm/org/apache/storm/utils/DisruptorQueue.java?utm_source=share#L300:16>


Thanks,


Thomas Cooper
PhD Student
Newcastle University, School of Computing
W: http://www.tomcooper.org.uk | Twitter: @tomncooper<https://twitter.com/tomncooper>