[jira] [Created] (STORM-2557) A bug in DisruptorQueue causing severe underestimation of queue arrival rates

["tangkailin (JIRA)" <ji...@apache.org>]

tangkailin created STORM-2557:
---------------------------------

             Summary: A bug in DisruptorQueue causing severe underestimation of queue arrival rates
                 Key: STORM-2557
                 URL: https://issues.apache.org/jira/browse/STORM-2557
             Project: Apache Storm
          Issue Type: Bug
            Reporter: tangkailin


Recently, we are tuning the performance of our topology and deploying some theoretical performance models that heavily rely on the metrics of query arrival rates. We found a bug in DisruptorQueue that leads to severe underestimation to the queue arrival rates. After further investigation, we finally found that in the current implementation of DisruptorQueue, the arrival rates are actually measured as the number of batches of tuples rather than the actual number of tuples, resulting in significant underestimation of the arrival rates. 

To be more specific, in DisruptorQueue.publishDirectSingle() and DisruptorQueue.publishDirect() functions, objects containing tuples are published to the buffer and the metrics are notified by calling _metric.notifyArrivals(1). This works fine when the object is simply a wrapper of a single tuple. However, the object could also be an instance of ArrayList<AddressedTuple> or HashMap<Integer, ArrayList<TaskMessage>>. In such case, we should get the actual number of tuples in the object and notify the _metrics with the right value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)