You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2015/09/21 22:04:04 UTC
[jira] [Commented] (STORM-350) Update disruptor to latest version
(3.2.1)
[ https://issues.apache.org/jira/browse/STORM-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901295#comment-14901295 ]
ASF GitHub Bot commented on STORM-350:
--------------------------------------
GitHub user revans2 opened a pull request:
https://github.com/apache/storm/pull/750
STORM-350: Upgrade to newer version of disruptor
This upgrades to version 3.3.2 of the Disruptor Queue. There have been questions about stability in the past, and also out of order delivery.
I really wanted to be sure that everything would be about the same. I ran the DisruptorQueue related unit tests over the weekend and got no failures at all, with well over 10,000 runs.
I did some performance tests too using the FastWordCountTopology I added as a part of this. I ran 5 times with the original 0.11.0-SNAPSHOT this is based off of (e85921035fa9bb59d25f0347dc6d26002aac9fab) and with this branch. By setting the topology.max.spout.pending to 200 I got essentially identical results. The new Queue was slightly faster but it was small enough it could just be noise.
Similarly when I did not set topology.max.spout.pending and relied on the automatic throttling I got very similar numbers, although the variance between the runs was much higher so having a real comparison there is much more difficult.
@HeartSaVioR in the past you did some testing to see if out of order delivery was happening, I would love it if you could take a look at this patch and test it similarly.
Anecdotally I have seen this version behave better than the current one we are using. I have seen no NPEs from tuples disappearing and I have seen that show up in some of my stress testing using the old queue. Again I don't know how often this happens so I cannot guarantee that it was a disruptor bug.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/revans2/incubator-storm disruptor-upgrade
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/750.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #750
----
commit 46df80cfc63f01c3a5587a879ce0d721fad3f07f
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Date: 2015-09-18T21:33:39Z
STORM-350: Upgrade to newer version of disruptor
----
> Update disruptor to latest version (3.2.1)
> ------------------------------------------
>
> Key: STORM-350
> URL: https://issues.apache.org/jira/browse/STORM-350
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Boris Aksenov
> Assignee: Boris Aksenov
> Priority: Minor
> Fix For: 0.10.0
>
> Attachments: 20141117-0.9.3-rc1-3-worker-separate-1-spout-and-2-bolts-failing-tuples.png, 20141117-0.9.3-rc1-one-worker-failing-tuples.png, 20141117-0.9.3-rc1-three-workers-1-spout-3-bolts-failing-tuples.png, 20141118-0.9.3-branch-3-worker-separate-1-spout-and-2-bolts-ok.png, 20141118-0.9.3-branch-one-worker-ok.png, 20141118-0.9.3-branch-three-workers-1-spout-3-bolts-ok.png, Storm UI1.pdf, Storm UI2.pdf, storm-0.9.3-rc1-failing-tuples.png, storm-0_9_2-incubating-failing-tuples.png, storm-0_9_2-incubating-no-failing-tuples.png, storm-failed-tuples-multi-node.png, storm-multi-node-without-350.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)