You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2015/10/24 16:20:27 UTC
[jira] [Commented] (STORM-1128) Make Stats+metrics fast
[ https://issues.apache.org/jira/browse/STORM-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972622#comment-14972622 ]
ASF GitHub Bot commented on STORM-1128:
---------------------------------------
GitHub user revans2 opened a pull request:
https://github.com/apache/storm/pull/816
STORM-1128: Make metrics fast
The metrics in my profiling are contributing significantly to the CPU utilization of storm, which slows it down. With this patch I was able to go from 6,500 sentences/second fully counted to 11,000 on a Mac Book Pro running two workers in the Throughput vs. Latency test. If I turn off sub-sampling of metrics the performance goes back to about where it was before. There is still more optimizations that could be done with the metrics, like having the latency metric also report counts, so we don't have to do an extra lookup and inc operation on the critical path, but for now this is good enough, and allows me to start looking at the next bottleneck.
These changes also compliment the batching changes very well. The batching went from 6,500 sentences/second to 15,000 by itself. This patch increases it to 25,000 sentences/second. That is almost a 4x improvement by using both together.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/revans2/incubator-storm fast-counting
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/816.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #816
----
commit e75219b5a4856c39c4031c3d6205105e034d10af
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Date: 2015-10-22T15:48:04Z
STORM-1128: Make metrics fast
----
> Make Stats+metrics fast
> -----------------------
>
> Key: STORM-1128
> URL: https://issues.apache.org/jira/browse/STORM-1128
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
>
> Stats + Built in metrics are slow. If we can make them fast we can can either process more data or not sample as much.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)