You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Ethanlm <gi...@git.apache.org> on 2017/09/19 14:24:16 UTC
[GitHub] storm pull request #2331: [STORM-2743] Add logging to monitor how long sched...
GitHub user Ethanlm opened a pull request:
https://github.com/apache/storm/pull/2331
[STORM-2743] Add logging to monitor how long scheduling is taking
A trivial improvement: Add logging to monitor how long scheduling is taking
https://issues.apache.org/jira/browse/STORM-2743
A test:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Ethanlm/storm STORM-2743
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/2331.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2331
----
commit 23786af8af6981332617c578ba7ede8dfc6a643e
Author: Ethan Li <et...@gmail.com>
Date: 2017-09-19T14:11:00Z
[STORM-2743] Add logging to monitor how long scheduling is taking
----
---
[GitHub] storm issue #2331: [STORM-2743] Add logging and metrics to monitor how long ...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2331
+1
---
[GitHub] storm issue #2331: [STORM-2743] Add logging to monitor how long scheduling i...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2331
@Ethanlm
I'm OK to have two metrics and IMO that is right way since we measure two different things, one for topology count, another one for topology schedule delay.
---
[GitHub] storm pull request #2331: [STORM-2743] Add logging and metrics to monitor ho...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/2331
---
[GitHub] storm issue #2331: [STORM-2743] Add logging to monitor how long scheduling i...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/2331
+1 and agree @jerrypeng that better to 'also' record with histogram via StormMetricsRegistry.registerHistogram() like Nimbus records for other operations with Meter.
---
[GitHub] storm issue #2331: [STORM-2743] Add logging to monitor how long scheduling i...
Posted by jerrypeng <gi...@git.apache.org>.
Github user jerrypeng commented on the issue:
https://github.com/apache/storm/pull/2331
+1 but perhaps its work while looking into using something like dropwizard metrics:
http://metrics.dropwizard.io/3.2.3/manual/core.html
---
[GitHub] storm issue #2331: [STORM-2743] Add logging to monitor how long scheduling i...
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2331
@jerrypeng @HeartSaVioR Thanks for the review. Just one question, do we use two Histograms one for the number of the topology and the other one for the time the scheduling takes? I didn't find any way to combine these two together in one Histogram. But I think we might not want to deal with these two separately.
---
[GitHub] storm issue #2331: [STORM-2743] Add logging and metrics to monitor how long ...
Posted by Ethanlm <gi...@git.apache.org>.
Github user Ethanlm commented on the issue:
https://github.com/apache/storm/pull/2331
A test:
I monitored the metrics through JMX. And after a while, I submitted a topology.
---
[GitHub] storm issue #2331: [STORM-2743] Add logging to monitor how long scheduling i...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/2331
@Ethanlm I am fine with having 2 histograms, but quite honestly the number of topologies is not that interesting or helpful and I would rather drop it.
The number of topologies has relatively little impact on how long scheduling takes, because for the most part we are going to cache the results, and just skip over topologies that are already scheduled.
For me I would like to see two histograms. One for how long it took to schedule all of the topologies in the loop. This indicates how responsive we are likely to be when a worker crashes and it needs to be rescheduled or when a topology is launched and it needs to be scheduled the first time.
The second one I would be interested in is a histogram for how long it took to schedule each individual topology. Sadly that one is much harder, because it would have to be something each scheduler reported on their own.
---