You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2014/02/21 07:17:21 UTC
[jira] [Commented] (STORM-200) Proposal for Multilang's Metrics
feature
[ https://issues.apache.org/jira/browse/STORM-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13908005#comment-13908005 ]
ASF GitHub Bot commented on STORM-200:
--------------------------------------
GitHub user dashengju opened a pull request:
https://github.com/apache/incubator-storm/pull/38
STORM-200 Proposal for Multilang's Metrics feature
for https://issues.apache.org/jira/browse/STORM-200
Storm 0.9.0.1 exposes a metrics interface to report summary statistics across the full topology. We can build our own metric, and build metrics consumer to use those statistics.
But when we use Multilang(ie. Python), we can not use this feature. So we want to summit a proposal for multilang's metrics.
The specifics of the proposal:
1. The main idea is: when user want to add a metric statistics in multilang(python) bolt，
a) he need first create a metric object and register in ShellBolt's sub-class,
b) then update the metric in Python bolt process through RPC call.
2. In Metrics API:
a) extend IMetric interface add a method for RPC call：
public void updateMetricFromRPC(List<Object> params);
b) modify IMetric implements, to support updateMetricFromRPC;
3. In ShellBolt，
a) we have a Map<String, IMetric> to hold user's registered metrics object；
b) we have a method registerMetric(String name, T metric) for user to register their metrics object;
c) we have a method handleMetrics(Map action) to handle RPC call from Python process；
4) In Multilang protocol: add a command "metrics" for shell process to make RPC call. The protocol is: {"command":"metrics", "name":"metric-registerd-name", "params":["param-1", param-2]}
5) In storm.py：add rpcMetrics(name, params), user can update remote metric through this RPC call.
--------ExamplePythonBolt.java----------------------------------------------------------------------------------------------------
public class ExamplePythonBolt extends ShellBolt implements IRichBolt {
private static final long serialVersionUID = 1999209252187463355L;
private TopologyConfig topologyConfig;
public ExamplePythonBolt(TopologyConfig config) {
super("python", "ExampleBolt.py", config.topologyName);
topologyConfig = config;
}
public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
super.prepare(stormConf, context, collector);
CountMetric cMetric = new CountMetric(); //generate a metric object
context.registerMetric("PythonBoltCount", cMetric, 120); //register metric object to context
this.registerMetric("PythonBoltCount", cMetric); //register metric object to ShellBolt
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("LogRecord"));
}
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
--------ExampleBolt.py---------------------------------------------------------------------------------------------------
class ExampleBolt(storm.BasicBolt):
def __init__(self, boltParams):
#from here, you can get params from java
pass
def initialize(self, stormconf, context):
pass
def process(self, tup):
try:
MTLOGGER.info("TestAction", "handle the json %s" % (logjson))
storm.emit([tup[0]])
storm.rpcMetrics("PythonBoltCount", [1]) //update your PythonBoltCount metric with param list "[1]"
except Exception,tx:
//handle error
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dashengju/incubator-storm master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-storm/pull/38.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #38
----
commit edadde30420e5b2cd3e3d248a6a2268f8dedc22f
Author: JuDasheng <ju...@meituan.com>
Date: 2014-01-28T06:47:30Z
modify metrics system to support shellbolt's metrics
commit 52a4fbbd670cfac3a9174bd6c7ad63d3a2282fa8
Author: JuDasheng <ju...@meituan.com>
Date: 2014-02-21T06:02:28Z
modify for metrics
----
> Proposal for Multilang's Metrics feature
> ----------------------------------------
>
> Key: STORM-200
> URL: https://issues.apache.org/jira/browse/STORM-200
> Project: Apache Storm (Incubating)
> Issue Type: New Feature
> Reporter: DashengJu
> Priority: Minor
>
> Storm 0.9.0.1 exposes a metrics interface to report summary statistics across the full topology. We can build our own metric, and build metrics consumer to use those statistics.
> But when we use Multilang(ie. Python), we can not use this feature. So we want to summit a proposal for multilang's metrics.
> The specifics of the proposal:
> 1. The main idea is: when user want to add a metric statistics in multilang(python) bolt，
> a) he need first create a metric object and register in ShellBolt's sub-class,
> b) then update the metric in Python bolt process through RPC call.
> 2. In Metrics API:
> a) extend IMetric interface add a method for RPC call：public void updateMetricFromRPC(List<Object> params);
> b) modify IMetric implements, to support updateMetricFromRPC;
> 3. In ShellBolt，
> a) we have a Map<String, IMetric> to hold user's registered metrics object；
> b) we have a method registerMetric(String name, T metric) for user to register their metris object;
> c) we have a method handleMetrics(Map action) to handle RPC call from Python process；
> 4) In Multilang protocol: add a command "metrics" for shell process to make RPC call. The protocol is: {"command":"metrics", "name":"metric-registerd-name", "params":["param-1", param-2]}
> 5) In storm.py：add rpcMetrics(name, params), user can update remote metric through this RPC call.
> any suggestions?
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)