You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by ka...@apache.org on 2018/04/13 17:16:18 UTC
[1/2] storm git commit: STORM-2908 Document Metrics V2 for 2.0.0
Repository: storm
Updated Branches:
refs/heads/master 4f5fdd928 -> 2ed793d9b
STORM-2908 Document Metrics V2 for 2.0.0
* commit squashed by Jungtaek Lim <ka...@gmail.com>
This closes #2628
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/2c61eba9
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/2c61eba9
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/2c61eba9
Branch: refs/heads/master
Commit: 2c61eba95caa2600e978972fc1749379cc57c5e1
Parents: 4f5fdd9
Author: Aaron Gresch <ag...@yahoo-inc.com>
Authored: Tue Apr 10 08:26:41 2018 -0500
Committer: Jungtaek Lim <ka...@gmail.com>
Committed: Sat Apr 14 02:15:36 2018 +0900
----------------------------------------------------------------------
conf/storm.yaml.example | 26 +++++++-
docs/index.md | 3 +-
docs/metrics_v2.md | 149 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 176 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/2c61eba9/conf/storm.yaml.example
----------------------------------------------------------------------
diff --git a/conf/storm.yaml.example b/conf/storm.yaml.example
index 6d75963..6a8d3c5 100644
--- a/conf/storm.yaml.example
+++ b/conf/storm.yaml.example
@@ -80,4 +80,28 @@
# - class: "org.apache.storm.metric.FileBasedEventLogger"
# - class: "org.mycompany.MyEventLogger"
# arguments:
-# endpoint: "event-logger.mycompany.org"
\ No newline at end of file
+# endpoint: "event-logger.mycompany.org"
+
+# Metrics v2 configuration (optional)
+#storm.metrics.reporters:
+# # Graphite Reporter
+# - class: "org.apache.storm.metrics2.reporters.GraphiteStormReporter"
+# daemons:
+# - "supervisor"
+# - "nimbus"
+# - "worker"
+# report.period: 60
+# report.period.units: "SECONDS"
+# graphite.host: "localhost"
+# graphite.port: 2003
+#
+# # Console Reporter
+# - class: "org.apache.storm.metrics2.reporters.ConsoleStormReporter"
+# daemons:
+# - "worker"
+# report.period: 10
+# report.period.units: "SECONDS"
+# filter:
+# class: "org.apache.storm.metrics2.filters.RegexFilter"
+# expression: ".*my_component.*emitted.*"
+
http://git-wip-us.apache.org/repos/asf/storm/blob/2c61eba9/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index af7c182..b1d881f 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -77,7 +77,8 @@ But small change will not affect the user experience. We will notify the user wh
* [Distributed RPC](Distributed-RPC.html)
* [Transactional topologies](Transactional-topologies.html)
* [Hooks](Hooks.html)
-* [Metrics](Metrics.html)
+* [Metrics (Deprecated)](Metrics.html)
+* [Metrics V2](metrics_v2.html)
* [State Checkpointing](State-checkpointing.html)
* [Windowing](Windowing.html)
* [Joining Streams](Joins.html)
http://git-wip-us.apache.org/repos/asf/storm/blob/2c61eba9/docs/metrics_v2.md
----------------------------------------------------------------------
diff --git a/docs/metrics_v2.md b/docs/metrics_v2.md
new file mode 100644
index 0000000..e90ec76
--- /dev/null
+++ b/docs/metrics_v2.md
@@ -0,0 +1,149 @@
+---
+title: Metrics Reporting API v2
+layout: documentation
+documentation: true
+---
+Apache Storm version 1.2 introduced a new metrics system for reporting
+internal statistics (e.g. acked, failed, emitted, transferred, disruptor queue metrics, etc.) as well as a
+new API for user defined metrics.
+
+The new metrics system is based on [Dropwizard Metrics](http://metrics.dropwizard.io).
+
+
+## User Defined Metrics
+To allow users to define custom metrics, the following methods have been added to the `TopologyContext`
+class, an instance of which is passed to spout's `open()` method and bolt's `prepare()` method:
+
+ public Timer registerTimer(String name)
+
+ public Histogram registerHistogram(String name)
+
+ public Meter registerMeter(String name)
+
+ public Counter registerCounter(String name)
+
+ public Gauge registerGauge(String name, Gauge gauge)
+
+API documentation: [Timer](http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Timer.html),
+[Histogram](http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Histogram.html),
+[Meter](http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Meter.html),
+[Counter](http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Counter.html),
+[Gauge](http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Gauge.html)
+
+Each of these methods takes a `name` parameter that acts as an identifier. When metrics are
+registered, Storm will add additional information such as hostname, port, topology ID, etc. to form a unique metric
+identifier. For example, if we register a metric named `myCounter` as follows:
+
+```java
+ Counter myCounter = topologyContext.registerCounter("myCounter");
+```
+the resulting name sent to metrics reporters will expand to:
+
+```
+ storm.topology.{topology ID}.{hostname}.{component ID}.{task ID}.{worker port}-myCounter
+```
+
+The additional information allows for the unique identification of metrics for component instances across the cluster.
+
+*Important Note:* In order to ensure metric names can be reliably parsed, any `.` characters in name components will
+be replaced with an underscore (`_`) character. For example, the hostname `storm.example.com` will appear as
+`storm_example_com` in the metric name. This character substitution *is not applied to the user-supplied `name` parameter.
+
+### Example: Tuple Counter Bolt
+The following example is a simple bolt implementation that will report the running total up tuples received by a bolt:
+
+```java
+public class TupleCountingBolt extends BaseRichBolt {
+ private Counter tupleCounter;
+ @Override
+ public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
+ this.tupleCounter = context.registerCounter("tupleCount");
+ }
+
+ @Override
+ public void execute(Tuple input) {
+ this.tupleCounter.inc();
+ }
+}
+```
+
+## Metric Reporter Configuration
+
+ For metrics to be useful they must be *reported*, in other words sent somewhere where they can be consumed and analyzed.
+ That can be as simple as writing them to a log file, sending them to a time series database, or exposing them via JMX.
+
+ As of Storm 1.2.0 the following metric reporters are supported
+
+ * Console Reporter (`org.apache.storm.metrics2.reporters.ConsoleStormReporter`):
+ Reports metrics to `System.out`.
+ * CSV Reporter (`org.apache.storm.metrics2.reporters.CsvReporter`):
+ Reports metrics to a CSV file.
+ * Ganglia Reporter (`org.apache.storm.metrics2.reporters.GagliaStormReporter`):
+ Reports metrics to a [Ganglia](http://ganglia.info) server.
+ * Graphite Reporter (`org.apache.storm.metrics2.reporters.GraphiteStormReporter`):
+ Reports metrics to a [Graphite](https://graphiteapp.org) server.
+ * JMX Reporter (`org.apache.storm.metrics2.reporters.JmxStormReporter`):
+ Exposes metrics via JMX.
+
+
+ Metrics reporters are configured in the `storm.yaml` file. By default, Storm will collect metrics but not "report" or
+ send the collected metrics anywhere. To enable metrics reporting, add a `storm.metrics.reporters` section to `storm.yaml`
+ and configure one or more reporters.
+
+ The following example configuration sets up two reporters: a Graphite Reporter and a Console Reporter:
+
+ ```yaml
+storm.metrics.reporters:
+ # Graphite Reporter
+ - class: "org.apache.storm.metrics2.reporters.GraphiteStormReporter"
+ daemons:
+ - "supervisor"
+ - "nimbus"
+ - "worker"
+ report.period: 60
+ report.period.units: "SECONDS"
+ graphite.host: "localhost"
+ graphite.port: 2003
+
+ # Console Reporter
+ - class: "org.apache.storm.metrics2.reporters.ConsoleStormReporter"
+ daemons:
+ - "worker"
+ report.period: 10
+ report.period.units: "SECONDS"
+ filter:
+ class: "org.apache.storm.metrics2.filters.RegexFilter"
+ expression: ".*my_component.*emitted.*"
+
+```
+
+Each reporter section begins with a `class` parameter representing the fully-qualified class name of the reporter
+implementation. The `daemons` section determines which daemons the reporter will apply to (in the example Graphite
+Reporter is configured to report metrics from all Storm daemons, while the Console reporter will only report worker and
+topology metrics).
+
+Many reporter implementations are *scheduled*, meaning they report metrics at regular intervals. The reporting interval
+is determined by the `report.period` and `report.period.units` parameters.
+
+Reporters can also be configured with an optional filter that determines which metrics get reported. Storm includes the
+`org.apache.storm.metrics2.filters.RegexFilter` filter which uses a regular expression to determine which metrics get
+reported. Custom filters can be created by implementing the `org.apache.storm.metrics2.filters.StormMetricFilter`
+interface:
+
+```java
+public interface StormMetricsFilter extends MetricFilter {
+
+ /**
+ * Called after the filter is instantiated.
+ * @param config A map of the properties from the 'filter' section of the reporter configuration.
+ */
+ void prepare(Map<String, Object> config);
+
+ /**
+ * Returns true if the given metric should be reported.
+ */
+ boolean matches(String name, Metric metric);
+
+}
+```
+
[2/2] storm git commit: Merge branch 'STORM-2908-merge'
Posted by ka...@apache.org.
Merge branch 'STORM-2908-merge'
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/2ed793d9
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/2ed793d9
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/2ed793d9
Branch: refs/heads/master
Commit: 2ed793d9ba78afd7311dcdc3b098dea82623e0fa
Parents: 4f5fdd9 2c61eba
Author: Jungtaek Lim <ka...@gmail.com>
Authored: Sat Apr 14 02:16:12 2018 +0900
Committer: Jungtaek Lim <ka...@gmail.com>
Committed: Sat Apr 14 02:16:12 2018 +0900
----------------------------------------------------------------------
conf/storm.yaml.example | 26 +++++++-
docs/index.md | 3 +-
docs/metrics_v2.md | 149 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 176 insertions(+), 2 deletions(-)
----------------------------------------------------------------------