You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by pt...@apache.org on 2016/04/21 20:46:37 UTC
[1/3] storm git commit: [STORM-1544] Document Debug/Sampling of
Topologies
Repository: storm
Updated Branches:
refs/heads/1.x-branch 8b3dff85c -> ed2848fff
[STORM-1544] Document Debug/Sampling of Topologies
Added documentation for logging/sampling events in a topology.
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/c6d0bc4e
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/c6d0bc4e
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/c6d0bc4e
Branch: refs/heads/1.x-branch
Commit: c6d0bc4ea3ba630dde0c09848cce08170fdac2d7
Parents: 8b3dff8
Author: Arun Mahadevan <ai...@hortonworks.com>
Authored: Mon Mar 28 14:19:31 2016 +0530
Committer: P. Taylor Goetz <pt...@gmail.com>
Committed: Thu Apr 21 14:45:33 2016 -0400
----------------------------------------------------------------------
docs/Eventlogging.md | 93 ++++++++++++++++++++
docs/images/disable-event-logging-topology.png | Bin 0 -> 128737 bytes
docs/images/enable-event-logging-spout.png | Bin 0 -> 136954 bytes
docs/images/enable-event-logging-topology.png | Bin 0 -> 161055 bytes
docs/images/event-logs-view.png | Bin 0 -> 192749 bytes
docs/index.md | 1 +
6 files changed, 94 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/c6d0bc4e/docs/Eventlogging.md
----------------------------------------------------------------------
diff --git a/docs/Eventlogging.md b/docs/Eventlogging.md
new file mode 100644
index 0000000..d864366
--- /dev/null
+++ b/docs/Eventlogging.md
@@ -0,0 +1,93 @@
+---
+title: Topology event inspector
+layout: documentation
+documentation: true
+---
+
+# Introduction
+
+Topology event inspector provides the ability to view the tuples as it flows through different stages in a storm topology.
+This could be useful for inspecting the tuples emitted at a spout or a bolt in the topology pipeline while the topology is running, without stopping or redeploying the topology. The normal flow of tuples from the spouts to the bolts is not affected by turning on event logging.
+
+## Enabling event logging
+
+Event logging can be enabled by clicking the "Debug" button under the topology actions in the topology view. This logs the
+tuples from all the spouts and bolts in a topology at the specified sampling percentage.
+
+<div align="center">
+<img title="Enable Eventlogging" src="images/enable-event-logging-topology.png" style="max-width: 80rem"/>
+
+<p>Figure 1: Enable event logging at topology level.</p>
+</div>
+
+You could also enable event logging at a specific spout or bolt level by going to the corresponding component page and
+clicking "Debug" under component actions.
+
+<div align="center">
+<img title="Enable Eventlogging at component level" src="images/enable-event-logging-spout.png" style="max-width: 80rem"/>
+
+<p>Figure 2: Enable event logging at component level.</p>
+</div>
+
+## Viewing the event logs
+The Storm "logviewer" should be running for viewing the logged tuples. If not already running log viewer can be started by running the "bin/storm logviewer" command from the storm installation directory. For viewing the tuples, go to the specific spout or bolt component page from storm UI and click on the "events" link under the component summary (as highlighted in Figure 2 above).
+
+This would open up a view like below where you can navigate between different pages and view the logged tuples.
+
+<div align="center">
+<img title="Viewing logged tuples" src="images/event-logs-view.png" style="max-width: 80rem"/>
+
+<p>Figure 3: Viewing the logged events.</p>
+</div>
+
+Each line in the event log contains an entry corresponding to a tuple emitted from a specific spout/bolt in a comma separated format.
+
+`Timestamp, Component name, Component task-id, MessageId (in case of anchoring), List of emitted values`
+
+## Disabling the event logs
+
+Event logging can be disabled at a specific component or at the topology level by clicking the "Stop Debug" under the topology or component actions in the Storm UI.
+
+<div align="center">
+<img title="Disable Eventlogging at topology level" src="images/disable-event-logging-topology.png" style="max-width: 80rem"/>
+
+<p>Figure 4: Disable event logging at topology level.</p>
+</div>
+
+## Configuration
+Eventlogging works by sending the events (tuples) from each component to an internal eventlogger bolt. By default storm automatically starts one event logger task per worker. This can be easily changed by setting the below parameter while running your topology (by setting it in storm.yaml or passing options via command line).
+
+| Parameter | Meaning |
+| -------------------------------------------|-----------------------|
+| "topology.eventlogger.executors": nil | One event logger task per worker (default). |
+| "topology.eventlogger.executors": 1 | One event logger task for the topology. |
+| "topology.eventlogger.executors": 0 | No event logger tasks are created. |
+
+
+## Extending eventlogging
+Storm provides an `IEventLogger` interface which is used by the event logger bolt to log the events. The default implementation for this is a FileBasedEventLogger which logs the events to an events.log file ( `logs/workers-artifacts/<topology-id>/<worker-port>/events.log`). Alternate implementations of the `IEventLogger` interface can be added to extend the event logging functionality (say build a search index or log the events in a database etc)
+
+```java
+/**
+ * EventLogger interface for logging the event info to a sink like log file or db
+ * for inspecting the events via UI for debugging.
+ */
+public interface IEventLogger {
+ /**
+ * Invoked during eventlogger bolt prepare.
+ */
+ void prepare(Map stormConf, TopologyContext context);
+
+ /**
+ * Invoked when the {@link EventLoggerBolt} receives a tuple from the spouts or bolts that has event logging enabled.
+ *
+ * @param e the event
+ */
+ void log(EventInfo e);
+
+ /**
+ * Invoked when the event logger bolt is cleaned up
+ */
+ void close();
+}
+```
http://git-wip-us.apache.org/repos/asf/storm/blob/c6d0bc4e/docs/images/disable-event-logging-topology.png
----------------------------------------------------------------------
diff --git a/docs/images/disable-event-logging-topology.png b/docs/images/disable-event-logging-topology.png
new file mode 100644
index 0000000..77405ed
Binary files /dev/null and b/docs/images/disable-event-logging-topology.png differ
http://git-wip-us.apache.org/repos/asf/storm/blob/c6d0bc4e/docs/images/enable-event-logging-spout.png
----------------------------------------------------------------------
diff --git a/docs/images/enable-event-logging-spout.png b/docs/images/enable-event-logging-spout.png
new file mode 100644
index 0000000..081bab1
Binary files /dev/null and b/docs/images/enable-event-logging-spout.png differ
http://git-wip-us.apache.org/repos/asf/storm/blob/c6d0bc4e/docs/images/enable-event-logging-topology.png
----------------------------------------------------------------------
diff --git a/docs/images/enable-event-logging-topology.png b/docs/images/enable-event-logging-topology.png
new file mode 100644
index 0000000..dc8ee58
Binary files /dev/null and b/docs/images/enable-event-logging-topology.png differ
http://git-wip-us.apache.org/repos/asf/storm/blob/c6d0bc4e/docs/images/event-logs-view.png
----------------------------------------------------------------------
diff --git a/docs/images/event-logs-view.png b/docs/images/event-logs-view.png
new file mode 100644
index 0000000..104f35c
Binary files /dev/null and b/docs/images/event-logs-view.png differ
http://git-wip-us.apache.org/repos/asf/storm/blob/c6d0bc4e/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index 67da582..7effdde 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -63,6 +63,7 @@ Trident is an alternative interface to Storm. It provides exactly-once processin
* [Dynamic Log Level Settings](dynamic-log-level-settings.html)
* [Searching Worker Logs](Logs.html)
* [Worker Profiling](dynamic-worker-profiling.html)
+* [Event Logging](Eventlogging.html)
### Integration With External Systems, and Other Libraries
* [Apache Kafka Integration](storm-kafka.html)
[3/3] storm git commit: add STORM-1544 to changelog
Posted by pt...@apache.org.
add STORM-1544 to changelog
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/ed2848ff
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/ed2848ff
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/ed2848ff
Branch: refs/heads/1.x-branch
Commit: ed2848fff73013a5a0f574d4cdfc1da6b7f947ea
Parents: 8ee43e7
Author: P. Taylor Goetz <pt...@gmail.com>
Authored: Thu Apr 21 14:46:24 2016 -0400
Committer: P. Taylor Goetz <pt...@gmail.com>
Committed: Thu Apr 21 14:46:24 2016 -0400
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/ed2848ff/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c734f27..8898e3c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
## 1.0.1
+ * STORM-1544: Document Debug/Sampling of Topologies
* STORM-1679: add storm Scheduler documents
* STORM-1704: When logviewer_search.html opens daemon file, next search always show no result
* STORM-1714: StatefulBolts ends up as normal bolts while using TopologyBuilder.setBolt without parallelism
[2/3] storm git commit: Updating the doc to reflect the new default
behavior
Posted by pt...@apache.org.
Updating the doc to reflect the new default behavior
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/8ee43e73
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/8ee43e73
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/8ee43e73
Branch: refs/heads/1.x-branch
Commit: 8ee43e7373e54583d7e9c980763df572e06d9aa7
Parents: c6d0bc4
Author: Arun Mahadevan <ai...@hortonworks.com>
Authored: Fri Apr 1 09:56:10 2016 +0530
Committer: P. Taylor Goetz <pt...@gmail.com>
Committed: Thu Apr 21 14:45:51 2016 -0400
----------------------------------------------------------------------
docs/Eventlogging.md | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/8ee43e73/docs/Eventlogging.md
----------------------------------------------------------------------
diff --git a/docs/Eventlogging.md b/docs/Eventlogging.md
index d864366..0895232 100644
--- a/docs/Eventlogging.md
+++ b/docs/Eventlogging.md
@@ -11,7 +11,10 @@ This could be useful for inspecting the tuples emitted at a spout or a bolt in t
## Enabling event logging
-Event logging can be enabled by clicking the "Debug" button under the topology actions in the topology view. This logs the
+Note: Event logging needs to be enabled first by setting the storm config "topology.eventlogger.executors" to a non zero value. Please see
+the [Configuration](#config) section for more details.
+
+Events can be logged by clicking the "Debug" button under the topology actions in the topology view. This logs the
tuples from all the spouts and bolts in a topology at the specified sampling percentage.
<div align="center">
@@ -54,14 +57,14 @@ Event logging can be disabled at a specific component or at the topology level b
<p>Figure 4: Disable event logging at topology level.</p>
</div>
-## Configuration
-Eventlogging works by sending the events (tuples) from each component to an internal eventlogger bolt. By default storm automatically starts one event logger task per worker. This can be easily changed by setting the below parameter while running your topology (by setting it in storm.yaml or passing options via command line).
+## <a name="config"></a>Configuration
+Eventlogging works by sending the events (tuples) from each component to an internal eventlogger bolt. By default Storm does not start any event logger tasks, but this can be easily changed by setting the below parameter while running your topology (by setting it in storm.yaml or passing options via command line).
| Parameter | Meaning |
| -------------------------------------------|-----------------------|
-| "topology.eventlogger.executors": nil | One event logger task per worker (default). |
+| "topology.eventlogger.executors": 0 | No event logger tasks are created (default). |
| "topology.eventlogger.executors": 1 | One event logger task for the topology. |
-| "topology.eventlogger.executors": 0 | No event logger tasks are created. |
+| "topology.eventlogger.executors": nil | One event logger task per worker. |
## Extending eventlogging