You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by pt...@apache.org on 2016/04/21 20:46:53 UTC
[1/4] storm git commit: [STORM-1544] Document Debug/Sampling of
Topologies
Repository: storm
Updated Branches:
refs/heads/master fdf2cd005 -> 4e3c5bd54
[STORM-1544] Document Debug/Sampling of Topologies
Added documentation for logging/sampling events in a topology.
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/09695564
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/09695564
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/09695564
Branch: refs/heads/master
Commit: 09695564b97be6303c50df53071dda0111be2a36
Parents: 31db7dc
Author: Arun Mahadevan <ai...@hortonworks.com>
Authored: Mon Mar 28 14:19:31 2016 +0530
Committer: Arun Mahadevan <ai...@hortonworks.com>
Committed: Tue Mar 29 10:03:07 2016 +0530
----------------------------------------------------------------------
docs/Eventlogging.md | 93 ++++++++++++++++++++
docs/images/disable-event-logging-topology.png | Bin 0 -> 128737 bytes
docs/images/enable-event-logging-spout.png | Bin 0 -> 136954 bytes
docs/images/enable-event-logging-topology.png | Bin 0 -> 161055 bytes
docs/images/event-logs-view.png | Bin 0 -> 192749 bytes
docs/index.md | 1 +
6 files changed, 94 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/09695564/docs/Eventlogging.md
----------------------------------------------------------------------
diff --git a/docs/Eventlogging.md b/docs/Eventlogging.md
new file mode 100644
index 0000000..d864366
--- /dev/null
+++ b/docs/Eventlogging.md
@@ -0,0 +1,93 @@
+---
+title: Topology event inspector
+layout: documentation
+documentation: true
+---
+
+# Introduction
+
+Topology event inspector provides the ability to view the tuples as it flows through different stages in a storm topology.
+This could be useful for inspecting the tuples emitted at a spout or a bolt in the topology pipeline while the topology is running, without stopping or redeploying the topology. The normal flow of tuples from the spouts to the bolts is not affected by turning on event logging.
+
+## Enabling event logging
+
+Event logging can be enabled by clicking the "Debug" button under the topology actions in the topology view. This logs the
+tuples from all the spouts and bolts in a topology at the specified sampling percentage.
+
+<div align="center">
+<img title="Enable Eventlogging" src="images/enable-event-logging-topology.png" style="max-width: 80rem"/>
+
+<p>Figure 1: Enable event logging at topology level.</p>
+</div>
+
+You could also enable event logging at a specific spout or bolt level by going to the corresponding component page and
+clicking "Debug" under component actions.
+
+<div align="center">
+<img title="Enable Eventlogging at component level" src="images/enable-event-logging-spout.png" style="max-width: 80rem"/>
+
+<p>Figure 2: Enable event logging at component level.</p>
+</div>
+
+## Viewing the event logs
+The Storm "logviewer" should be running for viewing the logged tuples. If not already running log viewer can be started by running the "bin/storm logviewer" command from the storm installation directory. For viewing the tuples, go to the specific spout or bolt component page from storm UI and click on the "events" link under the component summary (as highlighted in Figure 2 above).
+
+This would open up a view like below where you can navigate between different pages and view the logged tuples.
+
+<div align="center">
+<img title="Viewing logged tuples" src="images/event-logs-view.png" style="max-width: 80rem"/>
+
+<p>Figure 3: Viewing the logged events.</p>
+</div>
+
+Each line in the event log contains an entry corresponding to a tuple emitted from a specific spout/bolt in a comma separated format.
+
+`Timestamp, Component name, Component task-id, MessageId (in case of anchoring), List of emitted values`
+
+## Disabling the event logs
+
+Event logging can be disabled at a specific component or at the topology level by clicking the "Stop Debug" under the topology or component actions in the Storm UI.
+
+<div align="center">
+<img title="Disable Eventlogging at topology level" src="images/disable-event-logging-topology.png" style="max-width: 80rem"/>
+
+<p>Figure 4: Disable event logging at topology level.</p>
+</div>
+
+## Configuration
+Eventlogging works by sending the events (tuples) from each component to an internal eventlogger bolt. By default storm automatically starts one event logger task per worker. This can be easily changed by setting the below parameter while running your topology (by setting it in storm.yaml or passing options via command line).
+
+| Parameter | Meaning |
+| -------------------------------------------|-----------------------|
+| "topology.eventlogger.executors": nil | One event logger task per worker (default). |
+| "topology.eventlogger.executors": 1 | One event logger task for the topology. |
+| "topology.eventlogger.executors": 0 | No event logger tasks are created. |
+
+
+## Extending eventlogging
+Storm provides an `IEventLogger` interface which is used by the event logger bolt to log the events. The default implementation for this is a FileBasedEventLogger which logs the events to an events.log file ( `logs/workers-artifacts/<topology-id>/<worker-port>/events.log`). Alternate implementations of the `IEventLogger` interface can be added to extend the event logging functionality (say build a search index or log the events in a database etc)
+
+```java
+/**
+ * EventLogger interface for logging the event info to a sink like log file or db
+ * for inspecting the events via UI for debugging.
+ */
+public interface IEventLogger {
+ /**
+ * Invoked during eventlogger bolt prepare.
+ */
+ void prepare(Map stormConf, TopologyContext context);
+
+ /**
+ * Invoked when the {@link EventLoggerBolt} receives a tuple from the spouts or bolts that has event logging enabled.
+ *
+ * @param e the event
+ */
+ void log(EventInfo e);
+
+ /**
+ * Invoked when the event logger bolt is cleaned up
+ */
+ void close();
+}
+```
http://git-wip-us.apache.org/repos/asf/storm/blob/09695564/docs/images/disable-event-logging-topology.png
----------------------------------------------------------------------
diff --git a/docs/images/disable-event-logging-topology.png b/docs/images/disable-event-logging-topology.png
new file mode 100644
index 0000000..77405ed
Binary files /dev/null and b/docs/images/disable-event-logging-topology.png differ
http://git-wip-us.apache.org/repos/asf/storm/blob/09695564/docs/images/enable-event-logging-spout.png
----------------------------------------------------------------------
diff --git a/docs/images/enable-event-logging-spout.png b/docs/images/enable-event-logging-spout.png
new file mode 100644
index 0000000..081bab1
Binary files /dev/null and b/docs/images/enable-event-logging-spout.png differ
http://git-wip-us.apache.org/repos/asf/storm/blob/09695564/docs/images/enable-event-logging-topology.png
----------------------------------------------------------------------
diff --git a/docs/images/enable-event-logging-topology.png b/docs/images/enable-event-logging-topology.png
new file mode 100644
index 0000000..dc8ee58
Binary files /dev/null and b/docs/images/enable-event-logging-topology.png differ
http://git-wip-us.apache.org/repos/asf/storm/blob/09695564/docs/images/event-logs-view.png
----------------------------------------------------------------------
diff --git a/docs/images/event-logs-view.png b/docs/images/event-logs-view.png
new file mode 100644
index 0000000..104f35c
Binary files /dev/null and b/docs/images/event-logs-view.png differ
http://git-wip-us.apache.org/repos/asf/storm/blob/09695564/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index bfd68db..a514de8 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -62,6 +62,7 @@ Trident is an alternative interface to Storm. It provides exactly-once processin
* [Dynamic Log Level Settings](dynamic-log-level-settings.html)
* [Searching Worker Logs](Logs.html)
* [Worker Profiling](dynamic-worker-profiling.html)
+* [Event Logging](Eventlogging.html)
### Integration With External Systems, and Other Libraries
* [Apache Kafka Integration](storm-kafka.html)
[2/4] storm git commit: Updating the doc to reflect the new default
behavior
Posted by pt...@apache.org.
Updating the doc to reflect the new default behavior
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/bcf8c8a6
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/bcf8c8a6
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/bcf8c8a6
Branch: refs/heads/master
Commit: bcf8c8a6e88b6cb57852e3e9af24168e25c0f486
Parents: 0969556
Author: Arun Mahadevan <ai...@hortonworks.com>
Authored: Fri Apr 1 09:56:10 2016 +0530
Committer: Arun Mahadevan <ai...@hortonworks.com>
Committed: Fri Apr 1 09:56:10 2016 +0530
----------------------------------------------------------------------
docs/Eventlogging.md | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/bcf8c8a6/docs/Eventlogging.md
----------------------------------------------------------------------
diff --git a/docs/Eventlogging.md b/docs/Eventlogging.md
index d864366..0895232 100644
--- a/docs/Eventlogging.md
+++ b/docs/Eventlogging.md
@@ -11,7 +11,10 @@ This could be useful for inspecting the tuples emitted at a spout or a bolt in t
## Enabling event logging
-Event logging can be enabled by clicking the "Debug" button under the topology actions in the topology view. This logs the
+Note: Event logging needs to be enabled first by setting the storm config "topology.eventlogger.executors" to a non zero value. Please see
+the [Configuration](#config) section for more details.
+
+Events can be logged by clicking the "Debug" button under the topology actions in the topology view. This logs the
tuples from all the spouts and bolts in a topology at the specified sampling percentage.
<div align="center">
@@ -54,14 +57,14 @@ Event logging can be disabled at a specific component or at the topology level b
<p>Figure 4: Disable event logging at topology level.</p>
</div>
-## Configuration
-Eventlogging works by sending the events (tuples) from each component to an internal eventlogger bolt. By default storm automatically starts one event logger task per worker. This can be easily changed by setting the below parameter while running your topology (by setting it in storm.yaml or passing options via command line).
+## <a name="config"></a>Configuration
+Eventlogging works by sending the events (tuples) from each component to an internal eventlogger bolt. By default Storm does not start any event logger tasks, but this can be easily changed by setting the below parameter while running your topology (by setting it in storm.yaml or passing options via command line).
| Parameter | Meaning |
| -------------------------------------------|-----------------------|
-| "topology.eventlogger.executors": nil | One event logger task per worker (default). |
+| "topology.eventlogger.executors": 0 | No event logger tasks are created (default). |
| "topology.eventlogger.executors": 1 | One event logger task for the topology. |
-| "topology.eventlogger.executors": 0 | No event logger tasks are created. |
+| "topology.eventlogger.executors": nil | One event logger task per worker. |
## Extending eventlogging
[4/4] storm git commit: add STORM-1544 to changelog
Posted by pt...@apache.org.
add STORM-1544 to changelog
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/4e3c5bd5
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/4e3c5bd5
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/4e3c5bd5
Branch: refs/heads/master
Commit: 4e3c5bd5427cfad76f04d0c6e68dc321d516547e
Parents: 7536706
Author: P. Taylor Goetz <pt...@gmail.com>
Authored: Thu Apr 21 14:44:03 2016 -0400
Committer: P. Taylor Goetz <pt...@gmail.com>
Committed: Thu Apr 21 14:44:03 2016 -0400
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/4e3c5bd5/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a7f4985..9420e54 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
## 2.0.0
+ * STORM-1544: Document Debug/Sampling of Topologies
* STORM-1681: Bug in scheduling cyclic topologies when scheduling with RAS
* STORM-1679: add storm Scheduler documents
* STORM-1687: divide by zero in StatsUtil
[3/4] storm git commit: Merge branch 'STORM-1544' of
github.com:arunmahadevan/storm
Posted by pt...@apache.org.
Merge branch 'STORM-1544' of github.com:arunmahadevan/storm
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/75367068
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/75367068
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/75367068
Branch: refs/heads/master
Commit: 753670689a90ca3c918c975599069eb6a0872978
Parents: fdf2cd0 bcf8c8a
Author: P. Taylor Goetz <pt...@gmail.com>
Authored: Thu Apr 21 14:42:40 2016 -0400
Committer: P. Taylor Goetz <pt...@gmail.com>
Committed: Thu Apr 21 14:42:40 2016 -0400
----------------------------------------------------------------------
docs/Eventlogging.md | 96 ++++++++++++++++++++
docs/images/disable-event-logging-topology.png | Bin 0 -> 128737 bytes
docs/images/enable-event-logging-spout.png | Bin 0 -> 136954 bytes
docs/images/enable-event-logging-topology.png | Bin 0 -> 161055 bytes
docs/images/event-logs-view.png | Bin 0 -> 192749 bytes
docs/index.md | 1 +
6 files changed, 97 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/75367068/docs/index.md
----------------------------------------------------------------------