You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by ka...@apache.org on 2016/06/01 06:21:00 UTC
svn commit: r1746399 [6/31] - in /storm/site: _posts/ publish/
publish/2014/04/17/ publish/about/ publish/releases/0.10.0/
publish/releases/0.10.0/about/ publish/releases/0.10.1/
publish/releases/0.10.1/about/ publish/releases/1.0.0/ publish/releases/1...
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/Common-patterns.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Common-patterns.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Common-patterns.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Common-patterns.html Wed Jun 1 06:20:57 2016
@@ -194,7 +194,7 @@
<span class="n">builder</span><span class="o">.</span><span class="na">setBolt</span><span class="o">(</span><span class="s">"merge"</span><span class="o">,</span> <span class="k">new</span> <span class="n">MergeObjects</span><span class="o">())</span>
<span class="o">.</span><span class="na">globalGrouping</span><span class="o">(</span><span class="s">"rank"</span><span class="o">);</span>
</code></pre></div>
-<p>This pattern works because of the fields grouping done by the first bolt which gives the partitioning you need for this to be semantically correct. You can see an example of this pattern in storm-starter <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/RollingTopWords.java">here</a>.</p>
+<p>This pattern works because of the fields grouping done by the first bolt which gives the partitioning you need for this to be semantically correct. You can see an example of this pattern in storm-starter <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/org/apache/storm/starter/RollingTopWords.java">here</a>.</p>
<p>If however you have a known skew in the data being processed it can be advantageous to use partialKeyGrouping instead of fieldsGrouping. This will distribute the load for each key between two downstream bolts instead of a single one.</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">builder</span><span class="o">.</span><span class="na">setBolt</span><span class="o">(</span><span class="s">"count"</span><span class="o">,</span> <span class="k">new</span> <span class="n">CountObjects</span><span class="o">(),</span> <span class="n">parallelism</span><span class="o">)</span>
@@ -204,7 +204,7 @@
<span class="n">builder</span><span class="o">.</span><span class="na">setBolt</span><span class="o">(</span><span class="s">"merge"</span><span class="o">,</span> <span class="k">new</span> <span class="n">MergeRanksObjects</span><span class="o">())</span>
<span class="o">.</span><span class="na">globalGrouping</span><span class="o">(</span><span class="s">"rank"</span><span class="o">);</span>
</code></pre></div>
-<p>The topology needs an extra layer of processing to aggregate the partial counts from the upstream bolts but this only processes aggregated values now so the bolt it is not subject to the load caused by the skewed data. You can see an example of this pattern in storm-starter <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/SkewedRollingTopWords.java">here</a>.</p>
+<p>The topology needs an extra layer of processing to aggregate the partial counts from the upstream bolts but this only processes aggregated values now so the bolt it is not subject to the load caused by the skewed data. You can see an example of this pattern in storm-starter <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/org/apache/storm/starter/SkewedRollingTopWords.java">here</a>.</p>
<h3 id="timecachemap-for-efficiently-keeping-a-cache-of-things-that-have-been-recently-updated">TimeCacheMap for efficiently keeping a cache of things that have been recently updated</h3>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/Distributed-RPC.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Distributed-RPC.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Distributed-RPC.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Distributed-RPC.html Wed Jun 1 06:20:57 2016
@@ -241,7 +241,7 @@
<p>A single reach computation can involve thousands of database calls and tens of millions of follower records during the computation. It's a really, really intense computation. As you're about to see, implementing this function on top of Storm is dead simple. On a single machine, reach can take minutes to compute; on a Storm cluster, you can compute reach for even the hardest URLs in a couple seconds.</p>
-<p>A sample reach topology is defined in storm-starter <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/ReachTopology.java">here</a>. Here's how you define the reach topology:</p>
+<p>A sample reach topology is defined in storm-starter <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/org/apache/storm/starter/ReachTopology.java">here</a>. Here's how you define the reach topology:</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">LinearDRPCTopologyBuilder</span> <span class="n">builder</span> <span class="o">=</span> <span class="k">new</span> <span class="n">LinearDRPCTopologyBuilder</span><span class="o">(</span><span class="s">"reach"</span><span class="o">);</span>
<span class="n">builder</span><span class="o">.</span><span class="na">addBolt</span><span class="o">(</span><span class="k">new</span> <span class="n">GetTweeters</span><span class="o">(),</span> <span class="mi">3</span><span class="o">);</span>
<span class="n">builder</span><span class="o">.</span><span class="na">addBolt</span><span class="o">(</span><span class="k">new</span> <span class="n">GetFollowers</span><span class="o">(),</span> <span class="mi">12</span><span class="o">)</span>
Added: storm/site/publish/releases/2.0.0-SNAPSHOT/Eventlogging.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Eventlogging.html?rev=1746399&view=auto
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Eventlogging.html (added)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Eventlogging.html Wed Jun 1 06:20:57 2016
@@ -0,0 +1,319 @@
+<!DOCTYPE html>
+<html>
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <title>Topology event inspector</title>
+
+ <!-- Bootstrap core CSS -->
+ <link href="/assets/css/bootstrap.min.css" rel="stylesheet">
+ <!-- Bootstrap theme -->
+ <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
+
+ <!-- Custom styles for this template -->
+ <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
+ <link href="/css/style.css" rel="stylesheet">
+ <link href="/assets/css/owl.theme.css" rel="stylesheet">
+ <link href="/assets/css/owl.carousel.css" rel="stylesheet">
+ <script type="text/javascript" src="/assets/js/jquery.min.js"></script>
+ <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
+ <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
+ <script type="text/javascript" src="/assets/js/storm.js"></script>
+ <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
+ <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+
+
+ <body>
+ <header>
+ <div class="container-fluid">
+ <div class="row">
+ <div class="col-md-5">
+ <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
+ </div>
+ <div class="col-md-5">
+
+ <h1>Version: 2.0.0-SNAPSHOT</h1>
+
+ </div>
+ <div class="col-md-2">
+ <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
+ </div>
+ </div>
+ </div>
+</header>
+<!--Header End-->
+<!--Navigation Begin-->
+<div class="navbar" role="banner">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
+ <ul class="nav navbar-nav">
+ <li><a href="/index.html" id="home">Home</a></li>
+ <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
+ <li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+ <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li>
+
+
+
+
+
+ <li><a href="/releases/1.0.1/index.html">1.0.1</a></li>
+
+
+
+ <li><a href="/releases/1.0.0/index.html">1.0.0</a></li>
+
+
+
+ <li><a href="/releases/0.10.1/index.html">0.10.1</a></li>
+
+
+
+
+
+ <li><a href="/releases/0.10.0/index.html">0.10.0</a></li>
+
+
+
+ <li><a href="/releases/0.9.6/index.html">0.9.6</a></li>
+
+
+
+
+
+
+
+
+
+
+
+
+ </ul>
+ </li>
+ <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
+ <li><a href="/contribute/People.html">People</a></li>
+ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
+ </ul>
+ </li>
+ <li><a href="/2016/05/06/storm101-released.html" id="news">News</a></li>
+ </ul>
+ </nav>
+ </div>
+</div>
+
+
+
+ <div class="container-fluid">
+ <h1 class="page-title">Topology event inspector</h1>
+ <div class="row">
+ <div class="col-md-12">
+ <!-- Documentation -->
+
+<p class="post-meta"></p>
+
+<h1 id="introduction">Introduction</h1>
+
+<p>Topology event inspector provides the ability to view the tuples as it flows through different stages in a storm topology.
+This could be useful for inspecting the tuples emitted at a spout or a bolt in the topology pipeline while the topology is running, without stopping or redeploying the topology. The normal flow of tuples from the spouts to the bolts is not affected by turning on event logging.</p>
+
+<h2 id="enabling-event-logging">Enabling event logging</h2>
+
+<p>Note: Event logging needs to be enabled first by setting the storm config "topology.eventlogger.executors" to a non zero value. Please see
+the <a href="#config">Configuration</a> section for more details.</p>
+
+<p>Events can be logged by clicking the "Debug" button under the topology actions in the topology view. This logs the
+tuples from all the spouts and bolts in a topology at the specified sampling percentage.</p>
+
+<div align="center">
+<img title="Enable Eventlogging" src="images/enable-event-logging-topology.png" style="max-width: 80rem"/>
+
+<p>Figure 1: Enable event logging at topology level.</p>
+</div>
+
+<p>You could also enable event logging at a specific spout or bolt level by going to the corresponding component page and
+clicking "Debug" under component actions.</p>
+
+<div align="center">
+<img title="Enable Eventlogging at component level" src="images/enable-event-logging-spout.png" style="max-width: 80rem"/>
+
+<p>Figure 2: Enable event logging at component level.</p>
+</div>
+
+<h2 id="viewing-the-event-logs">Viewing the event logs</h2>
+
+<p>The Storm "logviewer" should be running for viewing the logged tuples. If not already running log viewer can be started by running the "bin/storm logviewer" command from the storm installation directory. For viewing the tuples, go to the specific spout or bolt component page from storm UI and click on the "events" link under the component summary (as highlighted in Figure 2 above).</p>
+
+<p>This would open up a view like below where you can navigate between different pages and view the logged tuples.</p>
+
+<div align="center">
+<img title="Viewing logged tuples" src="images/event-logs-view.png" style="max-width: 80rem"/>
+
+<p>Figure 3: Viewing the logged events.</p>
+</div>
+
+<p>Each line in the event log contains an entry corresponding to a tuple emitted from a specific spout/bolt in a comma separated format.</p>
+
+<p><code>Timestamp, Component name, Component task-id, MessageId (in case of anchoring), List of emitted values</code></p>
+
+<h2 id="disabling-the-event-logs">Disabling the event logs</h2>
+
+<p>Event logging can be disabled at a specific component or at the topology level by clicking the "Stop Debug" under the topology or component actions in the Storm UI.</p>
+
+<div align="center">
+<img title="Disable Eventlogging at topology level" src="images/disable-event-logging-topology.png" style="max-width: 80rem"/>
+
+<p>Figure 4: Disable event logging at topology level.</p>
+</div>
+
+<h2 id="configuration"><a name="config"></a>Configuration</h2>
+
+<p>Eventlogging works by sending the events (tuples) from each component to an internal eventlogger bolt. By default Storm does not start any event logger tasks, but this can be easily changed by setting the below parameter while running your topology (by setting it in storm.yaml or passing options via command line).</p>
+
+<table><thead>
+<tr>
+<th>Parameter</th>
+<th>Meaning</th>
+</tr>
+</thead><tbody>
+<tr>
+<td>"topology.eventlogger.executors": 0</td>
+<td>No event logger tasks are created (default).</td>
+</tr>
+<tr>
+<td>"topology.eventlogger.executors": 1</td>
+<td>One event logger task for the topology.</td>
+</tr>
+<tr>
+<td>"topology.eventlogger.executors": nil</td>
+<td>One event logger task per worker.</td>
+</tr>
+</tbody></table>
+
+<h2 id="extending-eventlogging">Extending eventlogging</h2>
+
+<p>Storm provides an <code>IEventLogger</code> interface which is used by the event logger bolt to log the events. The default implementation for this is a FileBasedEventLogger which logs the events to an events.log file ( <code>logs/workers-artifacts/<topology-id>/<worker-port>/events.log</code>). Alternate implementations of the <code>IEventLogger</code> interface can be added to extend the event logging functionality (say build a search index or log the events in a database etc)</p>
+<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="cm">/**
+ * EventLogger interface for logging the event info to a sink like log file or db
+ * for inspecting the events via UI for debugging.
+ */</span>
+<span class="kd">public</span> <span class="kd">interface</span> <span class="nc">IEventLogger</span> <span class="o">{</span>
+ <span class="cm">/**
+ * Invoked during eventlogger bolt prepare.
+ */</span>
+ <span class="kt">void</span> <span class="n">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">stormConf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">);</span>
+
+ <span class="cm">/**
+ * Invoked when the {@link EventLoggerBolt} receives a tuple from the spouts or bolts that has event logging enabled.
+ *
+ * @param e the event
+ */</span>
+ <span class="kt">void</span> <span class="n">log</span><span class="o">(</span><span class="n">EventInfo</span> <span class="n">e</span><span class="o">);</span>
+
+ <span class="cm">/**
+ * Invoked when the event logger bolt is cleaned up
+ */</span>
+ <span class="kt">void</span> <span class="n">close</span><span class="o">();</span>
+<span class="o">}</span>
+</code></pre></div>
+
+
+ </div>
+ </div>
+ </div>
+<footer>
+ <div class="container-fluid">
+ <div class="row">
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>Meetups</h5>
+ <ul class="latest-news">
+
+ <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
+
+ <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
+
+ <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
+
+ <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
+
+ <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
+
+ <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
+
+ <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
+ </ul>
+ </div>
+ </div>
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>About Storm</h5>
+ <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p>
+ </div>
+ </div>
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>First Look</h5>
+ <ul class="footer-list">
+ <li><a href="/releases/current/Rationale.html">Rationale</a></li>
+ <li><a href="/releases/current/Tutorial.html">Tutorial</a></li>
+ <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
+ <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
+ </ul>
+ </div>
+ </div>
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>Documentation</h5>
+ <ul class="footer-list">
+ <li><a href="/releases/current/index.html">Index</a></li>
+ <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
+ <li><a href="/releases/current/FAQ.html">FAQ</a></li>
+ </ul>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <div class="row">
+ <div class="col-md-12">
+ <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
+ <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
+ <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+ </div>
+ </div>
+</footer>
+<!--Footer End-->
+<!-- Scroll to top -->
+<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
+
+</body>
+
+</html>
+
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/Powered-By.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Powered-By.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Powered-By.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Powered-By.html Wed Jun 1 06:20:57 2016
@@ -230,6 +230,16 @@ Spotify serves streaming music to over 1
</td>
</tr>
+<tr>
+<td>
+<a href="http://www.paypal.com">PayPal</a>
+</td>
+<td>
+<p>
+PayPal is using Storm along with Esper and Kafka to monitor 100K+ merchants, their activities and any anomalies in real time. We employ machine learning to detect any issues as they occur on our 25-node storm cluster running along with other components on a lambda architecture.
+</p>
+</td>
+</tr>
<tr>
<td>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/SECURITY.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/SECURITY.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/SECURITY.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/SECURITY.html Wed Jun 1 06:20:57 2016
@@ -537,7 +537,7 @@ This config file also needs to be owned
it can do so by leveraging the impersonation feature.In order to submit topology as some other user , you can use <code>StormSubmitter.submitTopologyAs</code> API. Alternatively you can use <code>NimbusClient.getConfiguredClientAs</code>
to get a nimbus client as some other user and perform any nimbus action(i.e. kill/rebalance/activate/deactivate) using this client. </p>
-<p>To ensure only authorized users can perform impersonation you should start nimbus with <code>nimbus.impersonation.authorizer</code> set to <code>org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer</code>.
+<p>Impersonation authorization is disabled by default which means any user can perform impersonation. To ensure only authorized users can perform impersonation you should start nimbus with <code>nimbus.impersonation.authorizer</code> set to <code>org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer</code>.
The <code>ImpersonationAuthorizer</code> uses <code>nimbus.impersonation.acl</code> as the acl to authorize users. Following is a sample nimbus config for supporting impersonation:</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">nimbus.impersonation.authorizer</span><span class="pi">:</span> <span class="s">org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer</span>
<span class="s">nimbus.impersonation.acl</span><span class="pi">:</span>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/Setting-up-a-Storm-cluster.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Setting-up-a-Storm-cluster.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Setting-up-a-Storm-cluster.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Setting-up-a-Storm-cluster.html Wed Jun 1 06:20:57 2016
@@ -168,7 +168,7 @@
<p>Next you need to install Storm's dependencies on Nimbus and the worker machines. These are:</p>
<ol>
-<li>Java 6</li>
+<li>Java 7</li>
<li>Python 2.6.6</li>
</ol>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/State-checkpointing.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/State-checkpointing.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/State-checkpointing.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/State-checkpointing.html Wed Jun 1 06:20:57 2016
@@ -228,6 +228,9 @@ is saved and then the checkpoint tuple i
streams before it saves its state so that the state represents a consistent state across the topology. Once the checkpoint spout receives
ACK from all the bolts, the state commit is complete and the transaction is recorded as committed by the checkpoint spout.</p>
+<p>The state checkpointing does not currently checkpoint the state of the spout. Yet, once the state of all bolts are checkpointed, and once the checkpoint tuples are acked, the tuples emitted by the spout are also acked.
+It also implies that <code>topology.state.checkpoint.interval.ms</code> is lower than <code>topology.message.timeout.secs</code>. </p>
+
<p>The state commit works like a three phase commit protocol with a prepare and commit phase so that the state across the topology is saved
in a consistent and atomic manner.</p>
Added: storm/site/publish/releases/2.0.0-SNAPSHOT/Storm-Scheduler.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Storm-Scheduler.html?rev=1746399&view=auto
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Storm-Scheduler.html (added)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Storm-Scheduler.html Wed Jun 1 06:20:57 2016
@@ -0,0 +1,235 @@
+<!DOCTYPE html>
+<html>
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <title>Scheduler</title>
+
+ <!-- Bootstrap core CSS -->
+ <link href="/assets/css/bootstrap.min.css" rel="stylesheet">
+ <!-- Bootstrap theme -->
+ <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
+
+ <!-- Custom styles for this template -->
+ <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
+ <link href="/css/style.css" rel="stylesheet">
+ <link href="/assets/css/owl.theme.css" rel="stylesheet">
+ <link href="/assets/css/owl.carousel.css" rel="stylesheet">
+ <script type="text/javascript" src="/assets/js/jquery.min.js"></script>
+ <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
+ <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
+ <script type="text/javascript" src="/assets/js/storm.js"></script>
+ <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
+ <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+
+
+ <body>
+ <header>
+ <div class="container-fluid">
+ <div class="row">
+ <div class="col-md-5">
+ <a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
+ </div>
+ <div class="col-md-5">
+
+ <h1>Version: 2.0.0-SNAPSHOT</h1>
+
+ </div>
+ <div class="col-md-2">
+ <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
+ </div>
+ </div>
+ </div>
+</header>
+<!--Header End-->
+<!--Navigation Begin-->
+<div class="navbar" role="banner">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
+ <ul class="nav navbar-nav">
+ <li><a href="/index.html" id="home">Home</a></li>
+ <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
+ <li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+
+ <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li>
+
+
+
+
+
+ <li><a href="/releases/1.0.1/index.html">1.0.1</a></li>
+
+
+
+ <li><a href="/releases/1.0.0/index.html">1.0.0</a></li>
+
+
+
+ <li><a href="/releases/0.10.1/index.html">0.10.1</a></li>
+
+
+
+
+
+ <li><a href="/releases/0.10.0/index.html">0.10.0</a></li>
+
+
+
+ <li><a href="/releases/0.9.6/index.html">0.9.6</a></li>
+
+
+
+
+
+
+
+
+
+
+
+
+ </ul>
+ </li>
+ <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
+ <li><a href="/contribute/People.html">People</a></li>
+ <li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
+ </ul>
+ </li>
+ <li><a href="/2016/05/06/storm101-released.html" id="news">News</a></li>
+ </ul>
+ </nav>
+ </div>
+</div>
+
+
+
+ <div class="container-fluid">
+ <h1 class="page-title">Scheduler</h1>
+ <div class="row">
+ <div class="col-md-12">
+ <!-- Documentation -->
+
+<p class="post-meta"></p>
+
+<p>Storm now has 4 kinds of built-in schedulers: <a href="http://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/scheduler/DefaultScheduler.java">DefaultScheduler</a>, <a href="http://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/scheduler/IsolationScheduler.java">IsolationScheduler</a>, <a href="http://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/scheduler/multitenant/MultitenantScheduler.java">MultitenantScheduler</a>, <a href="Resource_Aware_Scheduler_overview.html">ResourceAwareScheduler</a>. </p>
+
+<h2 id="pluggable-scheduler">Pluggable scheduler</h2>
+
+<p>You can implement your own scheduler to replace the default scheduler to assign executors to workers. You configure the class to use the "storm.scheduler" config in your storm.yaml, and your scheduler must implement the <a href="http://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/scheduler/IScheduler.java">IScheduler</a> interface.</p>
+
+<h2 id="isolation-scheduler">Isolation Scheduler</h2>
+
+<p>The isolation scheduler makes it easy and safe to share a cluster among many topologies. The isolation scheduler lets you specify which topologies should be "isolated", meaning that they run on a dedicated set of machines within the cluster where no other topologies will be running. These isolated topologies are given priority on the cluster, so resources will be allocated to isolated topologies if there's competition with non-isolated topologies, and resources will be taken away from non-isolated topologies if necessary to get resources for an isolated topology. Once all isolated topologies are allocated, the remaining machines on the cluster are shared among all non-isolated topologies.</p>
+
+<p>You can configure the isolation scheduler in the Nimbus configuration by setting "storm.scheduler" to "org.apache.storm.scheduler.IsolationScheduler". Then, use the "isolation.scheduler.machines" config to specify how many machines each topology should get. This configuration is a map from topology name to the number of isolated machines allocated to this topology. For example:</p>
+<div class="highlight"><pre><code class="language-" data-lang="">isolation.scheduler.machines:
+ "my-topology": 8
+ "tiny-topology": 1
+ "some-other-topology": 3
+</code></pre></div>
+<p>Any topologies submitted to the cluster not listed there will not be isolated. Note that there is no way for a user of Storm to affect their isolation settings – this is only allowed by the administrator of the cluster (this is very much intentional).</p>
+
+<p>The isolation scheduler solves the multi-tenancy problem – avoiding resource contention between topologies – by providing full isolation between topologies. The intention is that "productionized" topologies should be listed in the isolation config, and test or in-development topologies should not. The remaining machines on the cluster serve the dual role of failover for isolated topologies and for running the non-isolated topologies.</p>
+
+
+
+ </div>
+ </div>
+ </div>
+<footer>
+ <div class="container-fluid">
+ <div class="row">
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>Meetups</h5>
+ <ul class="latest-news">
+
+ <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
+
+ <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
+
+ <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
+
+ <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
+
+ <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
+
+ <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
+
+ <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
+ </ul>
+ </div>
+ </div>
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>About Storm</h5>
+ <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p>
+ </div>
+ </div>
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>First Look</h5>
+ <ul class="footer-list">
+ <li><a href="/releases/current/Rationale.html">Rationale</a></li>
+ <li><a href="/releases/current/Tutorial.html">Tutorial</a></li>
+ <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
+ <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li>
+ </ul>
+ </div>
+ </div>
+ <div class="col-md-3">
+ <div class="footer-widget">
+ <h5>Documentation</h5>
+ <ul class="footer-list">
+ <li><a href="/releases/current/index.html">Index</a></li>
+ <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
+ <li><a href="/releases/current/FAQ.html">FAQ</a></li>
+ </ul>
+ </div>
+ </div>
+ </div>
+ <hr/>
+ <div class="row">
+ <div class="col-md-12">
+ <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
+ <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
+ <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+ </div>
+ </div>
+</footer>
+<!--Footer End-->
+<!-- Scroll to top -->
+<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
+
+</body>
+
+</html>
+
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/Transactional-topologies.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Transactional-topologies.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Transactional-topologies.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Transactional-topologies.html Wed Jun 1 06:20:57 2016
@@ -224,7 +224,7 @@
<h2 id="the-basics-through-example">The basics through example</h2>
-<p>You build transactional topologies by using <a href="javadocs/org/apache/storm/transactional/TransactionalTopologyBuilder.html">TransactionalTopologyBuilder</a>. Here's the transactional topology definition for a topology that computes the global count of tuples from the input stream. This code comes from <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/TransactionalGlobalCount.java">TransactionalGlobalCount</a> in storm-starter.</p>
+<p>You build transactional topologies by using <a href="javadocs/org/apache/storm/transactional/TransactionalTopologyBuilder.html">TransactionalTopologyBuilder</a>. Here's the transactional topology definition for a topology that computes the global count of tuples from the input stream. This code comes from <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/org/apache/storm/starter/TransactionalGlobalCount.java">TransactionalGlobalCount</a> in storm-starter.</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">MemoryTransactionalSpout</span> <span class="n">spout</span> <span class="o">=</span> <span class="k">new</span> <span class="n">MemoryTransactionalSpout</span><span class="o">(</span><span class="n">DATA</span><span class="o">,</span> <span class="k">new</span> <span class="n">Fields</span><span class="o">(</span><span class="s">"word"</span><span class="o">),</span> <span class="n">PARTITION_TAKE_PER_BATCH</span><span class="o">);</span>
<span class="n">TransactionalTopologyBuilder</span> <span class="n">builder</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TransactionalTopologyBuilder</span><span class="o">(</span><span class="s">"global-count"</span><span class="o">,</span> <span class="s">"spout"</span><span class="o">,</span> <span class="n">spout</span><span class="o">,</span> <span class="mi">3</span><span class="o">);</span>
<span class="n">builder</span><span class="o">.</span><span class="na">setBolt</span><span class="o">(</span><span class="s">"partial-count"</span><span class="o">,</span> <span class="k">new</span> <span class="n">BatchCount</span><span class="o">(),</span> <span class="mi">5</span><span class="o">)</span>
@@ -332,7 +332,7 @@
<p>The code for <code>finishBatch</code> in <code>UpdateGlobalCount</code> gets the current value from the database and compares its transaction id to the transaction id for this batch. If they are the same, it does nothing. Otherwise, it increments the value in the database by the partial count for this batch.</p>
-<p>A more involved transactional topology example that updates multiple databases idempotently can be found in storm-starter in the <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/TransactionalWords.java">TransactionalWords</a> class.</p>
+<p>A more involved transactional topology example that updates multiple databases idempotently can be found in storm-starter in the <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/org/apache/storm/starter/TransactionalWords.java">TransactionalWords</a> class.</p>
<h2 id="transactional-topology-api">Transactional Topology API</h2>
@@ -388,7 +388,7 @@
<h4 id="partitioned-transactional-spout">Partitioned Transactional Spout</h4>
-<p>A common kind of transactional spout is one that reads the batches from a set of partitions across many queue brokers. For example, this is how <a href="http://github.com/apache/storm/tree/master/external/storm-kafka/src/jvm/storm/kafka/TransactionalKafkaSpout.java">TransactionalKafkaSpout</a> works. An <code>IPartitionedTransactionalSpout</code> automates the bookkeeping work of managing the state for each partition to ensure idempotent replayability. See <a href="javadocs/org/apache/storm/transactional/partitioned/IPartitionedTransactionalSpout.html">the Javadoc</a> for more details.</p>
+<p>A common kind of transactional spout is one that reads the batches from a set of partitions across many queue brokers. For example, this is how <a href="http://github.com/apache/storm/tree/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/TransactionalKafkaSpout.java">TransactionalKafkaSpout</a> works. An <code>IPartitionedTransactionalSpout</code> automates the bookkeeping work of managing the state for each partition to ensure idempotent replayability. See <a href="javadocs/org/apache/storm/transactional/partitioned/IPartitionedTransactionalSpout.html">the Javadoc</a> for more details.</p>
<h3 id="configuration">Configuration</h3>
@@ -460,7 +460,7 @@
<p>By failing all subsequent transactions on failure, no tuples are skipped. This also shows that a requirement of transactional spouts is that they always emit where the last transaction left off.</p>
-<p>A non-idempotent transactional spout is more concisely referred to as an "OpaqueTransactionalSpout" (opaque is the opposite of idempotent). <a href="javadocs/org/apache/storm/transactional/partitioned/IOpaquePartitionedTransactionalSpout.html">IOpaquePartitionedTransactionalSpout</a> is an interface for implementing opaque partitioned transactional spouts, of which <a href="http://github.com/apache/storm/tree/master/external/storm-kafka/src/jvm/storm/kafka/OpaqueTransactionalKafkaSpout.java">OpaqueTransactionalKafkaSpout</a> is an example. <code>OpaqueTransactionalKafkaSpout</code> can withstand losing individual Kafka nodes without sacrificing accuracy as long as you use the update strategy as explained in this section.</p>
+<p>A non-idempotent transactional spout is more concisely referred to as an "OpaqueTransactionalSpout" (opaque is the opposite of idempotent). <a href="javadocs/org/apache/storm/transactional/partitioned/IOpaquePartitionedTransactionalSpout.html">IOpaquePartitionedTransactionalSpout</a> is an interface for implementing opaque partitioned transactional spouts, of which <a href="http://github.com/apache/storm/tree/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/OpaqueTransactionalKafkaSpout.java">OpaqueTransactionalKafkaSpout</a> is an example. <code>OpaqueTransactionalKafkaSpout</code> can withstand losing individual Kafka nodes without sacrificing accuracy as long as you use the update strategy as explained in this section.</p>
<h2 id="implementation">Implementation</h2>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/Trident-state.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Trident-state.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Trident-state.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Trident-state.html Wed Jun 1 06:20:57 2016
@@ -166,7 +166,7 @@
<li>Every tuple is in a batch (no tuples are skipped)</li>
</ol>
-<p>This is a pretty easy type of spout to understand, the stream is divided into fixed batches that never change. storm-contrib has <a href="http://github.com/apache/storm/tree/master/external/storm-kafka/src/jvm/storm/kafka/trident/TransactionalTridentKafkaSpout.java">an implementation of a transactional spout</a> for Kafka.</p>
+<p>This is a pretty easy type of spout to understand, the stream is divided into fixed batches that never change. storm-contrib has <a href="http://github.com/apache/storm/tree/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/trident/TransactionalTridentKafkaSpout.java">an implementation of a transactional spout</a> for Kafka.</p>
<p>You might be wondering – why wouldn't you just always use a transactional spout? They're simple and easy to understand. One reason you might not use one is because they're not necessarily very fault-tolerant. For example, the way TransactionalTridentKafkaSpout works is the batch for a txid will contain tuples from all the Kafka partitions for a topic. Once a batch has been emitted, any time that batch is re-emitted in the future the exact same set of tuples must be emitted to meet the semantics of transactional spouts. Now suppose a batch is emitted from TransactionalTridentKafkaSpout, the batch fails to process, and at the same time one of the Kafka nodes goes down. You're now incapable of replaying the same batch as you did before (since the node is down and some partitions for the topic are not unavailable), and processing will halt. </p>
@@ -203,7 +203,7 @@ apple => [count=10, txid=2]
<li>Every tuple is <em>successfully</em> processed in exactly one batch. However, it's possible for a tuple to fail to process in one batch and then succeed to process in a later batch.</li>
</ol>
-<p><a href="http://github.com/apache/storm/tree/master/external/storm-kafka/src/jvm/storm/kafka/trident/OpaqueTridentKafkaSpout.java">OpaqueTridentKafkaSpout</a> is a spout that has this property and is fault-tolerant to losing Kafka nodes. Whenever it's time for OpaqueTridentKafkaSpout to emit a batch, it emits tuples starting from where the last batch finished emitting. This ensures that no tuple is ever skipped or successfully processed by multiple batches.</p>
+<p><a href="http://github.com/apache/storm/tree/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/trident/OpaqueTridentKafkaSpout.java">OpaqueTridentKafkaSpout</a> is a spout that has this property and is fault-tolerant to losing Kafka nodes. Whenever it's time for OpaqueTridentKafkaSpout to emit a batch, it emits tuples starting from where the last batch finished emitting. This ensures that no tuple is ever skipped or successfully processed by multiple batches.</p>
<p>With opaque transactional spouts, it's no longer possible to use the trick of skipping state updates if the transaction id in the database is the same as the transaction id for the current batch. This is because the batch may have changed between state updates.</p>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/Tutorial.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Tutorial.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Tutorial.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Tutorial.html Wed Jun 1 06:20:57 2016
@@ -358,7 +358,7 @@
<p>When a task for Bolt A emits a tuple to Bolt B, which task should it send the tuple to?</p>
-<p>A "stream grouping" answers this question by telling Storm how to send tuples between sets of tasks. Before we dig into the different kinds of stream groupings, let's take a look at another topology from <a href="http://github.com/apache/storm/blob/2.0.0-SNAPSHOT/examples/storm-starter">storm-starter</a>. This <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/storm/starter/WordCountTopology.java">WordCountTopology</a> reads sentences off of a spout and streams out of <code>WordCountBolt</code> the total number of times it has seen that word before:</p>
+<p>A "stream grouping" answers this question by telling Storm how to send tuples between sets of tasks. Before we dig into the different kinds of stream groupings, let's take a look at another topology from <a href="http://github.com/apache/storm/blob/2.0.0-SNAPSHOT/examples/storm-starter">storm-starter</a>. This <a href="http://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/org/apache/storm/starter/WordCountTopology.java">WordCountTopology</a> reads sentences off of a spout and streams out of <code>WordCountBolt</code> the total number of times it has seen that word before:</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">TopologyBuilder</span> <span class="n">builder</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TopologyBuilder</span><span class="o">();</span>
<span class="n">builder</span><span class="o">.</span><span class="na">setSpout</span><span class="o">(</span><span class="s">"sentences"</span><span class="o">,</span> <span class="k">new</span> <span class="n">RandomSentenceSpout</span><span class="o">(),</span> <span class="mi">5</span><span class="o">);</span>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/Windowing.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Windowing.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Windowing.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Windowing.html Wed Jun 1 06:20:57 2016
@@ -151,14 +151,16 @@ following two parameters,</p>
<p>Tuples are grouped in windows and window slides every sliding interval. A tuple can belong to more than one window.</p>
<p>For example a time duration based sliding window with length 10 secs and sliding interval of 5 seconds.</p>
-<div class="highlight"><pre><code class="language-" data-lang="">| e1 e2 | e3 e4 e5 e6 | e7 e8 e9 |...
-0 5 10 15 -> time
-
-|<------- w1 -------->|
- |------------ w2 ------->|
+<div class="highlight"><pre><code class="language-" data-lang="">........| e1 e2 | e3 e4 e5 e6 | e7 e8 e9 |...
+-5 0 5 10 15 -> time
+|<------- w1 -->|
+ |<---------- w2 ----->|
+ |<-------------- w3 ---->|
</code></pre></div>
<p>The window is evaluated every 5 seconds and some of the tuples in the first window overlaps with the second one.</p>
+<p>Note: The window first slides at t = 5 secs and would contain events received up to the first five secs.</p>
+
<h2 id="tumbling-window">Tumbling Window</h2>
<p>Tuples are grouped in a single window based on time or count. Any tuple belongs to only one of the windows.</p>
Added: storm/site/publish/releases/2.0.0-SNAPSHOT/images/disable-event-logging-topology.png
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/images/disable-event-logging-topology.png?rev=1746399&view=auto
==============================================================================
Binary file - no diff available.
Propchange: storm/site/publish/releases/2.0.0-SNAPSHOT/images/disable-event-logging-topology.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: storm/site/publish/releases/2.0.0-SNAPSHOT/images/enable-event-logging-spout.png
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/images/enable-event-logging-spout.png?rev=1746399&view=auto
==============================================================================
Binary file - no diff available.
Propchange: storm/site/publish/releases/2.0.0-SNAPSHOT/images/enable-event-logging-spout.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: storm/site/publish/releases/2.0.0-SNAPSHOT/images/enable-event-logging-topology.png
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/images/enable-event-logging-topology.png?rev=1746399&view=auto
==============================================================================
Binary file - no diff available.
Propchange: storm/site/publish/releases/2.0.0-SNAPSHOT/images/enable-event-logging-topology.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: storm/site/publish/releases/2.0.0-SNAPSHOT/images/event-logs-view.png
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/images/event-logs-view.png?rev=1746399&view=auto
==============================================================================
Binary file - no diff available.
Propchange: storm/site/publish/releases/2.0.0-SNAPSHOT/images/event-logs-view.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/index.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/index.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/index.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/index.html Wed Jun 1 06:20:57 2016
@@ -143,6 +143,7 @@
<ul>
<li><a href="javadocs/index.html">Javadoc</a></li>
<li><a href="Concepts.html">Concepts</a></li>
+<li><a href="Storm-Scheduler.html">Scheduler</a></li>
<li><a href="Configuration.html">Configuration</a></li>
<li><a href="Guaranteeing-message-processing.html">Guaranteeing message processing</a></li>
<li><a href="Daemon-Fault-Tolerance.html">Daemon Fault Tolerance</a></li>
@@ -209,6 +210,7 @@
<li><a href="dynamic-log-level-settings.html">Dynamic Log Level Settings</a></li>
<li><a href="Logs.html">Searching Worker Logs</a></li>
<li><a href="dynamic-worker-profiling.html">Worker Profiling</a></li>
+<li><a href="Eventlogging.html">Event Logging</a></li>
</ul>
<h3 id="integration-with-external-systems-and-other-libraries">Integration With External Systems, and Other Libraries</h3>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/javadocs/allclasses-frame.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/javadocs/allclasses-frame.html?rev=1746399&r1=1746398&r2=1746399&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/javadocs/allclasses-frame.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/javadocs/allclasses-frame.html Wed Jun 1 06:20:57 2016
@@ -90,6 +90,8 @@
<li><a href="org/apache/storm/topology/base/BaseRichBolt.html" title="class in org.apache.storm.topology.base" target="classFrame">BaseRichBolt</a></li>
<li><a href="org/apache/storm/topology/base/BaseRichSpout.html" title="class in org.apache.storm.topology.base" target="classFrame">BaseRichSpout</a></li>
<li><a href="org/apache/storm/topology/base/BaseStatefulBolt.html" title="class in org.apache.storm.topology.base" target="classFrame">BaseStatefulBolt</a></li>
+<li><a href="org/apache/storm/topology/BaseStatefulBoltExecutor.html" title="class in org.apache.storm.topology" target="classFrame">BaseStatefulBoltExecutor</a></li>
+<li><a href="org/apache/storm/topology/BaseStatefulBoltExecutor.AnchoringOutputCollector.html" title="class in org.apache.storm.topology" target="classFrame">BaseStatefulBoltExecutor.AnchoringOutputCollector</a></li>
<li><a href="org/apache/storm/topology/base/BaseStatefulWindowedBolt.html" title="class in org.apache.storm.topology.base" target="classFrame">BaseStatefulWindowedBolt</a></li>
<li><a href="org/apache/storm/trident/state/BaseStateUpdater.html" title="class in org.apache.storm.trident.state" target="classFrame">BaseStateUpdater</a></li>
<li><a href="org/apache/storm/hooks/BaseTaskHook.html" title="class in org.apache.storm.hooks" target="classFrame">BaseTaskHook</a></li>
@@ -109,6 +111,7 @@
<li><a href="org/apache/storm/coordination/BatchBoltExecutor.html" title="class in org.apache.storm.coordination" target="classFrame">BatchBoltExecutor</a></li>
<li><a href="org/apache/storm/cassandra/bolt/BatchCassandraWriterBolt.html" title="class in org.apache.storm.cassandra.bolt" target="classFrame">BatchCassandraWriterBolt</a></li>
<li><a href="org/apache/storm/cassandra/query/impl/BatchCQLStatementTupleMapper.html" title="class in org.apache.storm.cassandra.query.impl" target="classFrame">BatchCQLStatementTupleMapper</a></li>
+<li><a href="org/apache/storm/utils/BatchHelper.html" title="class in org.apache.storm.utils" target="classFrame">BatchHelper</a></li>
<li><a href="org/apache/storm/trident/topology/BatchInfo.html" title="class in org.apache.storm.trident.topology" target="classFrame">BatchInfo</a></li>
<li><a href="org/apache/storm/testing/BatchNumberList.html" title="class in org.apache.storm.testing" target="classFrame">BatchNumberList</a></li>
<li><a href="org/apache/storm/coordination/BatchOutputCollector.html" title="class in org.apache.storm.coordination" target="classFrame">BatchOutputCollector</a></li>
@@ -158,6 +161,16 @@
<li><a href="org/apache/storm/kafka/BrokerHosts.html" title="interface in org.apache.storm.kafka" target="classFrame"><span class="interfaceName">BrokerHosts</span></a></li>
<li><a href="org/apache/storm/utils/BufferFileInputStream.html" title="class in org.apache.storm.utils" target="classFrame">BufferFileInputStream</a></li>
<li><a href="org/apache/storm/utils/BufferInputStream.html" title="class in org.apache.storm.utils" target="classFrame">BufferInputStream</a></li>
+<li><a href="org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.html" title="class in org.apache.storm.sql.compiler.backends.standalone" target="classFrame">BuiltinAggregateFunctions</a></li>
+<li><a href="org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.Count.html" title="class in org.apache.storm.sql.compiler.backends.standalone" target="classFrame">BuiltinAggregateFunctions.Count</a></li>
+<li><a href="org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.DoubleAvg.html" title="class in org.apache.storm.sql.compiler.backends.standalone" target="classFrame">BuiltinAggregateFunctions.DoubleAvg</a></li>
+<li><a href="org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.DoubleSum.html" title="class in org.apache.storm.sql.compiler.backends.standalone" target="classFrame">BuiltinAggregateFunctions.DoubleSum</a></li>
+<li><a href="org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.IntAvg.html" title="class in org.apache.storm.sql.compiler.backends.standalone" target="classFrame">BuiltinAggregateFunctions.IntAvg</a></li>
+<li><a href="org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.IntSum.html" title="class in org.apache.storm.sql.compiler.backends.standalone" target="classFrame">BuiltinAggregateFunctions.IntSum</a></li>
+<li><a href="org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.Max.html" title="class in org.apache.storm.sql.compiler.backends.standalone" target="classFrame">BuiltinAggregateFunctions.Max</a></li>
+<li><a href="org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.Min.html" title="class in org.apache.storm.sql.compiler.backends.standalone" target="classFrame">BuiltinAggregateFunctions.Min</a></li>
+<li><a href="org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.TypeClass.html" title="class in org.apache.storm.sql.compiler.backends.standalone" target="classFrame">BuiltinAggregateFunctions.TypeClass</a></li>
+<li><a href="org/apache/storm/sql/compiler/backends/standalone/BuiltinAggregateFunctions.TypeClass.GenericType.html" title="class in org.apache.storm.sql.compiler.backends.standalone" target="classFrame">BuiltinAggregateFunctions.TypeClass.GenericType</a></li>
<li><a href="org/apache/storm/daemon/metrics/BuiltinBoltMetrics.html" title="class in org.apache.storm.daemon.metrics" target="classFrame">BuiltinBoltMetrics</a></li>
<li><a href="org/apache/storm/daemon/metrics/BuiltinMetrics.html" title="class in org.apache.storm.daemon.metrics" target="classFrame">BuiltinMetrics</a></li>
<li><a href="org/apache/storm/daemon/metrics/BuiltinMetricsUtil.html" title="class in org.apache.storm.daemon.metrics" target="classFrame">BuiltinMetricsUtil</a></li>
@@ -200,7 +213,6 @@
<li><a href="org/apache/storm/spout/CheckPointState.Action.html" title="enum in org.apache.storm.spout" target="classFrame">CheckPointState.Action</a></li>
<li><a href="org/apache/storm/spout/CheckPointState.State.html" title="enum in org.apache.storm.spout" target="classFrame">CheckPointState.State</a></li>
<li><a href="org/apache/storm/topology/CheckpointTupleForwarder.html" title="class in org.apache.storm.topology" target="classFrame">CheckpointTupleForwarder</a></li>
-<li><a href="org/apache/storm/topology/CheckpointTupleForwarder.AnchoringOutputCollector.html" title="class in org.apache.storm.topology" target="classFrame">CheckpointTupleForwarder.AnchoringOutputCollector</a></li>
<li><a href="org/apache/storm/command/CLI.html" title="class in org.apache.storm.command" target="classFrame">CLI</a></li>
<li><a href="org/apache/storm/command/CLI.Assoc.html" title="interface in org.apache.storm.command" target="classFrame"><span class="interfaceName">CLI.Assoc</span></a></li>
<li><a href="org/apache/storm/command/CLI.CLIBuilder.html" title="class in org.apache.storm.command" target="classFrame">CLI.CLIBuilder</a></li>
@@ -411,7 +423,6 @@
<li><a href="org/apache/storm/kafka/bolt/selector/DefaultTopicSelector.html" title="class in org.apache.storm.kafka.bolt.selector" target="classFrame">DefaultTopicSelector</a></li>
<li><a href="org/apache/storm/kafka/trident/selector/DefaultTopicSelector.html" title="class in org.apache.storm.kafka.trident.selector" target="classFrame">DefaultTopicSelector</a></li>
<li><a href="org/apache/storm/nimbus/DefaultTopologyValidator.html" title="class in org.apache.storm.nimbus" target="classFrame">DefaultTopologyValidator</a></li>
-<li><a href="org/apache/storm/daemon/supervisor/DefaultUncaughtExceptionHandler.html" title="class in org.apache.storm.daemon.supervisor" target="classFrame">DefaultUncaughtExceptionHandler</a></li>
<li><a href="org/apache/storm/callback/DefaultWatcherCallBack.html" title="class in org.apache.storm.callback" target="classFrame">DefaultWatcherCallBack</a></li>
<li><a href="org/apache/storm/daemon/supervisor/workermanager/DefaultWorkerManager.html" title="class in org.apache.storm.daemon.supervisor.workermanager" target="classFrame">DefaultWorkerManager</a></li>
<li><a href="org/apache/storm/hdfs/bolt/format/DelimitedRecordFormat.html" title="class in org.apache.storm.hdfs.bolt.format" target="classFrame">DelimitedRecordFormat</a></li>
@@ -520,7 +531,6 @@
<li><a href="org/apache/storm/metric/EventLoggerBolt.html" title="class in org.apache.storm.metric" target="classFrame">EventLoggerBolt</a></li>
<li><a href="org/apache/storm/event/EventManager.html" title="interface in org.apache.storm.event" target="classFrame"><span class="interfaceName">EventManager</span></a></li>
<li><a href="org/apache/storm/event/EventManagerImp.html" title="class in org.apache.storm.event" target="classFrame">EventManagerImp</a></li>
-<li><a href="org/apache/storm/daemon/supervisor/EventManagerPushCallback.html" title="class in org.apache.storm.daemon.supervisor" target="classFrame">EventManagerPushCallback</a></li>
<li><a href="org/apache/storm/windowing/EvictionPolicy.html" title="interface in org.apache.storm.windowing" target="classFrame"><span class="interfaceName">EvictionPolicy</span></a></li>
<li><a href="org/apache/storm/windowing/EvictionPolicy.Action.html" title="enum in org.apache.storm.windowing" target="classFrame">EvictionPolicy.Action</a></li>
<li><a href="org/apache/storm/starter/ExclamationTopology.html" title="class in org.apache.storm.starter" target="classFrame">ExclamationTopology</a></li>
@@ -1700,11 +1710,10 @@
<li><a href="org/apache/storm/generated/SpoutStats._Fields.html" title="enum in org.apache.storm.generated" target="classFrame">SpoutStats._Fields</a></li>
<li><a href="org/apache/storm/daemon/metrics/SpoutThrottlingMetrics.html" title="class in org.apache.storm.daemon.metrics" target="classFrame">SpoutThrottlingMetrics</a></li>
<li><a href="org/apache/storm/testing/SpoutTracker.html" title="class in org.apache.storm.testing" target="classFrame">SpoutTracker</a></li>
+<li><a href="org/apache/storm/sql/parser/SqlCreateFunction.html" title="class in org.apache.storm.sql.parser" target="classFrame">SqlCreateFunction</a></li>
<li><a href="org/apache/storm/sql/parser/SqlCreateTable.html" title="class in org.apache.storm.sql.parser" target="classFrame">SqlCreateTable</a></li>
<li><a href="org/apache/storm/sql/parser/SqlDDLKeywords.html" title="enum in org.apache.storm.sql.parser" target="classFrame">SqlDDLKeywords</a></li>
<li><a href="org/apache/storm/mqtt/common/SslUtils.html" title="class in org.apache.storm.mqtt.common" target="classFrame">SslUtils</a></li>
-<li><a href="org/apache/storm/daemon/supervisor/StandaloneSupervisor.html" title="class in org.apache.storm.daemon.supervisor" target="classFrame">StandaloneSupervisor</a></li>
-<li><a href="org/apache/storm/daemon/supervisor/State.html" title="enum in org.apache.storm.daemon.supervisor" target="classFrame">State</a></li>
<li><a href="org/apache/storm/state/State.html" title="interface in org.apache.storm.state" target="classFrame"><span class="interfaceName">State</span></a></li>
<li><a href="org/apache/storm/trident/state/State.html" title="interface in org.apache.storm.trident.state" target="classFrame"><span class="interfaceName">State</span></a></li>
<li><a href="org/apache/storm/state/StateFactory.html" title="class in org.apache.storm.state" target="classFrame">StateFactory</a></li>
@@ -1714,7 +1723,6 @@
<li><a href="storm/starter/StatefulTopology.PrinterBolt.html" title="class in storm.starter" target="classFrame">StatefulTopology.PrinterBolt</a></li>
<li><a href="org/apache/storm/topology/StatefulWindowedBoltExecutor.html" title="class in org.apache.storm.topology" target="classFrame">StatefulWindowedBoltExecutor</a></li>
<li><a href="storm/starter/StatefulWindowingTopology.html" title="class in storm.starter" target="classFrame">StatefulWindowingTopology</a></li>
-<li><a href="org/apache/storm/daemon/supervisor/StateHeartbeat.html" title="class in org.apache.storm.daemon.supervisor" target="classFrame">StateHeartbeat</a></li>
<li><a href="org/apache/storm/metric/api/StateMetric.html" title="class in org.apache.storm.metric.api" target="classFrame">StateMetric</a></li>
<li><a href="org/apache/storm/state/StateProvider.html" title="interface in org.apache.storm.state" target="classFrame"><span class="interfaceName">StateProvider</span></a></li>
<li><a href="org/apache/storm/trident/planner/processor/StateQueryProcessor.html" title="class in org.apache.storm.trident.planner.processor" target="classFrame">StateQueryProcessor</a></li>
@@ -1777,23 +1785,16 @@
<li><a href="org/apache/storm/trident/planner/SubtopologyBolt.html" title="class in org.apache.storm.trident.planner" target="classFrame">SubtopologyBolt</a></li>
<li><a href="org/apache/storm/trident/planner/SubtopologyBolt.InitialReceiver.html" title="class in org.apache.storm.trident.planner" target="classFrame">SubtopologyBolt.InitialReceiver</a></li>
<li><a href="org/apache/storm/trident/operation/builtin/Sum.html" title="class in org.apache.storm.trident.operation.builtin" target="classFrame">Sum</a></li>
-<li><a href="org/apache/storm/daemon/supervisor/Supervisor.html" title="class in org.apache.storm.daemon.supervisor" target="classFrame">Supervisor</a></li>
-<li><a href="org/apache/storm/daemon/supervisor/SupervisorDaemon.html" title="interface in org.apache.storm.daemon.supervisor" target="classFrame"><span class="interfaceName">SupervisorDaemon</span></a></li>
-<li><a href="org/apache/storm/daemon/supervisor/SupervisorData.html" title="class in org.apache.storm.daemon.supervisor" target="classFrame">SupervisorData</a></li>
<li><a href="org/apache/storm/scheduler/SupervisorDetails.html" title="class in org.apache.storm.scheduler" target="classFrame">SupervisorDetails</a></li>
<li><a href="org/apache/storm/daemon/supervisor/timer/SupervisorHealthCheck.html" title="class in org.apache.storm.daemon.supervisor.timer" target="classFrame">SupervisorHealthCheck</a></li>
<li><a href="org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeat.html" title="class in org.apache.storm.daemon.supervisor.timer" target="classFrame">SupervisorHeartbeat</a></li>
<li><a href="org/apache/storm/generated/SupervisorInfo.html" title="class in org.apache.storm.generated" target="classFrame">SupervisorInfo</a></li>
<li><a href="org/apache/storm/generated/SupervisorInfo._Fields.html" title="enum in org.apache.storm.generated" target="classFrame">SupervisorInfo._Fields</a></li>
-<li><a href="org/apache/storm/daemon/supervisor/SupervisorManager.html" title="class in org.apache.storm.daemon.supervisor" target="classFrame">SupervisorManager</a></li>
<li><a href="org/apache/storm/generated/SupervisorSummary.html" title="class in org.apache.storm.generated" target="classFrame">SupervisorSummary</a></li>
<li><a href="org/apache/storm/generated/SupervisorSummary._Fields.html" title="enum in org.apache.storm.generated" target="classFrame">SupervisorSummary._Fields</a></li>
-<li><a href="org/apache/storm/daemon/supervisor/SupervisorUtils.html" title="class in org.apache.storm.daemon.supervisor" target="classFrame">SupervisorUtils</a></li>
<li><a href="org/apache/storm/state/SynchronizeOutputCollector.html" title="class in org.apache.storm.state" target="classFrame">SynchronizeOutputCollector</a></li>
<li><a href="org/apache/storm/hdfs/bolt/sync/SyncPolicy.html" title="interface in org.apache.storm.hdfs.bolt.sync" target="classFrame"><span class="interfaceName">SyncPolicy</span></a></li>
<li><a href="org/apache/storm/hdfs/trident/sync/SyncPolicy.html" title="interface in org.apache.storm.hdfs.trident.sync" target="classFrame"><span class="interfaceName">SyncPolicy</span></a></li>
-<li><a href="org/apache/storm/daemon/supervisor/SyncProcessEvent.html" title="class in org.apache.storm.daemon.supervisor" target="classFrame">SyncProcessEvent</a></li>
-<li><a href="org/apache/storm/daemon/supervisor/SyncSupervisorEvent.html" title="class in org.apache.storm.daemon.supervisor" target="classFrame">SyncSupervisorEvent</a></li>
<li><a href="org/apache/storm/metric/SystemBolt.html" title="class in org.apache.storm.metric" target="classFrame">SystemBolt</a></li>
<li><a href="org/apache/storm/container/cgroup/SystemOperation.html" title="class in org.apache.storm.container.cgroup" target="classFrame">SystemOperation</a></li>
<li><a href="org/apache/storm/daemon/Task.html" title="class in org.apache.storm.daemon" target="classFrame">Task</a></li>
@@ -1997,6 +1998,7 @@
<li><a href="org/apache/storm/trident/windowing/config/WindowConfig.Type.html" title="enum in org.apache.storm.trident.windowing.config" target="classFrame">WindowConfig.Type</a></li>
<li><a href="org/apache/storm/topology/WindowedBoltExecutor.html" title="class in org.apache.storm.topology" target="classFrame">WindowedBoltExecutor</a></li>
<li><a href="org/apache/storm/utils/WindowedTimeThrottler.html" title="class in org.apache.storm.utils" target="classFrame">WindowedTimeThrottler</a></li>
+<li><a href="org/apache/storm/trident/windowing/WindowKryoSerializer.html" title="class in org.apache.storm.trident.windowing" target="classFrame">WindowKryoSerializer</a></li>
<li><a href="org/apache/storm/windowing/WindowLifecycleListener.html" title="interface in org.apache.storm.windowing" target="classFrame"><span class="interfaceName">WindowLifecycleListener</span></a></li>
<li><a href="org/apache/storm/windowing/WindowManager.html" title="class in org.apache.storm.windowing" target="classFrame">WindowManager</a></li>
<li><a href="org/apache/storm/trident/windowing/WindowsState.html" title="class in org.apache.storm.trident.windowing" target="classFrame">WindowsState</a></li>