You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by ka...@apache.org on 2016/08/23 01:13:59 UTC
svn commit: r1757294 [1/2] - in /storm/site/publish: ./ releases/0.10.0/
releases/0.10.1/ releases/1.0.0/ releases/1.0.1/ releases/1.0.2/
releases/2.0.0-SNAPSHOT/ releases/current/
Author: kabhwan
Date: Tue Aug 23 01:13:59 2016
New Revision: 1757294
URL: http://svn.apache.org/viewvc?rev=1757294&view=rev
Log:
publish the latest changes
Modified:
storm/site/publish/feed.xml
storm/site/publish/releases/0.10.0/feed.xml
storm/site/publish/releases/0.10.1/feed.xml
storm/site/publish/releases/1.0.0/SECURITY.html
storm/site/publish/releases/1.0.0/flux.html
storm/site/publish/releases/1.0.1/SECURITY.html
storm/site/publish/releases/1.0.1/flux.html
storm/site/publish/releases/1.0.2/SECURITY.html
storm/site/publish/releases/1.0.2/flux.html
storm/site/publish/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html
storm/site/publish/releases/2.0.0-SNAPSHOT/SECURITY.html
storm/site/publish/releases/2.0.0-SNAPSHOT/STORM-UI-REST-API.html
storm/site/publish/releases/2.0.0-SNAPSHOT/Windowing.html
storm/site/publish/releases/2.0.0-SNAPSHOT/flux.html
storm/site/publish/releases/current/SECURITY.html
storm/site/publish/releases/current/flux.html
Modified: storm/site/publish/feed.xml
URL: http://svn.apache.org/viewvc/storm/site/publish/feed.xml?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/feed.xml (original)
+++ storm/site/publish/feed.xml Tue Aug 23 01:13:59 2016
@@ -5,8 +5,8 @@
<description></description>
<link>http://storm.apache.org/</link>
<atom:link href="http://storm.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Fri, 19 Aug 2016 13:27:21 +0900</pubDate>
- <lastBuildDate>Fri, 19 Aug 2016 13:27:21 +0900</lastBuildDate>
+ <pubDate>Tue, 23 Aug 2016 09:55:48 +0900</pubDate>
+ <lastBuildDate>Tue, 23 Aug 2016 09:55:48 +0900</lastBuildDate>
<generator>Jekyll v3.1.2</generator>
<item>
Modified: storm/site/publish/releases/0.10.0/feed.xml
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/0.10.0/feed.xml?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/0.10.0/feed.xml (original)
+++ storm/site/publish/releases/0.10.0/feed.xml Tue Aug 23 01:13:59 2016
@@ -5,8 +5,8 @@
<description></description>
<link>http://storm.apache.org/</link>
<atom:link href="http://storm.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Fri, 19 Aug 2016 13:27:21 +0900</pubDate>
- <lastBuildDate>Fri, 19 Aug 2016 13:27:21 +0900</lastBuildDate>
+ <pubDate>Tue, 23 Aug 2016 09:55:48 +0900</pubDate>
+ <lastBuildDate>Tue, 23 Aug 2016 09:55:48 +0900</lastBuildDate>
<generator>Jekyll v3.1.2</generator>
<item>
Modified: storm/site/publish/releases/0.10.1/feed.xml
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/0.10.1/feed.xml?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/0.10.1/feed.xml (original)
+++ storm/site/publish/releases/0.10.1/feed.xml Tue Aug 23 01:13:59 2016
@@ -5,8 +5,8 @@
<description></description>
<link>http://storm.apache.org/</link>
<atom:link href="http://storm.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Fri, 19 Aug 2016 13:27:21 +0900</pubDate>
- <lastBuildDate>Fri, 19 Aug 2016 13:27:21 +0900</lastBuildDate>
+ <pubDate>Tue, 23 Aug 2016 09:55:48 +0900</pubDate>
+ <lastBuildDate>Tue, 23 Aug 2016 09:55:48 +0900</lastBuildDate>
<generator>Jekyll v3.1.2</generator>
<item>
Modified: storm/site/publish/releases/1.0.0/SECURITY.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/1.0.0/SECURITY.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/1.0.0/SECURITY.html (original)
+++ storm/site/publish/releases/1.0.0/SECURITY.html Tue Aug 23 01:13:59 2016
@@ -266,9 +266,11 @@ curl -i --negotiate -u:anyUser -b ~/co
<ol>
<li>Firefox: Goto about:config and search for network.negotiate-auth.trusted-uris double-click to add value "<a href="http://storm-ui-hostname:8080">http://storm-ui-hostname:8080</a>"</li>
<li>Google-chrome: start from command line with: google-chrome --auth-server-whitelist="<em>storm-ui-hostname" --auth-negotiate-delegate-whitelist="</em>storm-ui-hostname"<br></li>
-<li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website </li>
+<li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website</li>
</ol>
+<p><strong>Note</strong>: For viewing any logs via <code>logviewer</code> in secure mode, all the hosts that runs <code>logviewer</code> should also be added to the above white list. For big clusters you could white list the host's domain (for e.g. set <code>network.negotiate-auth.trusted-uris</code> to <code>.yourdomain.com</code>).</p>
+
<p><strong>Caution</strong>: In AD MIT Keberos setup the key size is bigger than the default UI jetty server request header size. Make sure you set ui.header.buffer.bytes to 65536 in storm.yaml. More details are on <a href="https://issues.apache.org/jira/browse/STORM-633">STORM-633</a></p>
<h2 id="ui-drpc-ssl">UI / DRPC SSL</h2>
Modified: storm/site/publish/releases/1.0.0/flux.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/1.0.0/flux.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/1.0.0/flux.html (original)
+++ storm/site/publish/releases/1.0.0/flux.html Tue Aug 23 01:13:59 2016
@@ -253,6 +253,15 @@ recommended).</p>
</dependency>
</code></p>
+<p>Using shell spouts and bolts requires additional Flux Wrappers library:
+<code>xml
+<dependency>
+ <groupId>org.apache.storm</groupId>
+ <artifactId>flux-wrappers</artifactId>
+ <version>${storm.version}</version>
+</dependency>
+</code></p>
+
<h4 id="creating-a-flux-enabled-topology-jar">Creating a Flux-Enabled Topology JAR</h4>
<p>The example below illustrates Flux usage with the Maven shade plugin:</p>
@@ -264,6 +273,12 @@ recommended).</p>
<span class="nt"><artifactId></span>flux-core<span class="nt"></artifactId></span>
<span class="nt"><version></span>${storm.version}<span class="nt"></version></span>
<span class="nt"></dependency></span>
+ <span class="c"><!-- Flux Wrappers include --></span>
+ <span class="nt"><dependency></span>
+ <span class="nt"><groupId></span>org.apache.storm<span class="nt"></groupId></span>
+ <span class="nt"><artifactId></span>flux-wrappers<span class="nt"></artifactId></span>
+ <span class="nt"><version></span>${storm.version}<span class="nt"></version></span>
+ <span class="nt"></dependency></span>
<span class="c"><!-- add user dependencies here... --></span>
@@ -359,9 +374,9 @@ Parsing file: /Users/hsimpson/Projects/d
---------- TOPOLOGY DETAILS ----------
Name: shell-topology
--------------- SPOUTS ---------------
-sentence-spout[1](org.apache.storm.flux.spouts.GenericShellSpout)
+sentence-spout[1](org.apache.storm.flux.wrappers.spouts.FluxShellSpout)
---------------- BOLTS ---------------
-splitsentence[1](org.apache.storm.flux.bolts.GenericShellBolt)
+splitsentence[1](org.apache.storm.flux.wrappers.bolts.FluxShellBolt)
log[1](org.apache.storm.flux.wrappers.bolts.LogInfoBolt)
count[1](org.apache.storm.testing.TestWordCounter)
--------------- STREAMS ---------------
@@ -637,7 +652,7 @@ well.</p>
<p>Shell spout example:</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">spouts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">sentence-spout"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.spouts.GenericShellSpout"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.spouts.FluxShellSpout"</span>
<span class="c1"># shell spout constructor takes 2 arguments: String[], String[]</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
@@ -704,7 +719,7 @@ well.</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="c1"># bolt definitions</span>
<span class="s">bolts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">splitsentence"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.bolts.GenericShellBolt"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.bolts.FluxShellBolt"</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
<span class="pi">-</span> <span class="pi">[</span><span class="s2">"</span><span class="s">python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">splitsentence.py"</span><span class="pi">]</span>
@@ -826,7 +841,7 @@ same file. Includes may be either files,
<span class="c1"># spout definitions</span>
<span class="s">spouts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">sentence-spout"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.spouts.GenericShellSpout"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.spouts.FluxShellSpout"</span>
<span class="c1"># shell spout constructor takes 2 arguments: String[], String[]</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
@@ -838,7 +853,7 @@ same file. Includes may be either files,
<span class="c1"># bolt definitions</span>
<span class="s">bolts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">splitsentence"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.bolts.GenericShellBolt"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.bolts.FluxShellBolt"</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
<span class="pi">-</span> <span class="pi">[</span><span class="s2">"</span><span class="s">python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">splitsentence.py"</span><span class="pi">]</span>
Modified: storm/site/publish/releases/1.0.1/SECURITY.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/1.0.1/SECURITY.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/1.0.1/SECURITY.html (original)
+++ storm/site/publish/releases/1.0.1/SECURITY.html Tue Aug 23 01:13:59 2016
@@ -266,9 +266,11 @@ curl -i --negotiate -u:anyUser -b ~/co
<ol>
<li>Firefox: Goto about:config and search for network.negotiate-auth.trusted-uris double-click to add value "<a href="http://storm-ui-hostname:8080">http://storm-ui-hostname:8080</a>"</li>
<li>Google-chrome: start from command line with: google-chrome --auth-server-whitelist="<em>storm-ui-hostname" --auth-negotiate-delegate-whitelist="</em>storm-ui-hostname"<br></li>
-<li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website </li>
+<li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website</li>
</ol>
+<p><strong>Note</strong>: For viewing any logs via <code>logviewer</code> in secure mode, all the hosts that runs <code>logviewer</code> should also be added to the above white list. For big clusters you could white list the host's domain (for e.g. set <code>network.negotiate-auth.trusted-uris</code> to <code>.yourdomain.com</code>).</p>
+
<p><strong>Caution</strong>: In AD MIT Keberos setup the key size is bigger than the default UI jetty server request header size. Make sure you set ui.header.buffer.bytes to 65536 in storm.yaml. More details are on <a href="https://issues.apache.org/jira/browse/STORM-633">STORM-633</a></p>
<h2 id="ui-drpc-ssl">UI / DRPC SSL</h2>
Modified: storm/site/publish/releases/1.0.1/flux.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/1.0.1/flux.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/1.0.1/flux.html (original)
+++ storm/site/publish/releases/1.0.1/flux.html Tue Aug 23 01:13:59 2016
@@ -253,6 +253,15 @@ recommended).</p>
</dependency>
</code></p>
+<p>Using shell spouts and bolts requires additional Flux Wrappers library:
+<code>xml
+<dependency>
+ <groupId>org.apache.storm</groupId>
+ <artifactId>flux-wrappers</artifactId>
+ <version>${storm.version}</version>
+</dependency>
+</code></p>
+
<h4 id="creating-a-flux-enabled-topology-jar">Creating a Flux-Enabled Topology JAR</h4>
<p>The example below illustrates Flux usage with the Maven shade plugin:</p>
@@ -264,6 +273,12 @@ recommended).</p>
<span class="nt"><artifactId></span>flux-core<span class="nt"></artifactId></span>
<span class="nt"><version></span>${storm.version}<span class="nt"></version></span>
<span class="nt"></dependency></span>
+ <span class="c"><!-- Flux Wrappers include --></span>
+ <span class="nt"><dependency></span>
+ <span class="nt"><groupId></span>org.apache.storm<span class="nt"></groupId></span>
+ <span class="nt"><artifactId></span>flux-wrappers<span class="nt"></artifactId></span>
+ <span class="nt"><version></span>${storm.version}<span class="nt"></version></span>
+ <span class="nt"></dependency></span>
<span class="c"><!-- add user dependencies here... --></span>
@@ -359,9 +374,9 @@ Parsing file: /Users/hsimpson/Projects/d
---------- TOPOLOGY DETAILS ----------
Name: shell-topology
--------------- SPOUTS ---------------
-sentence-spout[1](org.apache.storm.flux.spouts.GenericShellSpout)
+sentence-spout[1](org.apache.storm.flux.wrappers.spouts.FluxShellSpout)
---------------- BOLTS ---------------
-splitsentence[1](org.apache.storm.flux.bolts.GenericShellBolt)
+splitsentence[1](org.apache.storm.flux.wrappers.bolts.FluxShellBolt)
log[1](org.apache.storm.flux.wrappers.bolts.LogInfoBolt)
count[1](org.apache.storm.testing.TestWordCounter)
--------------- STREAMS ---------------
@@ -637,7 +652,7 @@ well.</p>
<p>Shell spout example:</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">spouts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">sentence-spout"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.spouts.GenericShellSpout"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.spouts.FluxShellSpout"</span>
<span class="c1"># shell spout constructor takes 2 arguments: String[], String[]</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
@@ -704,7 +719,7 @@ well.</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="c1"># bolt definitions</span>
<span class="s">bolts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">splitsentence"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.bolts.GenericShellBolt"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.bolts.FluxShellBolt"</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
<span class="pi">-</span> <span class="pi">[</span><span class="s2">"</span><span class="s">python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">splitsentence.py"</span><span class="pi">]</span>
@@ -826,7 +841,7 @@ same file. Includes may be either files,
<span class="c1"># spout definitions</span>
<span class="s">spouts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">sentence-spout"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.spouts.GenericShellSpout"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.spouts.FluxShellSpout"</span>
<span class="c1"># shell spout constructor takes 2 arguments: String[], String[]</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
@@ -838,7 +853,7 @@ same file. Includes may be either files,
<span class="c1"># bolt definitions</span>
<span class="s">bolts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">splitsentence"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.bolts.GenericShellBolt"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.bolts.FluxShellBolt"</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
<span class="pi">-</span> <span class="pi">[</span><span class="s2">"</span><span class="s">python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">splitsentence.py"</span><span class="pi">]</span>
Modified: storm/site/publish/releases/1.0.2/SECURITY.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/1.0.2/SECURITY.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/1.0.2/SECURITY.html (original)
+++ storm/site/publish/releases/1.0.2/SECURITY.html Tue Aug 23 01:13:59 2016
@@ -266,9 +266,11 @@ curl -i --negotiate -u:anyUser -b ~/co
<ol>
<li>Firefox: Goto about:config and search for network.negotiate-auth.trusted-uris double-click to add value "<a href="http://storm-ui-hostname:8080">http://storm-ui-hostname:8080</a>"</li>
<li>Google-chrome: start from command line with: google-chrome --auth-server-whitelist="<em>storm-ui-hostname" --auth-negotiate-delegate-whitelist="</em>storm-ui-hostname"<br></li>
-<li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website </li>
+<li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website</li>
</ol>
+<p><strong>Note</strong>: For viewing any logs via <code>logviewer</code> in secure mode, all the hosts that runs <code>logviewer</code> should also be added to the above white list. For big clusters you could white list the host's domain (for e.g. set <code>network.negotiate-auth.trusted-uris</code> to <code>.yourdomain.com</code>).</p>
+
<p><strong>Caution</strong>: In AD MIT Keberos setup the key size is bigger than the default UI jetty server request header size. Make sure you set ui.header.buffer.bytes to 65536 in storm.yaml. More details are on <a href="https://issues.apache.org/jira/browse/STORM-633">STORM-633</a></p>
<h2 id="ui-drpc-ssl">UI / DRPC SSL</h2>
Modified: storm/site/publish/releases/1.0.2/flux.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/1.0.2/flux.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/1.0.2/flux.html (original)
+++ storm/site/publish/releases/1.0.2/flux.html Tue Aug 23 01:13:59 2016
@@ -253,6 +253,15 @@ recommended).</p>
</dependency>
</code></p>
+<p>Using shell spouts and bolts requires additional Flux Wrappers library:
+<code>xml
+<dependency>
+ <groupId>org.apache.storm</groupId>
+ <artifactId>flux-wrappers</artifactId>
+ <version>${storm.version}</version>
+</dependency>
+</code></p>
+
<h4 id="creating-a-flux-enabled-topology-jar">Creating a Flux-Enabled Topology JAR</h4>
<p>The example below illustrates Flux usage with the Maven shade plugin:</p>
@@ -264,6 +273,12 @@ recommended).</p>
<span class="nt"><artifactId></span>flux-core<span class="nt"></artifactId></span>
<span class="nt"><version></span>${storm.version}<span class="nt"></version></span>
<span class="nt"></dependency></span>
+ <span class="c"><!-- Flux Wrappers include --></span>
+ <span class="nt"><dependency></span>
+ <span class="nt"><groupId></span>org.apache.storm<span class="nt"></groupId></span>
+ <span class="nt"><artifactId></span>flux-wrappers<span class="nt"></artifactId></span>
+ <span class="nt"><version></span>${storm.version}<span class="nt"></version></span>
+ <span class="nt"></dependency></span>
<span class="c"><!-- add user dependencies here... --></span>
@@ -359,9 +374,9 @@ Parsing file: /Users/hsimpson/Projects/d
---------- TOPOLOGY DETAILS ----------
Name: shell-topology
--------------- SPOUTS ---------------
-sentence-spout[1](org.apache.storm.flux.spouts.GenericShellSpout)
+sentence-spout[1](org.apache.storm.flux.wrappers.spouts.FluxShellSpout)
---------------- BOLTS ---------------
-splitsentence[1](org.apache.storm.flux.bolts.GenericShellBolt)
+splitsentence[1](org.apache.storm.flux.wrappers.bolts.FluxShellBolt)
log[1](org.apache.storm.flux.wrappers.bolts.LogInfoBolt)
count[1](org.apache.storm.testing.TestWordCounter)
--------------- STREAMS ---------------
@@ -637,7 +652,7 @@ well.</p>
<p>Shell spout example:</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">spouts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">sentence-spout"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.spouts.GenericShellSpout"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.spouts.FluxShellSpout"</span>
<span class="c1"># shell spout constructor takes 2 arguments: String[], String[]</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
@@ -704,7 +719,7 @@ well.</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="c1"># bolt definitions</span>
<span class="s">bolts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">splitsentence"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.bolts.GenericShellBolt"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.bolts.FluxShellBolt"</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
<span class="pi">-</span> <span class="pi">[</span><span class="s2">"</span><span class="s">python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">splitsentence.py"</span><span class="pi">]</span>
@@ -826,7 +841,7 @@ same file. Includes may be either files,
<span class="c1"># spout definitions</span>
<span class="s">spouts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">sentence-spout"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.spouts.GenericShellSpout"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.spouts.FluxShellSpout"</span>
<span class="c1"># shell spout constructor takes 2 arguments: String[], String[]</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
@@ -838,7 +853,7 @@ same file. Includes may be either files,
<span class="c1"># bolt definitions</span>
<span class="s">bolts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">splitsentence"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.bolts.GenericShellBolt"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.bolts.FluxShellBolt"</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
<span class="pi">-</span> <span class="pi">[</span><span class="s2">"</span><span class="s">python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">splitsentence.py"</span><span class="pi">]</span>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html Tue Aug 23 01:13:59 2016
@@ -306,6 +306,53 @@ DEV => 20 – 29
<p><a href="http://web.engr.illinois.edu/%7Ebpeng/files/r-storm.pdf">http://web.engr.illinois.edu/~bpeng/files/r-storm.pdf</a></p>
+<h4 id="enhancements-on-original-defaultresourceawarestrategy">* Enhancements on original DefaultResourceAwareStrategy *</h4>
+
+<p>Originally the getBestClustering algorithm for RAS finds the "Best" cluster/rack based on which rack has the "most available" resources by finding the rack with the biggest sum of available memory + available across all nodes in the rack. This method is not very accurate since memory and cpu usage aree values on a different scale and the values are not normailized. This method is also not effective since it does not consider the number of slots available and it fails to identifying racks that are not schedulable due to the exhaustion of one of the resources either memory, cpu, or slots. Also the current method does not consider failures of workers. When executors of a topology gets unassigned and needs to be scheduled again, the current logic in getBestClustering may be inadequate since it will likely return a cluster that is different from where the majority of executors from the topology is originally scheduling in.</p>
+
+<p>The new strategy/algorithm to find the "best" cluster, I dub subordinate resource availability ordering (inspired by Dominant Resource Fairness), sorts racks by the subordinate (not dominant) resource availability.</p>
+
+<p>For example given 4 racks with the following resource availabilities</p>
+<div class="highlight"><pre><code class="language-" data-lang="">//generate some that has alot of memory but little of cpu
+rack-3 Avail [ CPU 100.0 MEM 200000.0 Slots 40 ] Total [ CPU 100.0 MEM 200000.0 Slots 40 ]
+//generate some supervisors that are depleted of one resource
+rack-2 Avail [ CPU 0.0 MEM 80000.0 Slots 40 ] Total [ CPU 0.0 MEM 80000.0 Slots 40 ]
+//generate some that has a lot of cpu but little of memory
+rack-4 Avail [ CPU 6100.0 MEM 10000.0 Slots 40 ] Total [ CPU 6100.0 MEM 10000.0 Slots 40 ]
+//generate another rack of supervisors with less resources than rack-0
+rack-1 Avail [ CPU 2000.0 MEM 40000.0 Slots 40 ] Total [ CPU 2000.0 MEM 40000.0 Slots 40 ]
+//best rack to choose
+rack-0 Avail [ CPU 4000.0 MEM 80000.0 Slots 40( ] Total [ CPU 4000.0 MEM 80000.0 Slots 40 ]
+Cluster Overall Avail [ CPU 12200.0 MEM 410000.0 Slots 200 ] Total [ CPU 12200.0 MEM 410000.0 Slots 200 ]
+</code></pre></div>
+<p>It is clear that rack-0 is the best cluster since its the most balanced and can potentially schedule the most executors, while rack-2 is the worst rack since rack-2 is depleted of cpu resource thus rendering it unschedulable even though there are other resources available.</p>
+
+<p>We first calculate the resource availability percentage of all the racks for each resource by computing:</p>
+<div class="highlight"><pre><code class="language-" data-lang="">(resource available on rack) / (resource available in cluster)
+</code></pre></div>
+<p>We do this calculation to normalize the values otherwise the resource values would not be comparable.</p>
+
+<p>So for our example:</p>
+<div class="highlight"><pre><code class="language-" data-lang="">rack-3 Avail [ CPU 0.819672131147541% MEM 48.78048780487805% Slots 20.0% ] effective resources: 0.00819672131147541
+rack-2 Avail [ 0.0% MEM 19.51219512195122% Slots 20.0% ] effective resources: 0.0
+rack-4 Avail [ CPU 50.0% MEM 2.4390243902439024% Slots 20.0% ] effective resources: 0.024390243902439025
+rack-1 Avail [ CPU 16.39344262295082% MEM 9.75609756097561% Slots 20.0% ] effective resources: 0.0975609756097561
+rack-0 Avail [ CPU 32.78688524590164% MEM 19.51219512195122% Slots 20.0% ] effective resources: 0.1951219512195122
+</code></pre></div>
+<p>The effective resource of a rack, which is also the subordinate resource, is computed by: </p>
+<div class="highlight"><pre><code class="language-" data-lang="">MIN(resource availability percentage of {CPU, Memory, # of free Slots}).
+</code></pre></div>
+<p>Then we order the racks by the effective resource.</p>
+
+<p>Thus for our example:</p>
+<div class="highlight"><pre><code class="language-" data-lang="">Sorted rack: [rack-0, rack-1, rack-4, rack-3, rack-2]
+</code></pre></div>
+<p>Also to deal with the presence of failures, if a topology is partially scheduled, we find the rack with the most scheduled executors for the topology and we try to schedule on that rack first.</p>
+
+<p>Thus we first sort by the number of executors already scheduled on the rack and then by the subordinate resource availability.</p>
+
+<p>Original Jira for this enhancement: <a href="https://issues.apache.org/jira/browse/STORM-1766">STORM-1766</a></p>
+
<h3 id="specifying-topology-prioritization-strategy">Specifying Topology Prioritization Strategy</h3>
<p>The order of scheduling is a pluggable interface in which a user could define a strategy that prioritizes topologies. For a user to define his or her own prioritization strategy, he or she needs to implement the ISchedulingPriorityStrategy interface. A user can set the scheduling priority strategy by setting the <em>Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY</em> to point to the class that implements the strategy. For instance:</p>
@@ -363,7 +410,7 @@ DEV => 20 – 29
<p>We should never evict a topology from a user that does not have his or her resource guarantees satisfied. The following flow chart should describe the logic for the eviction process.</p>
-<p><img src="images/resource_aware_scheduler_default_eviction_strategy.svg" alt="Viewing metrics with VisualVM"></p>
+<p><img src="images/resource_aware_scheduler_default_eviction_strategy.png" alt="Viewing metrics with VisualVM"></p>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/SECURITY.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/SECURITY.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/SECURITY.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/SECURITY.html Tue Aug 23 01:13:59 2016
@@ -266,9 +266,11 @@ curl -i --negotiate -u:anyUser -b ~/co
<ol>
<li>Firefox: Goto about:config and search for network.negotiate-auth.trusted-uris double-click to add value "<a href="http://storm-ui-hostname:8080">http://storm-ui-hostname:8080</a>"</li>
<li>Google-chrome: start from command line with: google-chrome --auth-server-whitelist="<em>storm-ui-hostname" --auth-negotiate-delegate-whitelist="</em>storm-ui-hostname"<br></li>
-<li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website </li>
+<li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website</li>
</ol>
+<p><strong>Note</strong>: For viewing any logs via <code>logviewer</code> in secure mode, all the hosts that runs <code>logviewer</code> should also be added to the above white list. For big clusters you could white list the host's domain (for e.g. set <code>network.negotiate-auth.trusted-uris</code> to <code>.yourdomain.com</code>).</p>
+
<p><strong>Caution</strong>: In AD MIT Keberos setup the key size is bigger than the default UI jetty server request header size. Make sure you set ui.header.buffer.bytes to 65536 in storm.yaml. More details are on <a href="https://issues.apache.org/jira/browse/STORM-633">STORM-633</a></p>
<h2 id="ui-drpc-ssl">UI / DRPC SSL</h2>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/STORM-UI-REST-API.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/STORM-UI-REST-API.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/STORM-UI-REST-API.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/STORM-UI-REST-API.html Tue Aug 23 01:13:59 2016
@@ -319,6 +319,11 @@ are used by nimbus.</p>
<td>Number of worker slots used on this supervisor</td>
</tr>
<tr>
+<td>schedulerDisplayResource</td>
+<td>Boolean</td>
+<td>Whether to display scheduler resource information</td>
+</tr>
+<tr>
<td>totalMem</td>
<td>Double</td>
<td>Total memory capacity on this supervisor</td>
@@ -452,6 +457,257 @@ are used by nimbus.</p>
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div>
+<h3 id="api-v1-supervisor-get">/api/v1/supervisor (GET)</h3>
+
+<p>Returns summary for a supervisor by id, or all supervisors running on a host.</p>
+
+<p>Examples:</p>
+<div class="highlight"><pre><code class="language-no-highlight" data-lang="no-highlight"> 1. By host: http://ui-daemon-host-name:8080/api/v1/supervisor?host=supervisor-daemon-host-name
+ 2. By id: http://ui-daemon-host-name:8080/api/v1/supervisor?id=f5449110-1daa-43e2-89e3-69917b16dec9-192.168.1.1
+</code></pre></div>
+<p>Request parameters:</p>
+
+<table><thead>
+<tr>
+<th>Parameter</th>
+<th>Value</th>
+<th>Description</th>
+</tr>
+</thead><tbody>
+<tr>
+<td>id</td>
+<td>String. Supervisor id</td>
+<td>If specified, respond with the supervisor and worker stats with id. Note that when id is specified, the host argument is ignored.</td>
+</tr>
+<tr>
+<td>host</td>
+<td>String. Host name</td>
+<td>If specified, respond with all supervisors and worker stats in the host (normally just one)</td>
+</tr>
+<tr>
+<td>sys</td>
+<td>String. Values 1 or 0. Default value 0</td>
+<td>Controls including sys stats part of the response</td>
+</tr>
+</tbody></table>
+
+<p>Response fields:</p>
+
+<table><thead>
+<tr>
+<th>Field</th>
+<th>Value</th>
+<th>Description</th>
+</tr>
+</thead><tbody>
+<tr>
+<td>supervisors</td>
+<td>Array</td>
+<td>Array of supervisor summaries</td>
+</tr>
+<tr>
+<td>workers</td>
+<td>Array</td>
+<td>Array of worker summaries</td>
+</tr>
+<tr>
+<td>schedulerDisplayResource</td>
+<td>Boolean</td>
+<td>Whether to display scheduler resource information</td>
+</tr>
+</tbody></table>
+
+<p>Each supervisor is defined by:</p>
+
+<table><thead>
+<tr>
+<th>Field</th>
+<th>Value</th>
+<th>Description</th>
+</tr>
+</thead><tbody>
+<tr>
+<td>id</td>
+<td>String</td>
+<td>Supervisor's id</td>
+</tr>
+<tr>
+<td>host</td>
+<td>String</td>
+<td>Supervisor's host name</td>
+</tr>
+<tr>
+<td>uptime</td>
+<td>String</td>
+<td>Shows how long the supervisor is running</td>
+</tr>
+<tr>
+<td>uptimeSeconds</td>
+<td>Integer</td>
+<td>Shows how long the supervisor is running in seconds</td>
+</tr>
+<tr>
+<td>slotsTotal</td>
+<td>Integer</td>
+<td>Total number of worker slots for this supervisor</td>
+</tr>
+<tr>
+<td>slotsUsed</td>
+<td>Integer</td>
+<td>Number of worker slots used on this supervisor</td>
+</tr>
+<tr>
+<td>totalMem</td>
+<td>Double</td>
+<td>Total memory capacity on this supervisor</td>
+</tr>
+<tr>
+<td>totalCpu</td>
+<td>Double</td>
+<td>Total CPU capacity on this supervisor</td>
+</tr>
+<tr>
+<td>usedMem</td>
+<td>Double</td>
+<td>Used memory capacity on this supervisor</td>
+</tr>
+<tr>
+<td>usedCpu</td>
+<td>Double</td>
+<td>Used CPU capacity on this supervisor</td>
+</tr>
+</tbody></table>
+
+<p>Each worker is defined by:</p>
+
+<table><thead>
+<tr>
+<th>Field</th>
+<th>Value</th>
+<th>Description</th>
+</tr>
+</thead><tbody>
+<tr>
+<td>supervisorId</td>
+<td>String</td>
+<td>Supervisor's id</td>
+</tr>
+<tr>
+<td>host</td>
+<td>String</td>
+<td>Worker's host name</td>
+</tr>
+<tr>
+<td>port</td>
+<td>Integer</td>
+<td>Worker's port</td>
+</tr>
+<tr>
+<td>topologyId</td>
+<td>String</td>
+<td>Topology Id</td>
+</tr>
+<tr>
+<td>topologyName</td>
+<td>String</td>
+<td>Topology Name</td>
+</tr>
+<tr>
+<td>executorsTotal</td>
+<td>Integer</td>
+<td>Number of executors used by the topology in this worker</td>
+</tr>
+<tr>
+<td>assignedMemOnHeap</td>
+<td>Double</td>
+<td>Assigned On-Heap Memory by Scheduler (MB)</td>
+</tr>
+<tr>
+<td>assignedMemOffHeap</td>
+<td>Double</td>
+<td>Assigned Off-Heap Memory by Scheduler (MB)</td>
+</tr>
+<tr>
+<td>assignedCpu</td>
+<td>Number</td>
+<td>Assigned CPU by Scheduler (%)</td>
+</tr>
+<tr>
+<td>componentNumTasks</td>
+<td>Dictionary</td>
+<td>Components -> # of executing tasks</td>
+</tr>
+<tr>
+<td>uptime</td>
+<td>String</td>
+<td>Shows how long the worker is running</td>
+</tr>
+<tr>
+<td>uptimeSeconds</td>
+<td>Integer</td>
+<td>Shows how long the worker is running in seconds</td>
+</tr>
+<tr>
+<td>workerLogLink</td>
+<td>String</td>
+<td>Link to worker log viewer page</td>
+</tr>
+</tbody></table>
+
+<p>Sample response:</p>
+<div class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
+ </span><span class="nt">"supervisors"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
+ </span><span class="nt">"totalMem"</span><span class="p">:</span><span class="w"> </span><span class="mf">4096.0</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"host"</span><span class="p">:</span><span class="s2">"192.168.10.237"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"id"</span><span class="p">:</span><span class="s2">"bdfe8eff-f1d8-4bce-81f5-9d3ae1bf432e-169.254.129.212"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"uptime"</span><span class="p">:</span><span class="s2">"7m 8s"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"totalCpu"</span><span class="p">:</span><span class="mf">400.0</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"usedCpu"</span><span class="p">:</span><span class="mf">495.0</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"usedMem"</span><span class="p">:</span><span class="mf">3432.0</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"slotsUsed"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"version"</span><span class="p">:</span><span class="s2">"0.10.1"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"slotsTotal"</span><span class="p">:</span><span class="mi">4</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"uptimeSeconds"</span><span class="p">:</span><span class="mi">428</span><span class="w">
+ </span><span class="p">}],</span><span class="w">
+ </span><span class="nt">"schedulerDisplayResource"</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"workers"</span><span class="p">:[{</span><span class="w">
+ </span><span class="nt">"topologyName"</span><span class="p">:</span><span class="s2">"ras"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"topologyId"</span><span class="p">:</span><span class="s2">"ras-4-1460229987"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"host"</span><span class="p">:</span><span class="s2">"192.168.10.237"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"supervisorId"</span><span class="p">:</span><span class="s2">"bdfe8eff-f1d8-4bce-81f5-9d3ae1bf432e-169.254.129.212"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"assignedMemOnHeap"</span><span class="p">:</span><span class="mf">704.0</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"uptime"</span><span class="p">:</span><span class="s2">"2m 47s"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"uptimeSeconds"</span><span class="p">:</span><span class="mi">167</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"port"</span><span class="p">:</span><span class="mi">6707</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"workerLogLink"</span><span class="p">:</span><span class="s2">"http:\/\/192.168.10.237:8000\/log?file=ras-4-1460229987%2F6707%2Fworker.log"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"componentNumTasks"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"word"</span><span class="p">:</span><span class="mi">5</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="nt">"executorsTotal"</span><span class="p">:</span><span class="mi">8</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"assignedCpu"</span><span class="p">:</span><span class="mf">130.0</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"assignedMemOffHeap"</span><span class="p">:</span><span class="mf">80.0</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"topologyName"</span><span class="p">:</span><span class="s2">"ras"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"topologyId"</span><span class="p">:</span><span class="s2">"ras-4-1460229987"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"host"</span><span class="p">:</span><span class="s2">"192.168.10.237"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"supervisorId"</span><span class="p">:</span><span class="s2">"bdfe8eff-f1d8-4bce-81f5-9d3ae1bf432e-169.254.129.212"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"assignedMemOnHeap"</span><span class="p">:</span><span class="mf">904.0</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"uptime"</span><span class="p">:</span><span class="s2">"2m 53s"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"port"</span><span class="p">:</span><span class="mi">6706</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"workerLogLink"</span><span class="p">:</span><span class="s2">"http:\/\/192.168.10.237:8000\/log?file=ras-4-1460229987%2F6706%2Fworker.log"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"componentNumTasks"</span><span class="p">:{</span><span class="w">
+ </span><span class="nt">"exclaim2"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"exclaim1"</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"word"</span><span class="p">:</span><span class="mi">5</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="nt">"executorsTotal"</span><span class="p">:</span><span class="mi">10</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"uptimeSeconds"</span><span class="p">:</span><span class="mi">173</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"assignedCpu"</span><span class="p">:</span><span class="mf">165.0</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"assignedMemOffHeap"</span><span class="p">:</span><span class="mf">80.0</span><span class="w">
+ </span><span class="p">}]</span><span class="w">
+</span><span class="p">}</span><span class="w">
+</span></code></pre></div>
<h3 id="api-v1-topology-summary-get">/api/v1/topology/summary (GET)</h3>
<p>Returns summary information for all topologies.</p>
@@ -550,6 +806,11 @@ are used by nimbus.</p>
<td>Double</td>
<td>Assigned CPU by Scheduler (%)</td>
</tr>
+<tr>
+<td>schedulerDisplayResource</td>
+<td>Boolean</td>
+<td>Whether to display scheduler resource information</td>
+</tr>
</tbody></table>
<p>Sample response:</p>
@@ -574,8 +835,8 @@ are used by nimbus.</p>
</span><span class="nt">"assignedTotalMem"</span><span class="p">:</span><span class="w"> </span><span class="mi">768</span><span class="p">,</span><span class="w">
</span><span class="nt">"assignedCpu"</span><span class="p">:</span><span class="w"> </span><span class="mi">80</span><span class="w">
</span><span class="p">}</span><span class="w">
- </span><span class="p">]</span><span class="w">
- </span><span class="s2">"schedulerDisplayResource"</span><span class="err">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"schedulerDisplayResource"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div>
<h3 id="api-v1-topology-workers-id-get">/api/v1/topology-workers/:id (GET)</h3>
@@ -1034,6 +1295,489 @@ are used by nimbus.</p>
</span><span class="nt">"replicationCount"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div>
+<h3 id="api-v1-topology-id-metrics">/api/v1/topology/:id/metrics</h3>
+
+<p>Returns detailed metrics for topology. It shows metrics per component, which are aggregated by stream.</p>
+
+<table><thead>
+<tr>
+<th>Parameter</th>
+<th>Value</th>
+<th>Description</th>
+</tr>
+</thead><tbody>
+<tr>
+<td>id</td>
+<td>String (required)</td>
+<td>Topology Id</td>
+</tr>
+<tr>
+<td>window</td>
+<td>String. Default value :all-time</td>
+<td>window duration for metrics in seconds</td>
+</tr>
+<tr>
+<td>sys</td>
+<td>String. Values 1 or 0. Default value 0</td>
+<td>Controls including sys stats part of the response</td>
+</tr>
+</tbody></table>
+
+<p>Response fields:</p>
+
+<table><thead>
+<tr>
+<th>Field</th>
+<th>Value</th>
+<th>Description</th>
+</tr>
+</thead><tbody>
+<tr>
+<td>window</td>
+<td>String. Default value ":all-time"</td>
+<td>window duration for metrics in seconds</td>
+</tr>
+<tr>
+<td></td>
+<td>windowHint</td>
+<td>String</td>
+</tr>
+<tr>
+<td>spouts</td>
+<td>Array</td>
+<td>Array of all the spout components in the topology</td>
+</tr>
+<tr>
+<td>spouts.id</td>
+<td>String</td>
+<td>Spout id</td>
+</tr>
+<tr>
+<td>spouts.emitted</td>
+<td>Array</td>
+<td>Array of all the output streams this spout emits messages</td>
+</tr>
+<tr>
+<td>spouts.emitted.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>spouts.emitted.value</td>
+<td>Long</td>
+<td>Number of messages emitted in given window</td>
+</tr>
+<tr>
+<td>spouts.transferred</td>
+<td>Array</td>
+<td>Array of all the output streams this spout transfers messages</td>
+</tr>
+<tr>
+<td>spouts.transferred.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>spouts.transferred.value</td>
+<td>Long</td>
+<td>Number messages transferred in given window</td>
+</tr>
+<tr>
+<td>spouts.acked</td>
+<td>Array</td>
+<td>Array of all the output streams this spout receives ack of messages</td>
+</tr>
+<tr>
+<td>spouts.acked.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>spouts.acked.value</td>
+<td>Long</td>
+<td>Number of messages acked in given window</td>
+</tr>
+<tr>
+<td>spouts.failed</td>
+<td>Array</td>
+<td>Array of all the output streams this spout receives fail of messages</td>
+</tr>
+<tr>
+<td>spouts.failed.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>spouts.failed.value</td>
+<td>Long</td>
+<td>Number of messages failed in given window</td>
+</tr>
+<tr>
+<td>spouts.complete_ms_avg</td>
+<td>Array</td>
+<td>Array of all the output streams this spout receives ack of messages</td>
+</tr>
+<tr>
+<td>spouts.complete_ms_avg.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>spouts.complete_ms_avg.value</td>
+<td>String (double value returned in String format)</td>
+<td>Total latency for processing the message</td>
+</tr>
+<tr>
+<td>bolts</td>
+<td>Array</td>
+<td>Array of all the bolt components in the topology</td>
+</tr>
+<tr>
+<td>bolts.id</td>
+<td>String</td>
+<td>Bolt id</td>
+</tr>
+<tr>
+<td>bolts.emitted</td>
+<td>Array</td>
+<td>Array of all the output streams this bolt emits messages</td>
+</tr>
+<tr>
+<td>bolts.emitted.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>bolts.emitted.value</td>
+<td>Long</td>
+<td>Number of messages emitted in given window</td>
+</tr>
+<tr>
+<td>bolts.transferred</td>
+<td>Array</td>
+<td>Array of all the output streams this bolt transfers messages</td>
+</tr>
+<tr>
+<td>bolts.transferred.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>bolts.transferred.value</td>
+<td>Long</td>
+<td>Number messages transferred in given window</td>
+</tr>
+<tr>
+<td>bolts.acked</td>
+<td>Array</td>
+<td>Array of all the input streams this bolt acknowledges of messages</td>
+</tr>
+<tr>
+<td>bolts.acked.component_id</td>
+<td>String</td>
+<td>Component id for this stream</td>
+</tr>
+<tr>
+<td>bolts.acked.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>bolts.acked.value</td>
+<td>Long</td>
+<td>Number of messages acked in given window</td>
+</tr>
+<tr>
+<td>bolts.failed</td>
+<td>Array</td>
+<td>Array of all the input streams this bolt receives fail of messages</td>
+</tr>
+<tr>
+<td>bolts.failed.component_id</td>
+<td>String</td>
+<td>Component id for this stream</td>
+</tr>
+<tr>
+<td>bolts.failed.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>bolts.failed.value</td>
+<td>Long</td>
+<td>Number of messages failed in given window</td>
+</tr>
+<tr>
+<td>bolts.process_ms_avg</td>
+<td>Array</td>
+<td>Array of all the input streams this spout acks messages</td>
+</tr>
+<tr>
+<td>bolts.process_ms_avg.component_id</td>
+<td>String</td>
+<td>Component id for this stream</td>
+</tr>
+<tr>
+<td>bolts.process_ms_avg.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>bolts.process_ms_avg.value</td>
+<td>String (double value returned in String format)</td>
+<td>Average time of the bolt to ack a message after it was received</td>
+</tr>
+<tr>
+<td>bolts.executed</td>
+<td>Array</td>
+<td>Array of all the input streams this bolt executes messages</td>
+</tr>
+<tr>
+<td>bolts.executed.component_id</td>
+<td>String</td>
+<td>Component id for this stream</td>
+</tr>
+<tr>
+<td>bolts.executed.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>bolts.executed.value</td>
+<td>Long</td>
+<td>Number of messages executed in given window</td>
+</tr>
+<tr>
+<td>bolts.executed_ms_avg</td>
+<td>Array</td>
+<td>Array of all the output streams this spout receives ack of messages</td>
+</tr>
+<tr>
+<td>bolts.executed_ms_avg.component_id</td>
+<td>String</td>
+<td>Component id for this stream</td>
+</tr>
+<tr>
+<td>bolts.executed_ms_avg.stream_id</td>
+<td>String</td>
+<td>Stream id for this stream</td>
+</tr>
+<tr>
+<td>bolts.executed_ms_avg.value</td>
+<td>String (double value returned in String format)</td>
+<td>Average time to run the execute method of the bolt</td>
+</tr>
+</tbody></table>
+
+<p>Examples:</p>
+<div class="highlight"><pre><code class="language-no-highlight" data-lang="no-highlight">1. http://ui-daemon-host-name:8080/api/v1/topology/WordCount3-1-1402960825/metrics
+1. http://ui-daemon-host-name:8080/api/v1/topology/WordCount3-1-1402960825/metrics?sys=1
+2. http://ui-daemon-host-name:8080/api/v1/topology/WordCount3-1-1402960825/metrics?window=600
+</code></pre></div>
+<p>Sample response:</p>
+<div class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
+ </span><span class="nt">"window"</span><span class="p">:</span><span class="s2">":all-time"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"window-hint"</span><span class="p">:</span><span class="s2">"All time"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"spouts"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"id"</span><span class="p">:</span><span class="s2">"spout"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"emitted"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__metrics"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">20</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">17350280</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__ack_init"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">17328160</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__system"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">20</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"transferred"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__metrics"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">20</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">17350280</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__ack_init"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">17328160</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__system"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">0</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"acked"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">17339180</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"failed"</span><span class="p">:[</span><span class="w">
+
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"complete_ms_avg"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="s2">"920.497"</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">]</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"bolts"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"id"</span><span class="p">:</span><span class="s2">"count"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"emitted"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__metrics"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">120</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">190748180</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__ack_ack"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">190718100</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__system"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">20</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"transferred"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__metrics"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">120</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">0</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__ack_ack"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">190718100</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__system"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">0</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"acked"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"component_id"</span><span class="p">:</span><span class="s2">"split"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">190733160</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"failed"</span><span class="p">:[</span><span class="w">
+
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"process_ms_avg"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"component_id"</span><span class="p">:</span><span class="s2">"split"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="s2">"0.004"</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"executed"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"component_id"</span><span class="p">:</span><span class="s2">"split"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">190733140</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"executed_ms_avg"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"component_id"</span><span class="p">:</span><span class="s2">"split"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="s2">"0.005"</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">]</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"id"</span><span class="p">:</span><span class="s2">"split"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"emitted"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__metrics"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">60</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">190754740</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__ack_ack"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">17317580</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__system"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">20</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"transferred"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__metrics"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">60</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">190754740</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__ack_ack"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">17317580</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"__system"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">0</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"acked"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"component_id"</span><span class="p">:</span><span class="s2">"spout"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">17339180</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"failed"</span><span class="p">:[</span><span class="w">
+
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"process_ms_avg"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"component_id"</span><span class="p">:</span><span class="s2">"spout"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="s2">"0.051"</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"executed"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"component_id"</span><span class="p">:</span><span class="s2">"spout"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="mi">17339240</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">],</span><span class="w">
+ </span><span class="nt">"executed_ms_avg"</span><span class="p">:[</span><span class="w">
+ </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"component_id"</span><span class="p">:</span><span class="s2">"spout"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"stream_id"</span><span class="p">:</span><span class="s2">"default"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"value"</span><span class="p">:</span><span class="s2">"0.052"</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">]</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+ </span><span class="p">]</span><span class="w">
+</span><span class="p">}</span><span class="w">
+</span></code></pre></div>
<h3 id="api-v1-topology-id-component-component-get">/api/v1/topology/:id/component/:component (GET)</h3>
<p>Returns detailed metrics and executor information</p>
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/Windowing.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/Windowing.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/Windowing.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/Windowing.html Tue Aug 23 01:13:59 2016
@@ -273,10 +273,6 @@ are performed based on the processing ti
<p>The value for the above <code>fieldName</code> will be looked up from the incoming tuple and considered for windowing calculations.
If the field is not present in the tuple an exception will be thrown. Along with the timestamp field name, a time lag parameter
can also be specified which indicates the max time limit for tuples with out of order timestamps. </p>
-
-<p>E.g. If the lag is 5 secs and a tuple <code>t1</code> arrived with timestamp <code>06:00:05</code> no tuples may arrive with tuple timestamp earlier than <code>06:00:00</code>. If a tuple
-arrives with timestamp 05:59:59 after <code>t1</code> and the window has moved past <code>t1</code>, it will be treated as a late tuple and not processed. Currently the late
- tuples are just logged in the worker log files at INFO level.</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="cm">/**
* Specify the maximum time lag of the tuple timestamp in milliseconds. It means that the tuple timestamps
* cannot be out of order by more than this amount.
@@ -285,6 +281,23 @@ arrives with timestamp 05:59:59 after <c
*/</span>
<span class="kd">public</span> <span class="n">BaseWindowedBolt</span> <span class="nf">withLag</span><span class="p">(</span><span class="n">Duration</span> <span class="n">duration</span><span class="o">)</span>
</code></pre></div>
+<p>E.g. If the lag is 5 secs and a tuple <code>t1</code> arrived with timestamp <code>06:00:05</code> no tuples may arrive with tuple timestamp earlier than <code>06:00:00</code>. If a tuple
+arrives with timestamp 05:59:59 after <code>t1</code> and the window has moved past <code>t1</code>, it will be treated as a late tuple. Late tuples are not processed by default,
+just logged in the worker log files at INFO level.</p>
+<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="cm">/**
+ * Specify a stream id on which late tuples are going to be emitted. They are going to be accessible via the
+ * {@link org.apache.storm.topology.WindowedBoltExecutor#LATE_TUPLE_FIELD} field.
+ * It must be defined on a per-component basis, and in conjunction with the
+ * {@link BaseWindowedBolt#withTimestampField}, otherwise {@link IllegalArgumentException} will be thrown.
+ *
+ * @param streamId the name of the stream used to emit late tuples on
+ */</span>
+<span class="kd">public</span> <span class="n">BaseWindowedBolt</span> <span class="nf">withLateTupleStream</span><span class="p">(</span><span class="n">String</span> <span class="n">streamId</span><span class="o">)</span>
+
+</code></pre></div>
+<p>This behaviour can be changed by specifying the above <code>streamId</code>. In this case late tuples are going to be emitted on the specified stream and accessible
+via the field <code>WindowedBoltExecutor.LATE_TUPLE_FIELD</code>.</p>
+
<h3 id="watermarks">Watermarks</h3>
<p>For processing tuples with timestamp field, storm internally computes watermarks based on the incoming tuple timestamp. Watermark is
Modified: storm/site/publish/releases/2.0.0-SNAPSHOT/flux.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/2.0.0-SNAPSHOT/flux.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/2.0.0-SNAPSHOT/flux.html (original)
+++ storm/site/publish/releases/2.0.0-SNAPSHOT/flux.html Tue Aug 23 01:13:59 2016
@@ -253,6 +253,15 @@ recommended).</p>
</dependency>
</code></p>
+<p>Using shell spouts and bolts requires additional Flux Wrappers library:
+<code>xml
+<dependency>
+ <groupId>org.apache.storm</groupId>
+ <artifactId>flux-wrappers</artifactId>
+ <version>${storm.version}</version>
+</dependency>
+</code></p>
+
<h4 id="creating-a-flux-enabled-topology-jar">Creating a Flux-Enabled Topology JAR</h4>
<p>The example below illustrates Flux usage with the Maven shade plugin:</p>
@@ -264,6 +273,12 @@ recommended).</p>
<span class="nt"><artifactId></span>flux-core<span class="nt"></artifactId></span>
<span class="nt"><version></span>${storm.version}<span class="nt"></version></span>
<span class="nt"></dependency></span>
+ <span class="c"><!-- Flux Wrappers include --></span>
+ <span class="nt"><dependency></span>
+ <span class="nt"><groupId></span>org.apache.storm<span class="nt"></groupId></span>
+ <span class="nt"><artifactId></span>flux-wrappers<span class="nt"></artifactId></span>
+ <span class="nt"><version></span>${storm.version}<span class="nt"></version></span>
+ <span class="nt"></dependency></span>
<span class="c"><!-- add user dependencies here... --></span>
@@ -359,9 +374,9 @@ Parsing file: /Users/hsimpson/Projects/d
---------- TOPOLOGY DETAILS ----------
Name: shell-topology
--------------- SPOUTS ---------------
-sentence-spout[1](org.apache.storm.flux.spouts.GenericShellSpout)
+sentence-spout[1](org.apache.storm.flux.wrappers.spouts.FluxShellSpout)
---------------- BOLTS ---------------
-splitsentence[1](org.apache.storm.flux.bolts.GenericShellBolt)
+splitsentence[1](org.apache.storm.flux.wrappers.bolts.FluxShellBolt)
log[1](org.apache.storm.flux.wrappers.bolts.LogInfoBolt)
count[1](org.apache.storm.testing.TestWordCounter)
--------------- STREAMS ---------------
@@ -637,7 +652,7 @@ well.</p>
<p>Shell spout example:</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">spouts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">sentence-spout"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.spouts.GenericShellSpout"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.spouts.FluxShellSpout"</span>
<span class="c1"># shell spout constructor takes 2 arguments: String[], String[]</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
@@ -704,7 +719,7 @@ well.</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="c1"># bolt definitions</span>
<span class="s">bolts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">splitsentence"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.bolts.GenericShellBolt"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.bolts.FluxShellBolt"</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
<span class="pi">-</span> <span class="pi">[</span><span class="s2">"</span><span class="s">python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">splitsentence.py"</span><span class="pi">]</span>
@@ -826,7 +841,7 @@ same file. Includes may be either files,
<span class="c1"># spout definitions</span>
<span class="s">spouts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">sentence-spout"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.spouts.GenericShellSpout"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.spouts.FluxShellSpout"</span>
<span class="c1"># shell spout constructor takes 2 arguments: String[], String[]</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
@@ -838,7 +853,7 @@ same file. Includes may be either files,
<span class="c1"># bolt definitions</span>
<span class="s">bolts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">id</span><span class="pi">:</span> <span class="s2">"</span><span class="s">splitsentence"</span>
- <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.bolts.GenericShellBolt"</span>
+ <span class="s">className</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.flux.wrappers.bolts.FluxShellBolt"</span>
<span class="s">constructorArgs</span><span class="pi">:</span>
<span class="c1"># command line</span>
<span class="pi">-</span> <span class="pi">[</span><span class="s2">"</span><span class="s">python"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">splitsentence.py"</span><span class="pi">]</span>
Modified: storm/site/publish/releases/current/SECURITY.html
URL: http://svn.apache.org/viewvc/storm/site/publish/releases/current/SECURITY.html?rev=1757294&r1=1757293&r2=1757294&view=diff
==============================================================================
--- storm/site/publish/releases/current/SECURITY.html (original)
+++ storm/site/publish/releases/current/SECURITY.html Tue Aug 23 01:13:59 2016
@@ -266,9 +266,11 @@ curl -i --negotiate -u:anyUser -b ~/co
<ol>
<li>Firefox: Goto about:config and search for network.negotiate-auth.trusted-uris double-click to add value "<a href="http://storm-ui-hostname:8080">http://storm-ui-hostname:8080</a>"</li>
<li>Google-chrome: start from command line with: google-chrome --auth-server-whitelist="<em>storm-ui-hostname" --auth-negotiate-delegate-whitelist="</em>storm-ui-hostname"<br></li>
-<li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website </li>
+<li>IE: Configure trusted websites to include "storm-ui-hostname" and allow negotiation for that website</li>
</ol>
+<p><strong>Note</strong>: For viewing any logs via <code>logviewer</code> in secure mode, all the hosts that runs <code>logviewer</code> should also be added to the above white list. For big clusters you could white list the host's domain (for e.g. set <code>network.negotiate-auth.trusted-uris</code> to <code>.yourdomain.com</code>).</p>
+
<p><strong>Caution</strong>: In AD MIT Keberos setup the key size is bigger than the default UI jetty server request header size. Make sure you set ui.header.buffer.bytes to 65536 in storm.yaml. More details are on <a href="https://issues.apache.org/jira/browse/STORM-633">STORM-633</a></p>
<h2 id="ui-drpc-ssl">UI / DRPC SSL</h2>