You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by bo...@apache.org on 2015/04/22 23:48:20 UTC
svn commit: r1675506 - in /storm/site/publish:
2013/01/11/storm082-released.html documentation/Multilang-protocol.html
documentation/Trident-API-Overview.html feed.xml
Author: bobby
Date: Wed Apr 22 21:48:19 2015
New Revision: 1675506
URL: http://svn.apache.org/r1675506
Log:
Updated for latest code commits
Modified:
storm/site/publish/2013/01/11/storm082-released.html
storm/site/publish/documentation/Multilang-protocol.html
storm/site/publish/documentation/Trident-API-Overview.html
storm/site/publish/feed.xml
Modified: storm/site/publish/2013/01/11/storm082-released.html
URL: http://svn.apache.org/viewvc/storm/site/publish/2013/01/11/storm082-released.html?rev=1675506&r1=1675505&r2=1675506&view=diff
==============================================================================
--- storm/site/publish/2013/01/11/storm082-released.html (original)
+++ storm/site/publish/2013/01/11/storm082-released.html Wed Apr 22 21:48:19 2015
@@ -87,7 +87,7 @@
<p>You configure the isolation scheduler in the Nimbus configuration. Set "storm.scheduler" to "backtype.storm.scheduler.IsolationScheduler". Then, use the "isolation.scheduler.machines" config to specify how many machines each topology should get. This config is a map from topology name to number of machines. For example:</p>
-<script src="https://gist.github.com/4514691.js"></script>
+<p><script src="https://gist.github.com/4514691.js"></script></p>
<p>Any topologies submitted to the cluster not listed there will not be isolated. Note that there is no way for a user of Storm to affect their isolation settings â this is only allowed by the administrator of the cluster (this is very much intentional).</p>
Modified: storm/site/publish/documentation/Multilang-protocol.html
URL: http://svn.apache.org/viewvc/storm/site/publish/documentation/Multilang-protocol.html?rev=1675506&r1=1675505&r2=1675506&view=diff
==============================================================================
--- storm/site/publish/documentation/Multilang-protocol.html (original)
+++ storm/site/publish/documentation/Multilang-protocol.html Wed Apr 22 21:48:19 2015
@@ -129,28 +129,54 @@ STDIN and STDOUT.</li>
<p>The initial handshake is the same for both types of shell components:</p>
<ul>
-<li>STDIN: Setup info. This is a JSON object with the Storm configuration, Topology context, and a PID directory, like this:</li>
+<li>STDIN: Setup info. This is a JSON object with the Storm configuration, a PID directory, and a topology context, like this:</li>
</ul>
<div class="highlight"><pre><code class="language-text" data-lang="text">{
"conf": {
"topology.message.timeout.secs": 3,
// etc
},
+ "pidDir": "...",
"context": {
"task->component": {
"1": "example-spout",
"2": "__acker",
- "3": "example-bolt"
+ "3": "example-bolt1",
+ "4": "example-bolt2"
},
- "taskid": 3
- },
- "pidDir": "..."
+ "taskid": 3,
+ // Everything below this line is only available in Storm 0.11.0+
+ "componentid": "example-bolt"
+ "stream->target->grouping": {
+ "default": {
+ "example-bolt2": {
+ "type": "SHUFFLE"}}},
+ "streams": ["default"],
+ "stream->outputfields": {"default": ["word"]},
+ "source->stream->grouping": {
+ "example-spout": {
+ "default": {
+ "type": "FIELDS",
+ "fields": ["word"]
+ }
+ }
+ }
+ }
}
</code></pre></div>
<p>Your script should create an empty file named with its PID in this directory. e.g.
the PID is 1234, so an empty file named 1234 is created in the directory. This
file lets the supervisor know the PID so it can shutdown the process later on.</p>
+<p>As of Storm 0.11.0, the context sent by Storm to shell components has been
+enhanced substantially to include all aspects of the topology context available
+to JVM components. One key addition is the ability to determine a shell
+component's source and targets (i.e., inputs and outputs) in the topology via
+the <code>stream->target->grouping</code> and <code>source->stream->grouping</code> dictionaries. At
+the innermost level of these nested dictionaries, groupings are represented as
+a dictionary that minimally has a <code>type</code> key, but can also have a <code>fields</code> key
+to specify which fields are involved in a <code>FIELDS</code> grouping.</p>
+
<ul>
<li>STDOUT: Your PID, in a JSON object, like <code>{"pid": 1234}</code>. The shell component will log the PID to its log.</li>
</ul>
@@ -280,34 +306,35 @@ emits, however.</p>
shell bolt to 'sync'.</li>
</ul>
-<h3 id="handling-heartbeat-(0.9.3-and-later)">Handling Heartbeat (0.9.3 and later)</h3>
+<h3 id="handling-heartbeats-(0.9.3-and-later)">Handling Heartbeats (0.9.3 and later)</h3>
-<p>ShellSpout/ShellBolt has been introduced from <a href="https://issues.apache.org/jira/browse/STORM-513">STORM-513</a> to prevent hanging/zombie subprocess.</p>
+<p>As of Storm 0.9.3, heartbeats have been between ShellSpout/ShellBolt and their
+multi-lang subprocesses to detect hanging/zombie subprocesses. Any libraries
+for interfacing with Storm via multi-lang must take the following actions
+regarding hearbeats:</p>
-<ul>
-<li>Spout</li>
-</ul>
+<h4 id="spout">Spout</h4>
-<p>Shell spouts are synchronous, and subprocess always send 'sync' at the end of next() so you don't need to take care of.
-One thing you have to take care of is, don't let subprocess sleep too much from next(), especially keep it less to worker timeout.</p>
+<p>Shell spouts are synchronous, so subprocesses always send <code>sync</code> commands at the
+end of <code>next()</code>, so you should not have to do much to support heartbeats for
+spouts. That said, you must not let subprocesses sleep more than the worker
+timeout during <code>next()</code>.</p>
-<ul>
-<li>Bolt</li>
-</ul>
+<h4 id="bolt">Bolt</h4>
-<p>Shell bolts are asynchronous, so ShellBolt will send heartbeat tuple periodically.
-Heartbeat tuple looks like:</p>
+<p>Shell bolts are asynchronous, so a ShellBolt will send heartbeat tuples to its
+subprocess periodically. Heartbeat tuple looks like:</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">{
"id": "-6955786537413359385",
"comp": "1",
- // heartbeat tuple
"stream": "__heartbeat",
- // it's system task id
+ // this shell bolt's system task id
"task": -1,
"tuple": []
}
</code></pre></div>
-<p>When subprocess receives heartbeat tuple, it should send 'sync' to ShellBolt.</p>
+<p>When subprocess receives heartbeat tuple, it must send a <code>sync</code> command back to
+ShellBolt.</p>
</article>
Modified: storm/site/publish/documentation/Trident-API-Overview.html
URL: http://svn.apache.org/viewvc/storm/site/publish/documentation/Trident-API-Overview.html?rev=1675506&r1=1675505&r2=1675506&view=diff
==============================================================================
--- storm/site/publish/documentation/Trident-API-Overview.html (original)
+++ storm/site/publish/documentation/Trident-API-Overview.html Wed Apr 22 21:48:19 2015
@@ -193,7 +193,7 @@ Partition 2:
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div>
-<p>The benefits of CombinerAggregators are seen when you use the with the aggregate method instead of partitionAggregate. In that case, Trident automatically optimizes the computation by doing partial aggregations before transferring tuples over the network.</p>
+<p>The benefits of CombinerAggregators are seen when you use them with the aggregate method instead of partitionAggregate. In that case, Trident automatically optimizes the computation by doing partial aggregations before transferring tuples over the network.</p>
<p>A ReducerAggregator has the following interface:</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">ReducerAggregator</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="kd">extends</span> <span class="n">Serializable</span> <span class="o">{</span>
Modified: storm/site/publish/feed.xml
URL: http://svn.apache.org/viewvc/storm/site/publish/feed.xml?rev=1675506&r1=1675505&r2=1675506&view=diff
==============================================================================
--- storm/site/publish/feed.xml (original)
+++ storm/site/publish/feed.xml Wed Apr 22 21:48:19 2015
@@ -5,9 +5,9 @@
<description></description>
<link>http://storm.apache.org/</link>
<atom:link href="http://storm.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Thu, 26 Mar 2015 15:39:35 -0400</pubDate>
- <lastBuildDate>Thu, 26 Mar 2015 15:39:35 -0400</lastBuildDate>
- <generator>Jekyll v2.5.3</generator>
+ <pubDate>Wed, 22 Apr 2015 16:44:54 -0500</pubDate>
+ <lastBuildDate>Wed, 22 Apr 2015 16:44:54 -0500</lastBuildDate>
+ <generator>Jekyll v2.5.2</generator>
<item>
<title>Storm 0.9.4 released</title>
@@ -29,7 +29,7 @@
<li>STORM-130: Supervisor getting killed due to java.io.FileNotFoundException: File &#39;../stormconf.ser&#39; does not exist.</li>
</ul>
</description>
- <pubDate>Wed, 25 Mar 2015 00:00:00 -0400</pubDate>
+ <pubDate>Wed, 25 Mar 2015 00:00:00 -0500</pubDate>
<link>http://storm.apache.org/2015/03/25/storm094-released.html</link>
<guid isPermaLink="true">http://storm.apache.org/2015/03/25/storm094-released.html</guid>
@@ -225,7 +225,7 @@
<li>STORM-514: Update storm-starter README now that Storm has graduated from Incubator</li>
</ul>
</description>
- <pubDate>Tue, 25 Nov 2014 00:00:00 -0500</pubDate>
+ <pubDate>Tue, 25 Nov 2014 00:00:00 -0600</pubDate>
<link>http://storm.apache.org/2014/11/25/storm093-released.html</link>
<guid isPermaLink="true">http://storm.apache.org/2014/11/25/storm093-released.html</guid>
@@ -240,7 +240,7 @@
<p>We heartily encourage you to <a href="http://storm.apache.org/downloads.html">test the 0.9.3 release candidate</a> and provide your feedback regarding any issues via <a href="http://storm.apache.org/community.html">our mailing lists</a>, which is an easy and valuable way to contribute back to the Storm community and to help us moving to an official 0.9.3 release. You can find the <a href="http://storm.apache.org/downloads.html">0.9.3 release candidate in our Downloads section</a>.</p>
</description>
- <pubDate>Mon, 20 Oct 2014 00:00:00 -0400</pubDate>
+ <pubDate>Mon, 20 Oct 2014 00:00:00 -0500</pubDate>
<link>http://storm.apache.org/2014/10/20/storm093-release-candidate.html</link>
<guid isPermaLink="true">http://storm.apache.org/2014/10/20/storm093-release-candidate.html</guid>
@@ -375,7 +375,7 @@ version: 0.9.2-incubating
<li>STORM-146: Unit test regression when storm is compiled with 3.4.5 zookeeper</li>
</ul>
</description>
- <pubDate>Wed, 25 Jun 2014 00:00:00 -0400</pubDate>
+ <pubDate>Wed, 25 Jun 2014 00:00:00 -0500</pubDate>
<link>http://storm.apache.org/2014/06/25/storm092-released.html</link>
<guid isPermaLink="true">http://storm.apache.org/2014/06/25/storm092-released.html</guid>
@@ -422,7 +422,7 @@ version: 0.9.2-incubating
</tr>
</tbody></table>
</description>
- <pubDate>Tue, 17 Jun 2014 00:00:00 -0400</pubDate>
+ <pubDate>Tue, 17 Jun 2014 00:00:00 -0500</pubDate>
<link>http://storm.apache.org/2014/06/17/contest-results.html</link>
<guid isPermaLink="true">http://storm.apache.org/2014/06/17/contest-results.html</guid>
@@ -517,7 +517,7 @@ version: 0.9.2-incubating
</tr>
</tbody></table>
</description>
- <pubDate>Tue, 27 May 2014 00:00:00 -0400</pubDate>
+ <pubDate>Tue, 27 May 2014 00:00:00 -0500</pubDate>
<link>http://storm.apache.org/2014/05/27/round1-results.html</link>
<guid isPermaLink="true">http://storm.apache.org/2014/05/27/round1-results.html</guid>
@@ -528,7 +528,7 @@ version: 0.9.2-incubating
<title>Logo Entry No. 11 - Jennifer Lee</title>
<description><p><img src="/images/logocontest/jlee2/storm_logo.jpg" alt="Storm Brand"></p>
</description>
- <pubDate>Tue, 29 Apr 2014 00:00:00 -0400</pubDate>
+ <pubDate>Tue, 29 Apr 2014 00:00:00 -0500</pubDate>
<link>http://storm.apache.org/2014/04/29/logo-jlee3.html</link>
<guid isPermaLink="true">http://storm.apache.org/2014/04/29/logo-jlee3.html</guid>
@@ -539,7 +539,7 @@ version: 0.9.2-incubating
<title>Logo Entry No. 10 - Jennifer Lee</title>
<description><p><img src="/images/logocontest/jlee3/storm_logo.jpg" alt="Storm Brand"></p>
</description>
- <pubDate>Tue, 29 Apr 2014 00:00:00 -0400</pubDate>
+ <pubDate>Tue, 29 Apr 2014 00:00:00 -0500</pubDate>
<link>http://storm.apache.org/2014/04/29/logo-jlee2.html</link>
<guid isPermaLink="true">http://storm.apache.org/2014/04/29/logo-jlee2.html</guid>
@@ -550,7 +550,7 @@ version: 0.9.2-incubating
<title>Logo Entry No. 9 - Jennifer Lee</title>
<description><p><img src="/images/logocontest/jlee1/storm_logo.jpg" alt="Storm Brand"></p>
</description>
- <pubDate>Tue, 29 Apr 2014 00:00:00 -0400</pubDate>
+ <pubDate>Tue, 29 Apr 2014 00:00:00 -0500</pubDate>
<link>http://storm.apache.org/2014/04/29/logo-jlee1.html</link>
<guid isPermaLink="true">http://storm.apache.org/2014/04/29/logo-jlee1.html</guid>
@@ -561,7 +561,7 @@ version: 0.9.2-incubating
<title>Logo Entry No. 8 - Stefano Asili</title>
<description><p><img src="/images/logocontest/sasili/storm_logo.png" alt="Storm Brand"></p>
</description>
- <pubDate>Sun, 27 Apr 2014 00:00:00 -0400</pubDate>
+ <pubDate>Sun, 27 Apr 2014 00:00:00 -0500</pubDate>
<link>http://storm.apache.org/2014/04/27/logo-sasili.html</link>
<guid isPermaLink="true">http://storm.apache.org/2014/04/27/logo-sasili.html</guid>