You are viewing a plain text version of this content. The canonical link for it is here.
Posted to s4-commits@incubator.apache.org by mm...@apache.org on 2013/10/06 11:56:43 UTC
svn commit: r1529612 - in /incubator/s4/site/doc/0.6.0:
application_dependencies/ configuration/ dev_tips/ event_dispatch/
fault_tolerance/ metrics/ tools/ twitter_trending_example/ walkthrough/
Author: mmorel
Date: Sun Oct 6 11:56:43 2013
New Revision: 1529612
URL: http://svn.apache.org/r1529612
Log:
updates for new twitter auth scheme + formatting
Modified:
incubator/s4/site/doc/0.6.0/application_dependencies/index.html
incubator/s4/site/doc/0.6.0/configuration/index.html
incubator/s4/site/doc/0.6.0/dev_tips/index.html
incubator/s4/site/doc/0.6.0/event_dispatch/index.html
incubator/s4/site/doc/0.6.0/fault_tolerance/index.html
incubator/s4/site/doc/0.6.0/metrics/index.html
incubator/s4/site/doc/0.6.0/tools/index.html
incubator/s4/site/doc/0.6.0/twitter_trending_example/index.html
incubator/s4/site/doc/0.6.0/walkthrough/index.html
Modified: incubator/s4/site/doc/0.6.0/application_dependencies/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/application_dependencies/index.html?rev=1529612&r1=1529611&r2=1529612&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/application_dependencies/index.html (original)
+++ incubator/s4/site/doc/0.6.0/application_dependencies/index.html Sun Oct 6 11:56:43 2013
@@ -77,25 +77,25 @@
<p>Add maven artifacts definitions to the gradle build file. For instance, add twitter4j_core and twitter4j_stream:</p>
<pre><code> project.ext["libraries"] = [
- Â Â Â Â Â Â twitter4j_core: Â Â 'org.twitter4j:twitter4j-core:2.2.5',
- Â Â Â Â Â Â twitter4j_stream: Â 'org.twitter4j:twitter4j-stream:2.2.5',
- Â Â Â Â Â Â s4_base:Â Â Â Â Â Â 'org.apache.s4:s4-base:0.6.0',
- Â Â Â Â Â Â s4_comm:Â Â Â Â Â Â 'org.apache.s4:s4-comm:0.6.0',
- Â Â Â Â Â Â s4_core:Â Â Â Â Â Â 'org.apache.s4:s4-core:0.6.0'
- Â Â Â Â ]
- </code></pre>
+ Â Â Â Â Â Â twitter4j_core: Â Â 'org.twitter4j:twitter4j-core:2.2.5',
+ Â Â Â Â Â Â twitter4j_stream: Â 'org.twitter4j:twitter4j-stream:2.2.5',
+ Â Â Â Â Â Â s4_base:Â Â Â Â Â Â 'org.apache.s4:s4-base:0.6.0',
+ Â Â Â Â Â Â s4_comm:Â Â Â Â Â Â 'org.apache.s4:s4-comm:0.6.0',
+ Â Â Â Â Â Â s4_core:Â Â Â Â Â Â 'org.apache.s4:s4-core:0.6.0'
+ Â Â Â Â ]
+</code></pre>
</li>
<li>
<p>Add these dependencies as compile-time dependencies. For instance:</p>
<pre><code> dependencies {
- compile (libraries.s4_base)
- compile (libraries.s4_comm)
- compile (libraries.s4_core)
- compile (libraries.twitter4j_core)
- compile (libraries.twitter4j_stream)
- }
- </code></pre>
+ compile (libraries.s4_base)
+ compile (libraries.s4_comm)
+ compile (libraries.s4_core)
+ compile (libraries.twitter4j_core)
+ compile (libraries.twitter4j_stream)
+ }
+</code></pre>
</li>
<li>
<p>If you use an IDE such as eclipse, you may update your projectâs classpath with: <code>./gradlew eclipse</code></p>
Modified: incubator/s4/site/doc/0.6.0/configuration/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/configuration/index.html?rev=1529612&r1=1529611&r2=1529612&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/configuration/index.html (original)
+++ incubator/s4/site/doc/0.6.0/configuration/index.html Sun Oct 6 11:56:43 2013
@@ -111,12 +111,12 @@
<p>The cluster configuration is maintained in Zookeeper, and can be set using S4 tools:</p>
<pre><code>./s4 newCluster -c=cluster1 -nbTasks=2 -flp=12000
- </code></pre>
+</code></pre>
<p>See tool documentation by typing:</p>
<pre><code>./s4 newCluster -help
- </code></pre>
+</code></pre>
<h1 id="node-configuration">Node configuration</h1>
@@ -191,16 +191,16 @@
<p>For instance, in order to enable file system based checkpointing, pass the corresponding checkpointing module class :</p>
<pre><code>./s4 deploy -s4r=uri/to/app.s4r -c=cluster1 -appName=myApp \
- -emc=org.apache.s4.core.ft.FileSystemBackendCheckpointingModule
- </code></pre>
+-emc=org.apache.s4.core.ft.FileSystemBackendCheckpointingModule
+</code></pre>
<p>You can also write your own custom modules. In that case, just package them into a jar file, and specify how to fetch that file when deploying the application, with the <code>mu</code> or <code>modulesURIs</code> option.</p>
<p>For instance, if you checkpoint through a specific key value store, you can write you own checkpointing implementation and module, package that into fancyKeyValueStoreCheckpointingModule.jar , and then:</p>
<pre><code>./s4 deploy -c=cluster1 -emc=my.project.FancyKeyValueStoreBackendCheckpointingModule \
- -mu=uri/to/fancyKeyValueStoreCheckpointingModule.jar
- </code></pre>
+-mu=uri/to/fancyKeyValueStoreCheckpointingModule.jar
+</code></pre>
<h3 id="overriding-parameters">overriding parameters</h3>
@@ -211,8 +211,8 @@
</ul>
<pre><code class="language-java"><span class="nd">@Inject</span>
- <span class="nd">@Named</span><span class="o">(</span><span class="s">"thePortNumber"</span><span class="o">)</span>
- <span class="kt">int</span> <span class="n">port</span></code></pre>
+<span class="nd">@Named</span><span class="o">(</span><span class="s">"thePortNumber"</span><span class="o">)</span>
+<span class="kt">int</span> <span class="n">port</span></code></pre>
<ul>
<li>specifying the parameter value when deploying the application (using <code>-p</code> inline with the node command, or with the â<code>@</code>â syntax)</li>
@@ -223,9 +223,9 @@
<p>Both application and platform parameters can be overriden. For instance, specifying a custom storage path for the file system based checkpointing mechanism would be passing the <code>s4.checkpointing.filesystem.storageRootPath</code> parameter:</p>
<pre><code>./s4 deploy -s4r=uri/to/app.s4r -c=cluster1 -appName=myApp \
- -emc=org.apache.s4.core.ft.FileSystemBackendCheckpointingModule \
- -p=s4.checkpointing.filesystem.storageRootPath=/custom/path
- </code></pre>
+-emc=org.apache.s4.core.ft.FileSystemBackendCheckpointingModule \
+-p=s4.checkpointing.filesystem.storageRootPath=/custom/path
+</code></pre>
<blockquote>
<p>Make sure you pass overriding <strong>platform</strong> parameters through the <code>-p</code> option in the <code>node</code> command, and pass <strong>application</strong> parameters through the <code>-p</code> option in the <code>deploy</code> command.</p>
@@ -238,11 +238,11 @@
With contents of the referenced file like:</p>
<pre><code>-s4r=uri/to/app.s4r
- -c=cluster1
- -appName=myApp
- -emc=org.apache.s4.core.ft.FileSystemBackendCheckpointingModule
- -p=param1=value1,param2=value2
- </code></pre>
+-c=cluster1
+-appName=myApp
+-emc=org.apache.s4.core.ft.FileSystemBackendCheckpointingModule
+-p=param1=value1,param2=value2
+</code></pre>
<h2 id="logging">Logging</h2>
Modified: incubator/s4/site/doc/0.6.0/dev_tips/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/dev_tips/index.html?rev=1529612&r1=1529611&r2=1529612&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/dev_tips/index.html (original)
+++ incubator/s4/site/doc/0.6.0/dev_tips/index.html Sun Oct 6 11:56:43 2013
@@ -78,8 +78,8 @@
<p>Install S4 and its dependencies in your local maven repository</p>
<pre><code> // from s4 source distribution root directory
- ./gradlew install -DskipTests
- </code></pre>
+ ./gradlew install -DskipTests
+</code></pre>
</li>
<li>Then run <code>gradlew eclipse</code> or <code>gradlew idea</code>
</li>
@@ -94,13 +94,13 @@
<p>Use the default test configuration (2 clusters with following configs: <code>-c=testCluster1:flp=12000:nbTasks=1</code> and <code>-c=testCluster2:flp=13000:nbTasks=1</code>)</p>
<pre><code> s4 zkServer -t
- </code></pre>
+</code></pre>
</li>
<li>
<p>Start a Zookeeper instance with your custom configuration, e.g. with 1 partition:</p>
<pre><code> s4 zkServer -clusters=c=testCluster1:flp=12000:nbTasks=1
- </code></pre>
+</code></pre>
</li>
</ul>
Modified: incubator/s4/site/doc/0.6.0/event_dispatch/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/event_dispatch/index.html?rev=1529612&r1=1529611&r2=1529612&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/event_dispatch/index.html (original)
+++ incubator/s4/site/doc/0.6.0/event_dispatch/index.html Sun Oct 6 11:56:43 2013
@@ -82,11 +82,11 @@
<p>A stream can be defined with a KeyFinder, as :</p>
<pre><code class="language-java"><span class="n">Stream</span><span class="o"><</span><span class="n">TopicEvent</span><span class="o">></span> <span class="n">topicSeenStream</span> <span class="o">=</span> <span class="n">createStream</span><span class="o">(</span><span class="s">"TopicSeen"</span><span class="o">,</span> <span class="k">new</span> <span class="n">KeyFinder</span><span class="o"><</span><span class="n">TopicEvent</span><span class="o">>()</span> <span class="o">{</span>
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="kd">final</span> <span class="n">TopicEvent</span> <span class="n">arg0</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">return</span> <span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">arg0</span><span class="o">.</span><span class="na">getTopic</span><span class="o">());</span>
- <span class="o">}</span>
- <span class="o">},</span> <span class="n">topicCountAndReportPE</span><span class="o">);</span></code></pre>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="kd">final</span> <span class="n">TopicEvent</span> <span class="n">arg0</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">arg0</span><span class="o">.</span><span class="na">getTopic</span><span class="o">());</span>
+ <span class="o">}</span>
+ <span class="o">},</span> <span class="n">topicCountAndReportPE</span><span class="o">);</span></code></pre>
<p>When an event is sent to the âTopicSeenâ stream, its key will be identified through the KeyFinder implementation, hashed and dispatched to the matching partition.</p>
@@ -95,16 +95,16 @@
<p>If we use an AdapterApp subclass, the <code>remoteStreamKeyFinder</code> should be defined in the <code>onInit()</code> method, <em>before</em> calling <code>super.onInit()</code>:</p>
<pre><code class="language-java"> <span class="nd">@Override</span>
- <span class="kd">protected</span> <span class="kt">void</span> <span class="nf">onInit</span><span class="o">()</span> <span class="o">{</span>
- <span class="o">...</span>
- <span class="n">remoteStreamKeyFinder</span> <span class="o">=</span> <span class="k">new</span> <span class="n">KeyFinder</span><span class="o"><</span><span class="n">Event</span><span class="o">>()</span> <span class="o">{</span>
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">return</span> <span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"theKeyField"</span><span class="o">));</span>
- <span class="o">}</span>
- <span class="o">};</span>
- <span class="kd">super</span><span class="o">.</span><span class="na">onInit</span><span class="o">()</span>
- <span class="o">...</span></code></pre>
+ <span class="kd">protected</span> <span class="kt">void</span> <span class="nf">onInit</span><span class="o">()</span> <span class="o">{</span>
+ <span class="o">...</span>
+ <span class="n">remoteStreamKeyFinder</span> <span class="o">=</span> <span class="k">new</span> <span class="n">KeyFinder</span><span class="o"><</span><span class="n">Event</span><span class="o">>()</span> <span class="o">{</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"theKeyField"</span><span class="o">));</span>
+ <span class="o">}</span>
+ <span class="o">};</span>
+ <span class="kd">super</span><span class="o">.</span><span class="na">onInit</span><span class="o">()</span>
+ <span class="o">...</span></code></pre>
<p>If we use a standard App, we use the <code>createOutputStream(String name, KeyFinder<Event> keyFinder)</code> method.</p>
@@ -117,20 +117,20 @@
<p>When receiving events from a remote application, we <em>must</em> define how external events are dispatched internally, to which PEs and based on which keys. For that purpose, we simply define and <em>input stream</em> with the corresponding KeyFinder:</p>
<pre><code class="language-java"><span class="n">createInputStream</span><span class="o">(</span><span class="s">"names"</span><span class="o">,</span> <span class="k">new</span> <span class="n">KeyFinder</span><span class="o"><</span><span class="n">Event</span><span class="o">>()</span> <span class="o">{</span>
-
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">return</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="k">new</span> <span class="n">String</span><span class="o">[]</span> <span class="o">{</span> <span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span> <span class="o">});</span>
- <span class="o">}</span>
- <span class="o">},</span> <span class="n">helloPE</span><span class="o">);</span></code></pre>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="k">new</span> <span class="n">String</span><span class="o">[]</span> <span class="o">{</span> <span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span> <span class="o">});</span>
+ <span class="o">}</span>
+ <span class="o">},</span> <span class="n">helloPE</span><span class="o">);</span></code></pre>
<p>In this case, a name is extracted from each event, the PE instance with this key is retrieved or created, and the event sent to that instance.</p>
<p>Alternatively, we can use a unique PE instance for processing events in a given node. For that we simply define the input stream without a KeyFinder, <em>and</em> use a singleton PE:</p>
<pre><code class="language-java"><span class="n">HelloPE</span> <span class="n">helloPE</span> <span class="o">=</span> <span class="n">createPE</span><span class="o">(</span><span class="n">HelloPE</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
- <span class="n">helloPE</span><span class="o">.</span><span class="na">setSingleton</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span>
- <span class="n">createInputStream</span><span class="o">(</span><span class="s">"names"</span><span class="o">,</span> <span class="n">helloPE</span><span class="o">);</span></code></pre>
+<span class="n">helloPE</span><span class="o">.</span><span class="na">setSingleton</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span>
+<span class="n">createInputStream</span><span class="o">(</span><span class="s">"names"</span><span class="o">,</span> <span class="n">helloPE</span><span class="o">);</span></code></pre>
<p>In this case, all events will be dispatched to the only HelloPE instance in this partition, regardless of the content of the event.</p>
Modified: incubator/s4/site/doc/0.6.0/fault_tolerance/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/fault_tolerance/index.html?rev=1529612&r1=1529611&r2=1529612&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/fault_tolerance/index.html (original)
+++ incubator/s4/site/doc/0.6.0/fault_tolerance/index.html Sun Oct 6 11:56:43 2013
@@ -176,7 +176,7 @@
<p>For instance, here is how to specify a checkpointing frequency of 20s on the TopNTopicPE prototype:</p>
<pre><code>topNTopicPE.setCheckpointingConfig(new CheckpointingConfig.Builder(CheckpointingMode.TIME).frequency(20).timeUnit(TimeUnit.SECONDS).build());
- </code></pre>
+</code></pre>
<h5 id="enabling-checkpointing">Enabling checkpointing</h5>
@@ -185,7 +185,7 @@
<p>We provide a default module (FileSystemBackendCheckpointingModule) that uses a file system backend (DefaultFileSystemStateStorage). It can be used with an NFS setup and introduces no dependency. You may use it by starting an S4 node in the following manner:</p>
<pre><code>./s4 node -c=cluster1 -emc=org.apache.s4.core.ft.FileSystemBackendCheckpointingModule
- </code></pre>
+</code></pre>
<h5 id="customizing-the-checkpointing-backend">Customizing the checkpointing backend</h5>
@@ -194,12 +194,12 @@
<p>Using an alternative backend is as simple as providing a new module to the S4 node. Here is an example of a module using a âCoolâ backend implementation:</p>
<pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">CoolBackendCheckpointingModule</span> <span class="kd">extends</span> <span class="n">AbstractModule</span> <span class="o">{</span>
- <span class="nd">@Override</span>
- <span class="kd">protected</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">()</span> <span class="o">{</span>
- <span class="n">bind</span><span class="o">(</span><span class="n">StateStorage</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">to</span><span class="o">(</span><span class="n">CoolStateStorage</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
- <span class="n">bind</span><span class="o">(</span><span class="n">CheckpointingFramework</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">to</span><span class="o">(</span><span class="n">SafeKeeper</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
- <span class="o">}</span>
- <span class="o">}</span></code></pre>
+ <span class="nd">@Override</span>
+ <span class="kd">protected</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">()</span> <span class="o">{</span>
+ <span class="n">bind</span><span class="o">(</span><span class="n">StateStorage</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">to</span><span class="o">(</span><span class="n">CoolStateStorage</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
+ <span class="n">bind</span><span class="o">(</span><span class="n">CheckpointingFramework</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">to</span><span class="o">(</span><span class="n">SafeKeeper</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre>
<h5 id="overriding-checkpointing-and-recovery-operations">Overriding checkpointing and recovery operations</h5>
Modified: incubator/s4/site/doc/0.6.0/metrics/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/metrics/index.html?rev=1529612&r1=1529611&r2=1529612&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/metrics/index.html (original)
+++ incubator/s4/site/doc/0.6.0/metrics/index.html Sun Oct 6 11:56:43 2013
@@ -88,17 +88,17 @@
<p>The <code>s4.metrics.config</code> parameter enables periodic dumps of aggregated statistics to the <strong>console</strong> or to <strong>files</strong> in csv format. This parameter is specified as a node or application parameter <sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>, and must match the following regular expression: </p>
<pre><code>(csv:.+|console):(\d+):(DAYS|HOURS|MICROSECONDS|MILLISECONDS|MINUTES|NANOSECONDS|SECONDS)
- </code></pre>
+</code></pre>
<p>Examples:</p>
<pre><code># dump metrics to csv files to /path/to/directory every 10 seconds
- # (recommendation: use a different and clean directory for each of the nodes)
- csv:/path/to/directory:10:SECONDS
-
- # dump metrics to the console every minute
- console:1:MINUTES
- </code></pre>
+# (recommendation: use a different and clean directory for each of the nodes)
+csv:/path/to/directory:10:SECONDS
+
+# dump metrics to the console every minute
+console:1:MINUTES
+</code></pre>
<p>Reporting to Ganglia or Graphite is not provided out of the box with S4, but itâs quite easy to add. You simply have to add the corresponding dependencies to your project and enable reporting to these systems during the initialization of your application. See the <a href="http://metrics.codahale.com">metrics</a> documentation for more information.</p>
Modified: incubator/s4/site/doc/0.6.0/tools/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/tools/index.html?rev=1529612&r1=1529611&r2=1529612&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/tools/index.html (original)
+++ incubator/s4/site/doc/0.6.0/tools/index.html Sun Oct 6 11:56:43 2013
@@ -71,7 +71,7 @@
<p>From the source distribution, these tools are built by running:</p>
<pre><code>gradlew s4-tools:installApp
- </code></pre>
+</code></pre>
<p>This compiles the s4-tools subproject and generates shell scripts.</p>
Modified: incubator/s4/site/doc/0.6.0/twitter_trending_example/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/twitter_trending_example/index.html?rev=1529612&r1=1529611&r2=1529612&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/twitter_trending_example/index.html (original)
+++ incubator/s4/site/doc/0.6.0/twitter_trending_example/index.html Sun Oct 6 11:56:43 2013
@@ -93,63 +93,93 @@
</blockquote>
<pre><code>debug=true
- user=<a twitter username>
- password=<matching password>
- </code></pre>
+// not valid since june 2013
+user=<a twitter username>
+password=<matching password>
+</code></pre>
+
+ <p><strong>UPDATE:</strong> In June 2013 Twitter updated the way external consumers can use Twitter APIs: <strong>external consumers must now use <a href="http://oauth.net/">OAuth</a></strong>. </p>
+
+ <p>As a result, the twitter example shipped with S4 distribution <em>does not work out of the box</em>. The good news is there is a workaround and we applied a <a href="https://issues.apache.org/jira/browse/S4-138">patch</a> on the <code>dev</code> branch.</p>
+
+ <p>To make things work with the new Twitter access control scheme:</p>
+
+ <ol>
+ <li>get the code updates
+ <ul>
+ <li>either by checking out the code from the <code>dev</code> branch</li>
+ <li>or by applying the relevant <a href="https://github.com/apache/incubator-s4/commit/276a49772f00a7880d270941a8e05c878ee2484e">change</a> e.g. by cherry-picking it</li>
+ </ul>
+ </li>
+ <li>
+ <p>get an OAuth token. <a href="https://dev.twitter.com/docs/auth/tokens-devtwittercom">This page</a> may help</p>
+ </li>
+ <li>
+ <p>use the token parameters in the twitter4j.properties file in the home directory:</p>
+
+ <pre><code> oauth.consumerKey=<consumer key>
+ oauth.consumerSecret=<consumer secret>
+ oauth.accessToken=<access token>
+ oauth.accessTokenSecret=<access token secret>
+</code></pre>
+ </li>
+ </ol>
+
+ <p>Once Twitter access is configured, we can start the S4 application.</p>
<ul>
<li>
<p>Start a Zookeeper clean instance. From the S4 base directory, do:</p>
<pre><code> ./s4 zkServer -clean
- </code></pre>
+</code></pre>
</li>
<li>
<p>Define 2 clusters : 1 for deploying the twitter-counter app, and 1 for the adapter app</p>
<pre><code> ./s4 newCluster -c=cluster1 -nbTasks=2 -flp=12000; ./s4 newCluster -c=cluster2 -nbTasks=1 -flp=13000
- </code></pre>
+</code></pre>
</li>
<li>
<p>Start 2 app nodes (you may want to start each node in a separate console) :</p>
<pre><code> ./s4 node -c=cluster1
- ./s4 node -c=cluster1
- </code></pre>
+ ./s4 node -c=cluster1
+</code></pre>
</li>
<li>
<p>Start 1 node for the adapter app:</p>
<pre><code> ./s4 node -c=cluster2
- </code></pre>
+</code></pre>
</li>
<li>
<p>Build and deploy twitter-counter app</p>
<pre><code> ./s4 s4r -b=`pwd`/test-apps/twitter-counter/build.gradle -appClass=org.apache.s4.example.twitter.TwitterCounterApp twitter-counter
-
- ./s4 deploy -appName=twitter-counter -c=cluster1 -s4r=`pwd`/test-apps/twitter-counter/build/libs/twitter-counter.s4r
- </code></pre>
+
+ ./s4 deploy -appName=twitter-counter -c=cluster1 -s4r=`pwd`/test-apps/twitter-counter/build/libs/twitter-counter.s4r
+</code></pre>
</li>
<li>
<p>Build and deploy twitter-adapter app. In this example, we donât directly specify the app class of the adapter, we use the deployment approach for apps (remember, the adapter is also an app). </p>
<pre><code> ./s4 s4r -b=`pwd`/test-apps/twitter-adapter/build.gradle -appClass=org.apache.s4.example.twitter.TwitterInputAdapter twitter-adapter
-
- ./s4 deploy -appName=twitter-adapter -c=cluster2 -s4r=`pwd`/test-apps/twitter-adapter/build/libs/twitter-adapter.s4r -p=s4.adapter.output.stream=RawStatus
- </code></pre>
+
+ ./s4 deploy -appName=twitter-adapter -c=cluster2 -s4r=`pwd`/test-apps/twitter-adapter/build/libs/twitter-adapter.s4r -p=s4.adapter.output.stream=RawStatus
+</code></pre>
</li>
<li>
<p>Observe the current 10 most popular topics in file TopNTopics.txt. The file gets updated at regular intervals, and only outputs topics with a minimum of 10 occurrences, so you may have to wait a little before the file is updated :</p>
<pre><code> tail -f TopNTopics.txt
- </code></pre>
+</code></pre>
</li>
<li>
<p>You may also check the status of the S4 node with:</p>
<pre><code> ./s4 status
- </code></pre>
+</code></pre>
</li>
</ul>
Modified: incubator/s4/site/doc/0.6.0/walkthrough/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/walkthrough/index.html?rev=1529612&r1=1529611&r2=1529612&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/walkthrough/index.html (original)
+++ incubator/s4/site/doc/0.6.0/walkthrough/index.html Sun Oct 6 11:56:43 2013
@@ -90,16 +90,16 @@
<p>Compile and install S4 in the local maven repository: (you can also let the tests run without the <code>-DskipTests</code> option)</p>
<pre><code> S4:incubator-s4$ ./gradlew install -DskipTests
- .... verbose logs ...
- </code></pre>
+ .... verbose logs ...
+</code></pre>
</li>
<li>
<p>Build the startup scripts:</p>
<pre><code> S4:incubator-s4$ ./gradlew s4-tools:installApp
- .... verbose logsÂ
- ...:s4-tools:installApp
- </code></pre>
+ .... verbose logsÂ
+ ...:s4-tools:installApp
+</code></pre>
</li>
</ul>
@@ -116,17 +116,17 @@
<p>Create a new application template (here, we create it in the /tmp directory):</p>
<pre><code> S4:incubator-s4$ ./s4 newApp myApp -parentDir=/tmp
- ... some instructions on how to start ...
- </code></pre>
+ ... some instructions on how to start ...
+</code></pre>
</li>
<li>
<p>This creates a sample application in the specified directory, with the following structure:</p>
<pre><code> build.gradle --> the template build file, that you'll need to customize
- gradlew --> references the gradlew script from the S4 installation
- s4 --> references the s4 script from the S4 installation, and adds an "adapter" task
- src/ --> sources (maven-like structure)
- </code></pre>
+ gradlew --> references the gradlew script from the S4 installation
+ s4 --> references the s4 script from the S4 installation, and adds an "adapter" task
+ src/ --> sources (maven-like structure)
+</code></pre>
</li>
</ul>
@@ -140,47 +140,47 @@
</ul>
<pre><code class="language-java"><span class="c1">// ProcessingElement provides integration with the S4 platform</span>
- <span class="kd">public</span> <span class="kd">class</span> <span class="nc">HelloPE</span> <span class="kd">extends</span> <span class="n">ProcessingElement</span> <span class="o">{</span>
-
- <span class="c1">// you should define downstream streams here and inject them in the app definition</span>
-
- <span class="c1">// PEs can maintain some state</span>
- <span class="kt">boolean</span> <span class="n">seen</span> <span class="o">=</span> <span class="kc">false</span><span class="o">;</span>
-
- <span class="c1">// This method is called upon a new Event on an incoming stream.</span>
- <span class="c1">// You may overload it for handling instances of your own specialized subclasses of Event</span>
- <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onEvent</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
- <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Hello "</span> <span class="o">+</span> <span class="o">(</span><span class="n">seen</span> <span class="o">?</span> <span class="s">"again "</span> <span class="o">:</span> <span class="s">""</span><span class="o">)</span> <span class="o">+</span> <span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span> <span class="o">+</span> <span class="s">"!"</span><span class="o">);</span>
- <span class="n">seen</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
- <span class="o">}</span>
- <span class="c1">// skipped remaining methods</span></code></pre>
+<span class="kd">public</span> <span class="kd">class</span> <span class="nc">HelloPE</span> <span class="kd">extends</span> <span class="n">ProcessingElement</span> <span class="o">{</span>
+
+ <span class="c1">// you should define downstream streams here and inject them in the app definition</span>
+
+ <span class="c1">// PEs can maintain some state</span>
+ <span class="kt">boolean</span> <span class="n">seen</span> <span class="o">=</span> <span class="kc">false</span><span class="o">;</span>
+
+ <span class="c1">// This method is called upon a new Event on an incoming stream.</span>
+ <span class="c1">// You may overload it for handling instances of your own specialized subclasses of Event</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onEvent</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Hello "</span> <span class="o">+</span> <span class="o">(</span><span class="n">seen</span> <span class="o">?</span> <span class="s">"again "</span> <span class="o">:</span> <span class="s">""</span><span class="o">)</span> <span class="o">+</span> <span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span> <span class="o">+</span> <span class="s">"!"</span><span class="o">);</span>
+ <span class="n">seen</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
+ <span class="o">}</span>
+ <span class="c1">// skipped remaining methods</span></code></pre>
<ul>
<li>HelloApp.java: defines a simple application: exposes an input stream (ânamesâ), connected to the HelloPE. See <a href="event_dispatch">the event dispatch configuration page</a> for more information about how events are dispatched.</li>
</ul>
<pre><code class="language-java"><span class="c1">// App parent class provides integration with the S4 platform</span>
- <span class="kd">public</span> <span class="kd">class</span> <span class="nc">HelloApp</span> <span class="kd">extends</span> <span class="n">App</span> <span class="o">{</span>
-
- <span class="nd">@Override</span>
- <span class="kd">protected</span> <span class="kt">void</span> <span class="nf">onStart</span><span class="o">()</span> <span class="o">{</span>
- <span class="o">}</span>
-
- <span class="nd">@Override</span>
- <span class="kd">protected</span> <span class="kt">void</span> <span class="nf">onInit</span><span class="o">()</span> <span class="o">{</span>
- <span class="c1">// That's where we define PEs and streams</span>
- <span class="c1">// create a prototype</span>
- <span class="n">HelloPE</span> <span class="n">helloPE</span> <span class="o">=</span> <span class="n">createPE</span><span class="o">(</span><span class="n">HelloPE</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
- <span class="c1">// Create a stream that listens to the "lines" stream and passes events to the helloPE instance.</span>
- <span class="n">createInputStream</span><span class="o">(</span><span class="s">"names"</span><span class="o">,</span> <span class="k">new</span> <span class="n">KeyFinder</span><span class="o"><</span><span class="n">Event</span><span class="o">>()</span> <span class="o">{</span>
- <span class="c1">// the KeyFinder is used to identify keys</span>
- <span class="nd">@Override</span>
- <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
- <span class="k">return</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="k">new</span> <span class="n">String</span><span class="o">[]</span> <span class="o">{</span> <span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span> <span class="o">});</span>
- <span class="o">}</span>
- <span class="o">},</span> <span class="n">helloPE</span><span class="o">);</span>
- <span class="o">}</span>
- <span class="c1">// skipped remaining methods</span></code></pre>
+<span class="kd">public</span> <span class="kd">class</span> <span class="nc">HelloApp</span> <span class="kd">extends</span> <span class="n">App</span> <span class="o">{</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">protected</span> <span class="kt">void</span> <span class="nf">onStart</span><span class="o">()</span> <span class="o">{</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">protected</span> <span class="kt">void</span> <span class="nf">onInit</span><span class="o">()</span> <span class="o">{</span>
+ <span class="c1">// That's where we define PEs and streams</span>
+ <span class="c1">// create a prototype</span>
+ <span class="n">HelloPE</span> <span class="n">helloPE</span> <span class="o">=</span> <span class="n">createPE</span><span class="o">(</span><span class="n">HelloPE</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
+ <span class="c1">// Create a stream that listens to the "lines" stream and passes events to the helloPE instance.</span>
+ <span class="n">createInputStream</span><span class="o">(</span><span class="s">"names"</span><span class="o">,</span> <span class="k">new</span> <span class="n">KeyFinder</span><span class="o"><</span><span class="n">Event</span><span class="o">>()</span> <span class="o">{</span>
+ <span class="c1">// the KeyFinder is used to identify keys</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="k">new</span> <span class="n">String</span><span class="o">[]</span> <span class="o">{</span> <span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span> <span class="o">});</span>
+ <span class="o">}</span>
+ <span class="o">},</span> <span class="n">helloPE</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="c1">// skipped remaining methods</span></code></pre>
<ul>
<li>
@@ -208,10 +208,10 @@
<p>Start a Zookeeper server instance (<code>-clean</code> option removes previous ZooKeeper data, if any):</p>
<pre><code> S4:incubator-s4$ ./s4 zkServer - clean
- S4:myApp$ calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
- [main] INFO org.apache.s4.tools.ZKServer - Starting zookeeper server on port [2181]
- [main] INFO org.apache.s4.tools.ZKServer - cleaning existing data in [/var/folders/8V/8VdgKWU3HCiy2yV4dzFpDk+++TI/-Tmp-/tmp/zookeeper/data] and [/var/folders/8V/8VdgKWU3HCiy2yV4dzFpDk+++TI/-Tmp-/tmp/zookeeper/log]
- </code></pre>
+ S4:myApp$ calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
+ [main] INFO org.apache.s4.tools.ZKServer - Starting zookeeper server on port [2181]
+ [main] INFO org.apache.s4.tools.ZKServer - cleaning existing data in [/var/folders/8V/8VdgKWU3HCiy2yV4dzFpDk+++TI/-Tmp-/tmp/zookeeper/data] and [/var/folders/8V/8VdgKWU3HCiy2yV4dzFpDk+++TI/-Tmp-/tmp/zookeeper/log]
+</code></pre>
</li>
<li>
<p>Define a new cluster. Say a cluster named âcluster1â with 2 partitions, nodes listening to ports starting from 12000:</p>
@@ -235,20 +235,20 @@
</ul>
<pre><code class="language-bash">S4:myApp<span class="nv">$ </span>./s4 node -c<span class="o">=</span>cluster1
- calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
- 15:50:18.996 <span class="o">[</span>main<span class="o">]</span> INFO org.apache.s4.core.Main - Initializing S4 node with :
- - comm module class <span class="o">[</span>org.apache.s4.comm.DefaultCommModule<span class="o">]</span>
- - comm configuration file <span class="o">[</span>default.s4.comm.properties from classpath<span class="o">]</span>
- - core module class <span class="o">[</span>org.apache.s4.core.DefaultCoreModule<span class="o">]</span>
- - core configuration file<span class="o">[</span>default.s4.core.properties from classpath<span class="o">]</span>
- -extra modules: <span class="o">[]</span>
- <span class="o">[</span>main<span class="o">]</span> INFO org.apache.s4.core.Main - Starting S4 node.</code></pre>
+calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
+15:50:18.996 <span class="o">[</span>main<span class="o">]</span> INFO org.apache.s4.core.Main - Initializing S4 node with :
+- comm module class <span class="o">[</span>org.apache.s4.comm.DefaultCommModule<span class="o">]</span>
+- comm configuration file <span class="o">[</span>default.s4.comm.properties from classpath<span class="o">]</span>
+- core module class <span class="o">[</span>org.apache.s4.core.DefaultCoreModule<span class="o">]</span>
+- core configuration file<span class="o">[</span>default.s4.core.properties from classpath<span class="o">]</span>
+-extra modules: <span class="o">[]</span>
+<span class="o">[</span>main<span class="o">]</span> INFO org.apache.s4.core.Main - Starting S4 node.</code></pre>
<p>This node will automatically download applications published for the cluster it belongs to
and again (maybe in another shell):</p>
<pre><code>S4:myApp$ ./s4 node -c=cluster1
- </code></pre>
+</code></pre>
<ul>
<li>Build, package and publish the app to cluster1:
@@ -260,13 +260,13 @@
Again specifying the app class is optional : </p>
<pre><code> ./s4 s4r -a=hello.HelloApp -b=`pwd`/build.gradle myApp
- </code></pre>
+</code></pre>
</li>
<li>
<p>Publish the s4r archive (you may first copy it to a more adequate place). The name of the app is arbitrary: </p>
<pre><code> ./s4 deploy -s4r=`pwd`/build/libs/myApp.s4r -c=cluster1 -appName=myApp
- </code></pre>
+</code></pre>
</li>
</ol>
</li>
@@ -276,24 +276,24 @@
<p>S4 nodes will detect the new application, download it, load it and start it. You will get something like:</p>
<pre><code> [ZkClient-EventThread-15-localhost:2181] INFO o.a.s.d.DistributedDeploymentManager - Detected new application(s) to deploy {}[myApp]
- [ZkClient-EventThread-15-localhost:2181] INFO org.apache.s4.core.Server - Local app deployment: using s4r file name [myApp] as application name
- [ZkClient-EventThread-15-localhost:2181] INFO org.apache.s4.core.Server - App class name is: hello.HelloApp
- [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=0,name=unknown,mode=unicast,type=,nodes=[]} from null
- [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@79b2591c
- [ZkClient-EventThread-15-localhost:2181] INFO o.a.s.comm.topology.AssignmentFromZK - New session:87684175268872203; state is : SyncConnected
- [ZkClient-EventThread-19-localhost:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=myMachine.myNetwork,taskId=Task-0}]} from { nbNodes=0,name=unknown,mode=unicast,type=,nodes=[]}
- [ZkClient-EventThread-15-localhost:2181] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by myMachine.myNetwork
- [ZkClient-EventThread-19-localhost:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=myMachine.myNetwork,taskId=Task-0}, {partition=1,port=12001,machineName=myMachine.myNetwork,taskId=Task-1}]} from { nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=myMachine.myNetwork,taskId=Task-0}]}
- [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:87684175268872205
- [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [names]
- [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:87684175268872206
- [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=myMachine.myNetwork,taskId=Task-0}, {partition=1,port=12001,machineName=myMachine.myNetwork,taskId=Task-1}]} from null
- [ZkClient-EventThread-15-localhost:2181] INFO org.apache.s4.core.Server - Loaded application from file /tmp/deploy-test/cluster1/myApp.s4r
- [ZkClient-EventThread-15-localhost:2181] INFO o.a.s.d.DistributedDeploymentManager - Successfully installed application myApp
- [ZkClient-EventThread-15-localhost:2181] DEBUG o.a.s.c.g.OverloadDispatcherGenerator - Dumping generated overload dispatcher class for PE of class [class hello.HelloPE]
- [ZkClient-EventThread-15-localhost:2181] DEBUG o.a.s4.comm.topology.ClustersFromZK - Adding input stream [names] for app [-1] in cluster [cluster1]
- [ZkClient-EventThread-15-localhost:2181] INFO org.apache.s4.core.App - Init prototype [hello.HelloPE].
- </code></pre>
+ [ZkClient-EventThread-15-localhost:2181] INFO org.apache.s4.core.Server - Local app deployment: using s4r file name [myApp] as application name
+ [ZkClient-EventThread-15-localhost:2181] INFO org.apache.s4.core.Server - App class name is: hello.HelloApp
+ [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=0,name=unknown,mode=unicast,type=,nodes=[]} from null
+ [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Adding topology change listener:org.apache.s4.comm.tcp.TCPEmitter@79b2591c
+ [ZkClient-EventThread-15-localhost:2181] INFO o.a.s.comm.topology.AssignmentFromZK - New session:87684175268872203; state is : SyncConnected
+ [ZkClient-EventThread-19-localhost:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=myMachine.myNetwork,taskId=Task-0}]} from { nbNodes=0,name=unknown,mode=unicast,type=,nodes=[]}
+ [ZkClient-EventThread-15-localhost:2181] INFO o.a.s.comm.topology.AssignmentFromZK - Successfully acquired task:Task-1 by myMachine.myNetwork
+ [ZkClient-EventThread-19-localhost:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=myMachine.myNetwork,taskId=Task-0}, {partition=1,port=12001,machineName=myMachine.myNetwork,taskId=Task-1}]} from { nbNodes=1,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=myMachine.myNetwork,taskId=Task-0}]}
+ [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:87684175268872205
+ [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClustersFromZK - Detected new stream [names]
+ [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClustersFromZK - New session:87684175268872206
+ [ZkClient-EventThread-15-localhost:2181] INFO o.a.s4.comm.topology.ClusterFromZK - Changing cluster topology to { nbNodes=2,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=myMachine.myNetwork,taskId=Task-0}, {partition=1,port=12001,machineName=myMachine.myNetwork,taskId=Task-1}]} from null
+ [ZkClient-EventThread-15-localhost:2181] INFO org.apache.s4.core.Server - Loaded application from file /tmp/deploy-test/cluster1/myApp.s4r
+ [ZkClient-EventThread-15-localhost:2181] INFO o.a.s.d.DistributedDeploymentManager - Successfully installed application myApp
+ [ZkClient-EventThread-15-localhost:2181] DEBUG o.a.s.c.g.OverloadDispatcherGenerator - Dumping generated overload dispatcher class for PE of class [class hello.HelloPE]
+ [ZkClient-EventThread-15-localhost:2181] DEBUG o.a.s4.comm.topology.ClustersFromZK - Adding input stream [names] for app [-1] in cluster [cluster1]
+ [ZkClient-EventThread-15-localhost:2181] INFO org.apache.s4.core.App - Init prototype [hello.HelloPE].
+</code></pre>
</li>
</ul>
@@ -302,7 +302,7 @@
<p>You can check the status of the application, nodes and streams with the âstatusâ command:</p>
<pre><code>./s4 status
- </code></pre>
+</code></pre>
<p>Now what we need is some input!</p>
@@ -321,7 +321,7 @@
<p>First, we need to define a new S4 subcluster for that app:</p>
<pre><code> S4:myApp$ ./s4 newCluster -c=cluster2 -nbTasks=1 -flp=13000
- </code></pre>
+</code></pre>
</li>
<li>Then we configure the application:
<ul>
@@ -331,7 +331,7 @@
<p><em>there is also a -s4r parameter, indicating where to fetch the application package from. We donât need it here, since we skip that step and use a special âadapterâ tool</em></p>
<pre><code> ./s4 deploy -appClass=hello.HelloInputAdapter -p=s4.adapter.output.stream=names -c=cluster2 -appName=adapter
- </code></pre>
+</code></pre>
</li>
</ul>
</li>
@@ -340,19 +340,19 @@
> The adapter command must be run from the root of your S4 project (myApp dir in our case).</p>
<pre><code> ./s4 adapter -c=cluster2
- </code></pre>
+</code></pre>
</li>
<li>
<p>Now letâs just provide some data to the external stream (our adapter is listening to port 15000):</p>
<pre><code> S4:~$ echo "Bob" | nc localhost 15000
- </code></pre>
+</code></pre>
</li>
<li>
<p>One of the nodes should output in its console:</p>
<pre><code> Hello Bob!
- </code></pre>
+</code></pre>
</li>
</ul>