You are viewing a plain text version of this content. The canonical link for it is here.
Posted to s4-commits@incubator.apache.org by mm...@apache.org on 2013/02/27 18:59:52 UTC
svn commit: r1450917 [1/2] - in /incubator/s4/site: ./ contrib/ doc/0.6.0/
doc/0.6.0/application_dependencies/ doc/0.6.0/configuration/
doc/0.6.0/dev_tips/ doc/0.6.0/event_dispatch/ doc/0.6.0/fault_tolerance/
doc/0.6.0/metrics/ doc/0.6.0/overview/ doc/...
Author: mmorel
Date: Wed Feb 27 18:59:51 2013
New Revision: 1450917
URL: http://svn.apache.org/r1450917
Log:
0.6.0 doc updates
Added:
incubator/s4/site/doc/0.6.0/metrics/
incubator/s4/site/doc/0.6.0/metrics/index.html
incubator/s4/site/images/doc/0.6.0/executors.png (with props)
incubator/s4/site/images/doc/0.6.0/sources/executors.odg (with props)
incubator/s4/site/images/doc/0.6.0/sources/queues.odg (with props)
incubator/s4/site/style/coderay.css
incubator/s4/site/style/pygmentize.css
Modified:
incubator/s4/site/contrib/index.html
incubator/s4/site/doc/0.6.0/application_dependencies/index.html
incubator/s4/site/doc/0.6.0/configuration/index.html
incubator/s4/site/doc/0.6.0/dev_tips/index.html
incubator/s4/site/doc/0.6.0/event_dispatch/index.html
incubator/s4/site/doc/0.6.0/fault_tolerance/index.html
incubator/s4/site/doc/0.6.0/index.html
incubator/s4/site/doc/0.6.0/overview/index.html
incubator/s4/site/doc/0.6.0/walkthrough/index.html
incubator/s4/site/download/index.html
incubator/s4/site/index.html
incubator/s4/site/style/style.css
incubator/s4/site/team/index.html
Modified: incubator/s4/site/contrib/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/contrib/index.html?rev=1450917&r1=1450916&r2=1450917&view=diff
==============================================================================
--- incubator/s4/site/contrib/index.html (original)
+++ incubator/s4/site/contrib/index.html Wed Feb 27 18:59:51 2013
@@ -20,7 +20,8 @@
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
- </script>
+ </script><link href="../style/coderay.css" rel="stylesheet" type="text/css">
+<link href="../style/pygmentize.css" rel="stylesheet" type="text/css">
</head>
<body>
<div id="header">
Modified: incubator/s4/site/doc/0.6.0/application_dependencies/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/application_dependencies/index.html?rev=1450917&r1=1450916&r2=1450917&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/application_dependencies/index.html (original)
+++ incubator/s4/site/doc/0.6.0/application_dependencies/index.html Wed Feb 27 18:59:51 2013
@@ -22,6 +22,8 @@
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
+ <link href='/style/coderay.css' rel='stylesheet' type='text/css' />
+ <link href='/style/pygmentize.css' rel='stylesheet' type='text/css' />
</head>
<body>
<div id='header'>
@@ -68,7 +70,7 @@
<h2 id="dependencies-on-public-artifacts">Dependencies on public artifacts</h2>
<ul>
- <li>
+<li>
<p>Add maven artifacts definitions to the gradle build file. For instance, add twitter4j_core and twitter4j_stream:</p>
<pre><code> project.ext["libraries"] = [
@@ -93,11 +95,9 @@
</code></pre>
</li>
<li>
- <p>If you use an IDE such as eclipse, you may update your project’s classpath with: <code>./gradlew eclipse</code></p>
+ <p>If you use an IDE such as eclipse, you may update your projectâs classpath with: <code>./gradlew eclipse</code></p>
</li>
-</ul>
-
-<p>A good source for finding dependencies is for instance <a href="http://search.maven.org/">http://search.maven.org/</a> where you also get the syntax for gradle scripts (see grails syntax).</p>
+</ul><p>A good source for finding dependencies is for instance <a href="http://search.maven.org/">http://search.maven.org/</a> where you also get the syntax for gradle scripts (see grails syntax).</p>
<blockquote>
<p>The application dependencies will be automatically included in the s4r archive that you create and publish.</p>
@@ -108,11 +108,10 @@
<p>You may have dependencies that are not published to maven repositories. In that case you should either:</p>
<ul>
- <li>publish them to your local maven repository, see <a href="[http://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html">http://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html</a></li>
+<li>publish them to your local maven repository, see <a href="%5Bhttp://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html">http://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html</a>
+</li>
<li>add them to the <em>lib</em> directory</li>
-</ul>
-
-<p>In both cases you still have to declare them as compile-time dependencies.</p></div>
+</ul><p>In both cases you still have to declare them as compile-time dependencies.</p></div>
</div>
<div id='footer'>
<div class='container'>
Modified: incubator/s4/site/doc/0.6.0/configuration/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/configuration/index.html?rev=1450917&r1=1450916&r2=1450917&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/configuration/index.html (original)
+++ incubator/s4/site/doc/0.6.0/configuration/index.html Wed Feb 27 18:59:51 2013
@@ -22,6 +22,8 @@
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
+ <link href='/style/coderay.css' rel='stylesheet' type='text/css' />
+ <link href='/style/pygmentize.css' rel='stylesheet' type='text/css' />
</head>
<body>
<div id='header'>
@@ -57,101 +59,96 @@
</div>
</div>
<div id='wrapper'>
- <div class='container' id='container'><h1 id="toolset">Toolset</h1>
+ <div class='container' id='container'><blockquote>
+ <p>How to configure S4 clusters and applications</p>
+</blockquote>
+
+<h1 id="toolset">Toolset</h1>
<p>S4 provides a set of tools to:</p>
<ul>
- <li>define S4 clusters: <code>s4 newCluster</code></li>
- <li>start S4 nodes: <code>s4 node</code></li>
- <li>package applications: <code>s4 s4r</code></li>
- <li>deploy applications: <code>s4 deploy</code></li>
+<li>define S4 clusters: <code>s4 newCluster</code>
+</li>
+ <li>start S4 nodes: <code>s4 node</code>
+</li>
+ <li>package applications: <code>s4 s4r</code>
+</li>
+ <li>deploy applications: <code>s4 deploy</code>
+</li>
<li>start a Zookeeper server for easy testing: <code>s4 zkServer</code>
<ul>
- <li><code>s4 zkServer -t</code> will start a Zookeeper server and automatically configure 2 clusters</li>
+<li>
+<code>s4 zkServer -t</code> will start a Zookeeper server and automatically configure 2 clusters</li>
</ul>
- </li>
+</li>
+ <li>view the status of S4 clusters coordinated by a given Zookeeper ensemble: <code>s4 status</code>
+</li>
<li>
- <p>view the status of S4 clusters coordinated by a given Zookeeper ensemble: <code>s4 status</code></p>
-
- <pre><code> ./s4
-</code></pre>
- </li>
-</ul>
-
-<p>will give you a list of available commands.</p>
-
-<pre><code>./s4 <command> -help
-</code></pre>
-
-<p>will provide detailed documentation for each of these commands.</p>
-
-<h1 id="cluster-configuration">Cluster configuration</h1>
+<code>s4</code> will give you a list of available commands.</li>
+ <li>
+<code>./s4 <command> -help</code> will provide detailed documentation for each of these commands.</li>
+</ul><h1 id="cluster-configuration">Cluster configuration</h1>
<p>Before starting S4 nodes, you must define a logical cluster by specifying:</p>
<ul>
- <li>a name for the cluster</li>
+<li>a name for the cluster</li>
<li>a number of partitions (~ tasks)</li>
<li>an initial port number for listener sockets
<ul>
- <li>you must specify a free port, considering that each of the nodes of the cluster will open a different port, with a number monotonically increasing from the initial number. For instance, for a cluster of 10 nodes and an initial port 12000, ports 12000 to 12009 will be used among the nodes.</li>
+<li>you must specify a free port, considering that each of the nodes of the cluster will open a different port, with a number monotonically increasing from the initial number. For instance, for a cluster of 10 nodes and an initial port 12000, ports 12000 to 12009 will be used among the nodes.</li>
<li>those ports are used for inter node communication.</li>
</ul>
- </li>
-</ul>
+</li>
+</ul><p>The cluster configuration is maintained in Zookeeper, and can be set using S4 tools:</p>
-<p>The cluster configuration is maintained in Zookeeper, and can be set using S4 tools:</p>
+<pre><code>./s4 newCluster -c=cluster1 -nbTasks=2 -flp=12000
+</code></pre>
-<pre><code>./s4 newCluster -c=cluster1 -nbTasks=2 -flp=12000 See tool documentation by typing:
+<p>See tool documentation by typing:</p>
-./s4 newCluster -help
+<pre><code>./s4 newCluster -help
</code></pre>
<h1 id="node-configuration">Node configuration</h1>
-<p><em>Platform</em> <em>code and</em> <em>application</em> <em>code are fully configurable,</em> <em>at deployment time{</em>}<em>.</em></p>
+<p><strong>Platform code and application code are fully configurable, at deployment time.</strong></p>
-<p>S4 nodes start as simple <em>bootstrap</em> processes whose initial role is merely to connect the cluster manager:</p>
+<p>S4 nodes start as simple <strong>bootstrap</strong> processes whose initial role is merely to connect the cluster manager:</p>
-<ul>
- <li>the bootstrap code connects to the cluster manager</li>
+<ol>
+<li>the bootstrap code connects to the cluster manager</li>
<li>when an application is available on the cluster, the node gets notified</li>
<li>it downloads the platform configuration and code, as specified in the configuration of the deployed application.</li>
<li>the communication and core components are loaded, bound and initialized</li>
<li>the application configuration and code, as specified in the configuration of the deployed applciation, is downloaded</li>
<li>the application is initialized and started</li>
-</ul>
-
-<p>This figure illustrates the separation between the bootstrap code, the S4 platform code, and application code in an S4 node:</p>
+</ol><p>This figure illustrates the separation between the bootstrap code, the S4 platform code, and application code in an S4 node:</p>
-<p><img src="/images/doc/0.6.0/s4_node_layers.png" alt="image" /></p>
+<p><img src="/images/doc/0.6.0/s4_node_layers.png" alt="image"></p>
<p>Therefore, for starting an S4 node on a given host, you only need to specify:</p>
<ul>
- <li>the connection string to the cluster management system (Zookeeper) <code>localhost:2181</code> by default</li>
+<li>the connection string to the cluster management system (Zookeeper) <code>localhost:2181</code> by default</li>
<li>the name of the logical cluster to which this node will belong</li>
-</ul>
-
-<p>Example:
+</ul><p>Example:
<code>./s4 node -c=cluster1 -zk=host.domain.com</code></p>
<h1 id="application-configuration">Application configuration</h1>
-<p>Deploying applications is easier when we can define both the parameters of the application <em>and</em> the target environment.</p>
+<p>Deploying applications is easier when we can define both the parameters of the application <strong>and</strong> the target environment.</p>
-<p>In S4, we achieve this by specifying <em>both</em> application parameters and S4 platform parameters in the deployment phase :</p>
+<p>In S4, we achieve this by specifying <strong>both</strong> application parameters and S4 platform parameters in the deployment phase :</p>
<ul>
- <li>which application class to use</li>
+<li>which application class to use</li>
<li>where to fetch application code</li>
<li>which specific modules to use</li>
<li>where to fetch these modules</li>
<li>string configuration parameters - that can be used by the application and the modules</li>
-</ul>
-
-<h2 id="modules-configuration">Modules configuration</h2>
+</ul><h2 id="modules-configuration">Modules configuration</h2>
<p>S4 follows a modular design and uses<a href="http://code.google.com/p/google-guice/">Guice</a> for defining modules and injecting dependencies.</p>
@@ -183,7 +180,7 @@
<p>For instance, if you checkpoint through a specific key value store, you can write you own checkpointing implementation and module, package that into fancyKeyValueStoreCheckpointingModule.jar , and then:</p>
-<pre><code>./s4 node -c=cluster1 -emc=my.project.FancyKeyValueStoreBackendCheckpointingModule \
+<pre><code>./s4 deploy -c=cluster1 -emc=my.project.FancyKeyValueStoreBackendCheckpointingModule \
-mu=uri/to/fancyKeyValueStoreCheckpointingModule.jar
</code></pre>
@@ -192,20 +189,14 @@
<p>A simple way to pass parameters to your application code is by:</p>
<ul>
- <li>
- <p>injecting them in the application class:</p>
+<li>injecting them in the application class (primitive types, enums and class literals are automatically converted), for instance:</li>
+</ul><pre><code class="language-java"><span class="nd">@Inject</span>
+<span class="nd">@Named</span><span class="o">(</span><span class="s">"thePortNumber"</span><span class="o">)</span>
+<span class="kt">int</span> <span class="n">port</span></code></pre>
- <pre><code> @Inject
- @Named('myParam')
- param
-</code></pre>
- </li>
- <li>
- <p>specifying the parameter value at node startup (using -p inline with the node command, or with the ‘@’ syntax)</p>
- </li>
-</ul>
-
-<p>S4 uses an internal Guice module that automatically injects configuration parameters passed through the deploy command to matching <code>@Named</code> parameters.</p>
+<ul>
+<li>specifying the parameter value at node startup (using -p inline with the node command, or with the â@â syntax)</li>
+</ul><p>S4 uses an internal Guice module that automatically injects configuration parameters passed through the deploy command to matching <code>@Named</code> parameters.</p>
<p>Both application and platform parameters can be overriden. For instance, specifying a custom storage path for the file system based checkpointing mechanism would be passing the <code>s4.checkpointing.filesystem.storageRootPath</code> parameter:</p>
@@ -216,8 +207,8 @@
<h2 id="file-based-configuration">File-based configuration</h2>
-<p>Instead of specifying node parameters inline, you may refer to a file with the ‘@’ notation:
-./s4 deploy @/path/to/config/file
+<p>Instead of specifying node parameters inline, you may refer to a file with the â@â notation:
+<code>./s4 deploy @/path/to/config/file</code>
With contents of the referenced file like:</p>
<pre><code>-s4r=uri/to/app.s4r
Modified: incubator/s4/site/doc/0.6.0/dev_tips/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/dev_tips/index.html?rev=1450917&r1=1450916&r2=1450917&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/dev_tips/index.html (original)
+++ incubator/s4/site/doc/0.6.0/dev_tips/index.html Wed Feb 27 18:59:51 2013
@@ -22,6 +22,8 @@
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
+ <link href='/style/coderay.css' rel='stylesheet' type='text/css' />
+ <link href='/style/pygmentize.css' rel='stylesheet' type='text/css' />
</head>
<body>
<div id='header'>
@@ -57,16 +59,18 @@
</div>
</div>
<div id='wrapper'>
- <div class='container' id='container'><p>Here are a few tips to ease the development of S4 applications.</p>
+ <div class='container' id='container'><blockquote>
+ <p>Here are a few tips to ease the development of S4 applications.</p>
+</blockquote>
-<h3 id="import-an-s4-project-into-your-ide">Import an S4 project into your IDE</h3>
+<h1 id="import-an-s4-project-into-your-ide">Import an S4 project into your IDE</h1>
-<p>You can run <code>gradlew eclipse</code> or <code>gradlew idea</code> at the root of your S4 application directory. Then simply import the project into eclipse or intellij. You’ll have both your application classes <em>and</em> S4 libraries imported to the classpath of the project.</p>
+<p>You can run <code>gradlew eclipse</code> or <code>gradlew idea</code> at the root of your S4 application directory. Then simply import the project into eclipse or intellij. Youâll have both your application classes <em>and</em> S4 libraries imported to the classpath of the project.</p>
<p>In order to get the transitive dependencies of the platform included as well, you should:</p>
<ul>
- <li>Download a source distribution</li>
+<li>Download a source distribution</li>
<li>
<p>Install S4 and its dependencies in your local maven repository</p>
@@ -74,13 +78,12 @@
./gradlew install -DskipTests
</code></pre>
</li>
- <li>Then run <code>gradlew eclipse</code> or <code>gradlew idea</code></li>
-</ul>
-
-<h3 id="start-a-local-zookeeper-instance">Start a local Zookeeper instance</h3>
+ <li>Then run <code>gradlew eclipse</code> or <code>gradlew idea</code>
+</li>
+</ul><hr><h1 id="start-a-local-zookeeper-instance">Start a local Zookeeper instance</h1>
<ul>
- <li>
+<li>
<p>Use the default test configuration (2 clusters with following configs: <code>c=testCluster1:flp=12000:nbTasks=1</code> and <code>c=testCluster2:flp=13000:nbTasks=1</code>)</p>
<pre><code> s4 zkServer -t
@@ -92,9 +95,7 @@
<pre><code> s4 zkServer -clusters=c=testCluster1:flp=12000:nbTasks=1
</code></pre>
</li>
-</ul>
-
-<h3 id="load-an-application-in-a-new-node-directly-from-an-ide">Load an application in a new node directly from an IDE</h3>
+</ul><hr><h1 id="load-an-application-in-a-new-node-directly-from-an-ide">Load an application in a new node directly from an IDE</h1>
<p>This allows to <em>skip the packaging phase!</em></p>
@@ -103,11 +104,11 @@
<p>Then you just need to run the <code>org.apache.s4.core.Main</code> class and pass:</p>
<ul>
- <li>the cluster name: <code>-c=testCluster1</code></li>
- <li>the app class name: <code>-appClass=myAppClass</code></li>
-</ul>
-
-<p>If you use a local Zookeeper instance, there is no need to specify the <code>-zk</code> option.</p></div>
+<li>the cluster name: <code>-c=testCluster1</code>
+</li>
+ <li>the app class name: <code>-appClass=myAppClass</code>
+</li>
+</ul><p>If you use a local Zookeeper instance, there is no need to specify the <code>-zk</code> option.</p></div>
</div>
<div id='footer'>
<div class='container'>
Modified: incubator/s4/site/doc/0.6.0/event_dispatch/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/event_dispatch/index.html?rev=1450917&r1=1450916&r2=1450917&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/event_dispatch/index.html (original)
+++ incubator/s4/site/doc/0.6.0/event_dispatch/index.html Wed Feb 27 18:59:51 2013
@@ -22,6 +22,8 @@
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
+ <link href='/style/coderay.css' rel='stylesheet' type='text/css' />
+ <link href='/style/pygmentize.css' rel='stylesheet' type='text/css' />
</head>
<body>
<div id='header'>
@@ -57,74 +59,145 @@
</div>
</div>
<div id='wrapper'>
- <div class='container' id='container'><p>Events are dispatched according to their key.</p>
+ <div class='container' id='container'><blockquote>
+ <p>Exploring how events are dispatched to, from and within S4 nodes</p>
+</blockquote>
+
+<p>Events are dispatched according to their key.</p>
<p>The key is identified in an <code>Event</code> through a <code>KeyFinder</code>.</p>
-<p>Dispatch can be configured for:
-* dispatching events to partitions (<em>outgoing dispatch</em>)
-* dispatching external events within a partition (<em>incoming dispatch</em>)</p>
+<p>Dispatch can be configured for:</p>
-<h1 id="outgoing-dispatch">Outgoing dispatch</h1>
+<ul>
+<li>dispatching events to partitions (<em>outgoing dispatch</em>)</li>
+ <li>dispatching external events within a partition (<em>incoming dispatch</em>)</li>
+</ul><h1 id="outgoing-dispatch">Outgoing dispatch</h1>
<p>A stream can be defined with a KeyFinder, as :</p>
-<p>Stream<topicevent> topicSeenStream = createStream("TopicSeen", new KeyFinder<topicevent>() {</topicevent></topicevent></p>
-
-<pre><code> @Override
- public List<String> get(final TopicEvent arg0) {
- return ImmutableList.of(arg0.getTopic());
- }
-}, topicCountAndReportPE);
-</code></pre>
+<pre><code class="language-java"><span class="n">Stream</span><span class="o"><</span><span class="n">TopicEvent</span><span class="o">></span> <span class="n">topicSeenStream</span> <span class="o">=</span> <span class="n">createStream</span><span class="o">(</span><span class="s">"TopicSeen"</span><span class="o">,</span> <span class="k">new</span> <span class="n">KeyFinder</span><span class="o"><</span><span class="n">TopicEvent</span><span class="o">>()</span> <span class="o">{</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="kd">final</span> <span class="n">TopicEvent</span> <span class="n">arg0</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">arg0</span><span class="o">.</span><span class="na">getTopic</span><span class="o">());</span>
+ <span class="o">}</span>
+ <span class="o">},</span> <span class="n">topicCountAndReportPE</span><span class="o">);</span></code></pre>
-<p>When an event is sent to the “TopicSeen” stream, its key will be identified through the KeyFinder implementation, hashed and dispatched to the matching partition.</p>
+<p>When an event is sent to the âTopicSeenâ stream, its key will be identified through the KeyFinder implementation, hashed and dispatched to the matching partition.</p>
<p>The same logic applies when defining <em>output streams</em>.</p>
<p>If we use an AdapterApp subclass, the <code>remoteStreamKeyFinder</code> should be defined in the <code>onInit()</code> method, <em>before</em> calling <code>super.onInit()</code>:</p>
-<pre><code>@Override
-protected void onInit() {
-...
-remoteStreamKeyFinder = new KeyFinder<Event>() {
-
- @Override
- public List<String> get(Event event) {
- return ImmutableList.of(event.get("theKeyField"));
- }
- };
-super.onInit()
-...
-</code></pre>
+<pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">protected</span> <span class="kt">void</span> <span class="nf">onInit</span><span class="o">()</span> <span class="o">{</span>
+ <span class="o">...</span>
+ <span class="n">remoteStreamKeyFinder</span> <span class="o">=</span> <span class="k">new</span> <span class="n">KeyFinder</span><span class="o"><</span><span class="n">Event</span><span class="o">>()</span> <span class="o">{</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"theKeyField"</span><span class="o">));</span>
+ <span class="o">}</span>
+ <span class="o">};</span>
+ <span class="kd">super</span><span class="o">.</span><span class="na">onInit</span><span class="o">()</span>
+ <span class="o">...</span></code></pre>
<p>If we use a standard App, we use the <code>createOutputStream(String name, KeyFinder<Event> keyFinder)</code> method.</p>
-<p>bq. If the KeyFinder is not defined for the output streams, events are sent to partitions of the connected cluster in a round robin fashion.</p>
+<blockquote>
+ <p>If the KeyFinder is not defined for the output streams, events are sent to partitions of the connected cluster in a round robin fashion.</p>
+</blockquote>
<h1 id="incoming-dispatch-from-external-events">Incoming dispatch from external events</h1>
<p>When receiving events from a remote application, we <em>must</em> define how external events are dispatched internally, to which PEs and based on which keys. For that purpose, we simply define and <em>input stream</em> with the corresponding KeyFinder:</p>
-<p>createInputStream(“names”, new KeyFinder<event>() {</event></p>
+<pre><code class="language-java"><span class="n">createInputStream</span><span class="o">(</span><span class="s">"names"</span><span class="o">,</span> <span class="k">new</span> <span class="n">KeyFinder</span><span class="o"><</span><span class="n">Event</span><span class="o">>()</span> <span class="o">{</span>
-<pre><code>@Override
-public List<String> get(Event event) {
- return Arrays.asList(new String[] { event.get("name") });
- }
-}, helloPE);
-</code></pre>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="k">new</span> <span class="n">String</span><span class="o">[]</span> <span class="o">{</span> <span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span> <span class="o">});</span>
+ <span class="o">}</span>
+ <span class="o">},</span> <span class="n">helloPE</span><span class="o">);</span></code></pre>
<p>In this case, a name is extracted from each event, the PE instance with this key is retrieved or created, and the event sent to that instance.</p>
<p>Alternatively, we can use a unique PE instance for processing events in a given node. For that we simply define the input stream without a KeyFinder, <em>and</em> use a singleton PE:</p>
-<pre><code>HelloPE helloPE = createPE(HelloPE.class);
-helloPE.setSingleton(true);
-createInputStream("names", helloPE);
-</code></pre>
-
-<p>In this case, all events will be dispatched to the only HelloPE instance in this partition, regardless of the content of the event.</p></div>
+<pre><code class="language-java"><span class="n">HelloPE</span> <span class="n">helloPE</span> <span class="o">=</span> <span class="n">createPE</span><span class="o">(</span><span class="n">HelloPE</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
+<span class="n">helloPE</span><span class="o">.</span><span class="na">setSingleton</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span>
+<span class="n">createInputStream</span><span class="o">(</span><span class="s">"names"</span><span class="o">,</span> <span class="n">helloPE</span><span class="o">);</span></code></pre>
+
+<p>In this case, all events will be dispatched to the only HelloPE instance in this partition, regardless of the content of the event.</p>
+
+<h1 id="internals-and-tuning">Internals and tuning</h1>
+
+<p>S4 follows a staged event driven architecture and uses a pipeline of executors to process messages. </p>
+
+<h2 id="executors">executors</h2>
+<p>An executor is an object that executes tasks. It usually keeps a bounded queue of task items and schedules their execution through a pool of threads.</p>
+
+<p>When processing queues are full, executors may adopt various possible behaviours, in particular, in S4:</p>
+
+<ul>
+<li>
+<strong>blocking</strong>: the current thread simply waits until the queue is not full</li>
+ <li>
+<strong>shedding</strong>: the current event is dropped</li>
+ <li>
+<strong>throttling</strong>, i.e. placing an upper bound on the processing rate, is a convenient way to avoid sending too many messages too fast.</li>
+</ul><p>S4 provides various default implementations of these behaviours and you can also define your own custom executors as appropriate.</p>
+
+<h2 id="workflow">workflow</h2>
+
+<p>The following picture illustrates the pipeline of executors.</p>
+
+<p><img src="/images/doc/0.6.0/executors.png" alt="image"></p>
+
+<h3 id="when-a-node-receives-a-message">When a node receives a message:</h3>
+
+<ol>
+<li>data is received on a socket and chunked into a message, in the form of an array of bytes</li>
+ <li>the message is passed to a deserializer executor
+ <ul>
+<li>this executor is loaded with the application, and therefore has access to application classes, so that application specific messages can be deserialized</li>
+ <li>by default it uses 1 thread and <strong>blocks</strong> if the processing queue is full</li>
+ </ul>
+</li>
+ <li>the event (the deserialized message) is dispatched to a stream executor
+ <ul>
+<li>the stream executor is selected according to the stream information contained in the event</li>
+ <li>by default it <strong>blocks</strong> if the processing queue is full</li>
+ </ul>
+</li>
+ <li>the event is processed in the PE instance that matches the key of the event</li>
+</ol><h3 id="when-a-pe-emits-a-message">When a PE emits a message:</h3>
+
+<ol>
+<li>an event is passed to a referenced stream</li>
+ <li>if the target cluster is remote, the event is passed to a remote sender executor</li>
+ <li>otherwise,
+ <ul>
+<li>if the target partition is the current one, the event is directly passed to the corresponding stream executor (see step 3 above)</li>
+ <li>otherwise, the event is passed to a local sender executor</li>
+ </ul>
+</li>
+ <li>remote or local sender executors serialize the event into an array of bytes
+ <ul>
+<li>remote sender executors are <strong>blocking</strong> by default, if their processing queue is full</li>
+ <li>local sender executors are <strong>throttling</strong> by default, with a configurable maximum rate. If events arrive at a higher rate, they are <strong>dropped</strong>.</li>
+ </ul>
+</li>
+</ol><h2 id="configuration-parameters">configuration parameters</h2>
+
+<ul>
+<li>blocking executors can lead to deadlocks, depending on the application graph</li>
+ <li>back pressure can be taken into account when using TCP: if downstream systems saturate, messages cannot be sent downstream, and sending queues fill up. With knowledge of the application, is is possible to add some mechanisms to react appropriately</li>
+ <li>executors can be replaced by other implementations in custom modules, by overriding the appropriate bindings (see <code>DefaultCommModule</code> and <code>DefaultCodeModule</code>)</li>
+ <li>the maximum number of threads to use to process a given stream can be specified <strong>in the application</strong>, using the <code>setParallelism()</code> method of <code>Stream</code>
+</li>
+ <li>default parameters are specified in <code>default.s4.comm.properties</code>
+</li>
+</ul></div>
</div>
<div id='footer'>
<div class='container'>
Modified: incubator/s4/site/doc/0.6.0/fault_tolerance/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/fault_tolerance/index.html?rev=1450917&r1=1450916&r2=1450917&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/fault_tolerance/index.html (original)
+++ incubator/s4/site/doc/0.6.0/fault_tolerance/index.html Wed Feb 27 18:59:51 2013
@@ -22,6 +22,8 @@
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
+ <link href='/style/coderay.css' rel='stylesheet' type='text/css' />
+ <link href='/style/pygmentize.css' rel='stylesheet' type='text/css' />
</head>
<body>
<div id='header'>
@@ -59,26 +61,22 @@
<div id='wrapper'>
<div class='container' id='container'><p>Stream processing applications are typically long running applications, and they may accumulate state over extended periods of time.</p>
-<p>Running a distributed system over a long period of time implies there will be:</p>
+<p>When running a distributed system over a long period of time, expect:</p>
<ul>
- <li>failures</li>
+<li>failures</li>
<li>infrastructure updates</li>
<li>scheduled restarts</li>
<li>application updates</li>
-</ul>
-
-<p>In each of these situations, some or all of S4 nodes will be shutdown. The system may therefore be partly unavailable, and in-memory state accumulated during the execution may be lost.</p>
+</ul><p>In each of these situations, some or all of S4 nodes will be shutdown. The system may therefore be partly unavailable, and in-memory state accumulated during the execution may be lost.</p>
<p>In order to deal with this kind of situation, S4 provides:</p>
<ul>
- <li>high availability</li>
+<li>high availability</li>
<li>state recovery (based on checkpointing)</li>
<li>while preserving low processing latency</li>
-</ul>
-
-<p>In this document, we first describe the high availability mechanism implemented in S4, then we describe the checkpointing and recovery mechanism, and how to customize it, then we describe future improvements.</p>
+</ul><p>In this document, we first describe the high availability mechanism implemented in S4, then we describe the checkpointing and recovery mechanism, and how to customize it, then we describe future improvements.</p>
<h1 id="fail-over-mechanism">Fail-over mechanism</h1>
@@ -86,7 +84,7 @@
<p>The following figure illustrates this fail-over mechanism: </p>
-<p><img src="/images/doc/0.6.0/failover.png" alt="image" /></p>
+<p><img src="/images/doc/0.6.0/failover.png" alt="image"></p>
<p>This technique provides high availability but does not prevent state loss.</p>
@@ -113,11 +111,11 @@
<p>For PEs to recover a previous state, the technique we use is to:</p>
<ul>
- <li><em>periodically checkpoint</em> the state of PEs across the S4 cluster</li>
- <li><em>lazily recover</em> (triggered by messages)</li>
-</ul>
-
-<p>This means that if there is a previous state that was checkpointed, and that a new PE is instantiated because a new key is seen, the PE instance will fetch the corresponding checkpoint, recover the corresponding state, and only then start processing events. State loss is minimal!</p>
+<li>
+<em>periodically checkpoint</em> the state of PEs across the S4 cluster</li>
+ <li>
+<em>lazily recover</em> (triggered by messages)</li>
+</ul><p>This means that if there is a previous state that was checkpointed, and that a new PE is instantiated because a new key is seen, the PE instance will fetch the corresponding checkpoint, recover the corresponding state, and only then start processing events. State loss is minimal!</p>
<h3 id="design">Design</h3>
@@ -130,13 +128,11 @@ Asynchronous checkpointing aims at minim
<p>Taking a checkpoint is a 2 steps operations, both handled outside of the event processing path:</p>
<ul>
- <li>serialize the PE instance</li>
+<li>serialize the PE instance</li>
<li>save the serialized PE instance to remote storage</li>
-</ul>
-
-<p>The following figure shows the various components involved: the checkpointing framework handles the serialization and passes serialized state to a pluggable storage backend:</p>
+</ul><p>The following figure shows the various components involved: the checkpointing framework handles the serialization and passes serialized state to a pluggable storage backend:</p>
-<p><img src="/images/doc/0.6.0/checkpointing-framework.png" alt="image" /></p>
+<p><img src="/images/doc/0.6.0/checkpointing-framework.png" alt="image"></p>
<h5 id="recovery">Recovery</h5>
@@ -150,11 +146,9 @@ When a message for a new key arrives in
<p>A PE can be checkpointed if:</p>
<ul>
- <li>the PE class provides an empty no-arg constructor (that restriction should be lifted in next releases)</li>
+<li>the PE class provides an empty no-arg constructor (that restriction should be lifted in next releases)</li>
<li>it has non transient serializable fields (and by opposition, transient fields will never be checkpointed)</li>
-</ul>
-
-<h5 id="checkpointing-application-configuration">Checkpointing application configuration</h5>
+</ul><h5 id="checkpointing-application-configuration">Checkpointing application configuration</h5>
<p>Checkpointing intervals are defined per prototype, in time intervals or event counts (for now). This is specified in the application module, using API methods from the ProcessingElement class, and passing a CheckpointingConfiguration object. Please refer to the API documentation.</p>
@@ -176,55 +170,56 @@ When a message for a new key arrives in
<h5 id="customizing-the-checkpointing-backend">Customizing the checkpointing backend</h5>
-<p>It is quite straightforward to implement backends for other kinds of storage (key value stores, datagrid, cache, RDBMS). Using an alternative backend is as simple as providing a new module to the S4 node. Here is an example of a module using a ‘Cool’ backend implementation:</p>
+<p>It is quite straightforward to implement backends for other kinds of storage (key value stores, datagrid, cache, RDBMS). Writing a checkpointing backend consists of implementing a simple interface (<code>StateStorage</code>) matching your infrastructure or system.</p>
-<pre><code>public class CoolBackendCheckpointingModule extends AbstractModule {
- @Override
- protected void configure() {
- bind(StateStorage.class).to(CoolStateStorage.class);
- bind(CheckpointingFramework.class).to(SafeKeeper.class);
- }
-}
-</code></pre>
+<p>Using an alternative backend is as simple as providing a new module to the S4 node. Here is an example of a module using a âCoolâ backend implementation:</p>
+
+<pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">CoolBackendCheckpointingModule</span> <span class="kd">extends</span> <span class="n">AbstractModule</span> <span class="o">{</span>
+ <span class="nd">@Override</span>
+ <span class="kd">protected</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">()</span> <span class="o">{</span>
+ <span class="n">bind</span><span class="o">(</span><span class="n">StateStorage</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">to</span><span class="o">(</span><span class="n">CoolStateStorage</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
+ <span class="n">bind</span><span class="o">(</span><span class="n">CheckpointingFramework</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">to</span><span class="o">(</span><span class="n">SafeKeeper</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre>
<h5 id="overriding-checkpointing-and-recovery-operations">Overriding checkpointing and recovery operations</h5>
-<p>By default, S4 uses <a href="http://code.google.com/p/kryo">kryo</a> to serialize and deserialize checkpoints, but it is possible to use a different mechanism, by overriding the <code>checkpoint()</code>, <code>serializeState()</code> and <code>restoreState()</code> methods of the <code>ProcessingElement</code> class.</p>
+<p>By default, S4 keeps all non transient fields as part of the state, and uses <a href="http://code.google.com/p/kryo">kryo</a> to serialize and deserialize checkpoints, but it is possible to use a different mechanism, by overriding the <code>checkpoint()</code>, <code>serializeState()</code> and <code>restoreState()</code> methods of the <code>ProcessingElement</code> class.</p>
-<p>PEs are eligible for checkpointing when their state is ‘dirty’. The dirty flag is checked through the <code>isDirty()</code> method, and cleared by calling the <code>clearDirty()</code> method. In some cases, dependent on the application code, only some of the events may actually change the state of the PE. You should override these methods in order to avoid unjustified checkpointing operations.</p>
+<p>PEs are eligible for checkpointing when their state is âdirtyâ. The dirty flag is checked through the <code>isDirty()</code> method, and cleared by calling the <code>clearDirty()</code> method. In some cases, dependent on the application code, only some of the events may actually change the state of the PE. You should override these methods in order to avoid unjustified checkpointing operations.</p>
<h5 id="tuning">Tuning</h5>
<p>The checkpointing framework has a number of overridable parameters, mostly for sizing thread pools:</p>
<ul>
- <li>Serialization thread pool
+<li>Serialization thread pool
<ul>
- <li>s4.checkpointing.serializationMaxThreads (default = 1)</li>
+<li>s4.checkpointing.serializationMaxThreads (default = 1)</li>
<li>s4.checkpointing.serializationThreadKeepAliveSeconds (default = 120)</li>
<li>s4.checkpointing.serializationMaxOutstandingRequests (default = 1000)</li>
</ul>
- </li>
+</li>
<li>Storage backend thread pool
<ul>
- <li>s4.checkpointing.storageMaxThreads (default = 1)</li>
+<li>s4.checkpointing.storageMaxThreads (default = 1)</li>
<li>s4.checkpointing.storageThreadKeepAliveSeconds (default = 120)</li>
<li>s4.checkpointing.storageMaxOutstandingRequests (default = 1000)</li>
</ul>
- </li>
+</li>
<li>Fetching thread pool: fetching is a blocking operation, which can timeout:
<ul>
- <li>s4.checkpointing.fetchingMaxThreads (default = 1)</li>
+<li>s4.checkpointing.fetchingMaxThreads (default = 1)</li>
<li>s4.checkpointing.fetchingThreadKeepAliveSeconds (default = 120)</li>
<li>s4.checkpointing.fetchingMaxWaitMs (default = 1000) (corresponds to the timeout)</li>
</ul>
- </li>
+</li>
<li>In the case the backend is unresponsive, it can be bypassed:
<ul>
- <li>s4.checkpointing.fetchingMaxConsecutiveFailuresBeforeDisabling (default = 10)</li>
+<li>s4.checkpointing.fetchingMaxConsecutiveFailuresBeforeDisabling (default = 10)</li>
<li>s4.checkpointing.fetchingDisabledDurationMs (default = 600000)</li>
</ul>
- </li>
+</li>
</ul></div>
</div>
<div id='footer'>
Modified: incubator/s4/site/doc/0.6.0/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/index.html?rev=1450917&r1=1450916&r2=1450917&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/index.html (original)
+++ incubator/s4/site/doc/0.6.0/index.html Wed Feb 27 18:59:51 2013
@@ -22,6 +22,8 @@
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
+ <link href='/style/coderay.css' rel='stylesheet' type='text/css' />
+ <link href='/style/pygmentize.css' rel='stylesheet' type='text/css' />
</head>
<body>
<div id='header'>
@@ -66,30 +68,31 @@
<h2 id="getting-started">Getting Started</h2>
<ul>
- <li>You may start with an <a href="overview">overview</a> of the platform</li>
+<li>You may start with an <a href="overview">overview</a> of the platform</li>
<li>Then follow a <a href="walkthrough">walkthrough</a> for an hands-on introduction</li>
<li>And <a href="dev_tips">here</a> are some tips to ease the development process</li>
-</ul>
-
-<h2 id="configuration">Configuration</h2>
+</ul><h2 id="configuration">Configuration</h2>
<ul>
- <li>How to <a href="configuration">customize the platform and pass configuration parameters</a></li>
- <li>How to <a href="application_dependencies">add application dependencies</a></li>
- <li>How to <a href="event_dispatch">dispatch events </a> within an application and between applications</li>
-</ul>
-
-<h2 id="features">Features</h2>
+<li><a href="configuration">Customize the platform and pass configuration parameters</a></li>
+ <li>Add <a href="application_dependencies">application dependencies</a>
+</li>
+ <li>
+<a href="event_dispatch">Dispatch events </a> within an application and between applications</li>
+ <li>
+<a href="metrics">Monitor</a> the system</li>
+</ul><h2 id="features">Features</h2>
<ul>
- <li>Details about <a href="fault_tolerance">fault tolerance</a></li>
-</ul>
-
-<h2 id="troubleshooting">Troubleshooting</h2>
+<li>Details about <a href="fault_tolerance">fault tolerance</a>
+</li>
+</ul><h2 id="troubleshooting">Troubleshooting</h2>
<ul>
- <li>Try the <a href="https://cwiki.apache.org/confluence/display/S4/FAQ">FAQ</a></li>
- <li>Try the <a href="https://cwiki.apache.org/S4/s4-apache-mailing-lists.html">mailing lists</a></li>
+<li>Try the <a href="https://cwiki.apache.org/confluence/display/S4/FAQ">FAQ</a>
+</li>
+ <li>Try the <a href="https://cwiki.apache.org/S4/s4-apache-mailing-lists.html">mailing lists</a>
+</li>
</ul></div>
</div>
<div id='footer'>
Added: incubator/s4/site/doc/0.6.0/metrics/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/metrics/index.html?rev=1450917&view=auto
==============================================================================
--- incubator/s4/site/doc/0.6.0/metrics/index.html (added)
+++ incubator/s4/site/doc/0.6.0/metrics/index.html Wed Feb 27 18:59:51 2013
@@ -0,0 +1,107 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html>
+ <head>
+ <title>S4: Metrics</title>
+ <meta content='text/html; charset=utf-8' http-equiv='Content-Type' />
+ <meta content='A general-purpose distributed stream computing platform' name='description' />
+ <link href='/style/screen.css' media='screen' rel='stylesheet' type='text/css' />
+ <link href='/style/print.css' media='print' rel='stylesheet' type='text/css' />
+ <!--[if lt IE 9]>
+ <link href='/style/ie.css' media='screen' rel='stylesheet' type='text/css' />
+ <![endif]-->
+ <link href='/style/style.css' rel='stylesheet' type='text/css' />
+ <link href='/style/nav.css' rel='stylesheet' type='text/css' />
+ <script type='text/javascript'>
+ var _gaq = _gaq || [];
+ _gaq.push(['_setAccount', 'UA-19490961-1']);
+ _gaq.push(['_setDomainName', '.s4.io']);
+ _gaq.push(['_trackPageview']);
+ (function() {
+ var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
+ ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
+ var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
+ })();
+ </script>
+ <link href='/style/coderay.css' rel='stylesheet' type='text/css' />
+ <link href='/style/pygmentize.css' rel='stylesheet' type='text/css' />
+ </head>
+ <body>
+ <div id='header'>
+ <div class='container'>
+ <div id='logo'>
+ <a href='/'>
+ <img src='/images/s4_test.png' />
+ </a>
+ </div>
+ <div id='navbar'><ul id='nav'>
+ <li>
+ <a href='/'>home</a>
+ </li>
+ <li>
+ <a href='https://cwiki.apache.org/confluence/display/S4/S4+Wiki'>doc [0.5]</a>
+ </li>
+ <li>
+ <a href='http://git-wip-us.apache.org/repos/asf?p=incubator-s4.git' onClick="_gaq.push(['_trackEvent', 'External', 'Apache Git', 'http://github.com/s4']);">code</a>
+ </li>
+ <li>
+ <a href='http://people.apache.org/~mmorel/apache-s4-0.5.0-incubating-doc/javadoc/'>API</a>
+ </li>
+ <li>
+ <a href='/contrib'>get involved</a>
+ </li>
+ <li>
+ <a href='/team'>team</a>
+ </li>
+ <li>
+ <a href='/download'>download</a>
+ </li>
+</ul></div>
+ </div>
+ </div>
+ <div id='wrapper'>
+ <div class='container' id='container'><blockquote>
+ <p>S4 continuously collects runtime statistics. Letâs see how to access these and add custom ones.</p>
+</blockquote>
+
+<h1 id="why">Why?</h1>
+
+<p>S4 aims at processing large quantities of events with low latency. In order to achieve this goal, a key requirement is to be able to monitor system internals at runtime.</p>
+
+<h1 id="how">How?</h1>
+<p>For that purpose, we include a system for gathering statistics about various parts of the S4 platform.</p>
+
+<p>We rely on the <a href="http://metrics.codahale.com">metrics</a> library, which offers an efficient way to gather such information and relies on statistical techniques to minimize memory consumption.</p>
+
+<h1 id="what">What?</h1>
+
+<p>By default, S4 instruments queues, caches, checkpointing, event reception and emission and statistics are available for all of these components.</p>
+
+<p>You can also monitor your own PEs. Simply add new probes (<code>Meter</code>, <code>Gauge</code>, etc..) and report interesting updates to them. There is nothing else to do, these custom metrics will be reported along with the S4 metrics, as explained next.</p>
+
+<h1 id="where">Where?</h1>
+
+<p>By default, metrics are exposed by each node through JMX.</p>
+
+<p>The <code>s4.metrics.config</code> parameter enables periodic dumps of aggregated statistics to the <strong>console</strong> or to <strong>files</strong> in csv format. This parameter is specified as an application parameter, and must match the following regular expression: </p>
+
+<pre><code>(csv:.+|console):(\d+):(DAYS|HOURS|MICROSECONDS|MILLISECONDS|MINUTES|NANOSECONDS|SECONDS)
+</code></pre>
+
+<p>Examples:</p>
+
+<pre><code># dump metrics to csv files to /path/to/directory every 10 seconds
+csv:file://path/to/directory:10:SECONDS
+
+# dump metrics to the console every minute
+console:1:MINUTES
+</code></pre>
+
+<p>Reporting to Ganglia or Graphite is not provided out of the box with S4, but itâs quite easy to add. You simply have to add the corresponding dependencies to your project and enable reporting to these systems during the initialization of your application. See the <a href="http://metrics.codahale.com">metrics</a> documentation for more information.</p></div>
+ </div>
+ <div id='footer'>
+ <div class='container'>
+ <span class='copyright'>Apache S4 - Copyright 2013 The Apache Software Foundation</span>
+ </div>
+ </div>
+ </body>
+</html>
Modified: incubator/s4/site/doc/0.6.0/overview/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/overview/index.html?rev=1450917&r1=1450916&r2=1450917&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/overview/index.html (original)
+++ incubator/s4/site/doc/0.6.0/overview/index.html Wed Feb 27 18:59:51 2013
@@ -22,6 +22,8 @@
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
+ <link href='/style/coderay.css' rel='stylesheet' type='text/css' />
+ <link href='/style/pygmentize.css' rel='stylesheet' type='text/css' />
</head>
<body>
<div id='header'>
@@ -66,48 +68,47 @@
<p>S4 0.6 builds on this basis and brings plenty of exciting features, in particular:</p>
<ul>
- <li><em>performance improvements</em>: stream throughput improved by 1000 % (~200k messages / s / stream)</li>
+<li>
+<strong>performance improvements</strong>: stream throughput improved by 1000 % (~200k messages / s / stream)</li>
<li>improved [configurability](S4:Configuration - 0.6.0], for both the S4 platform and deployed applications</li>
- <li><em>elasticity</em> and fine partition tuning, through an integration with Apache Helix</li>
-</ul>
-
-<h1 id="what-are-the-cool-features">What are the cool features?</h1>
+ <li>
+<strong>elasticity</strong> and fine partition tuning, through an integration with Apache Helix</li>
+</ul><h1 id="what-are-the-cool-features">What are the cool features?</h1>
<p><strong>Flexible deployment</strong>:</p>
<ul>
+<li>Application packages are standard jar files (suffixed <code>.s4r</code>)</li>
<li>By default keys are homogeneously sparsed over the cluster: helps balance the load, especially for fine grained partitioning</li>
- <li>S4 also provides fine control over the partitioning</li>
- <li>Features automatic rebalancing</li>
-</ul>
-
-<p><strong>Modular design</strong>:</p>
+ <li>S4 also provides fine control over the partitioning (with Apache Helix)</li>
+ <li>Semi-automatic Rebalancing</li>
+</ul><p><strong>Modular design</strong>:</p>
<ul>
- <li>both the platform and the applications are built by dependency injection, and configured through independent modules.</li>
- <li>makes it easy to customize the system according to specific requirements</li>
- <li>pluggable event serving policies: load shedding, throttling, blocking</li>
-</ul>
-
-<p><strong>Dynamic and loose coupling of S4 applications</strong>:</p>
+<li>both the platform and the applications are built by dependency injection, and configured through independent modules.</li>
+ <li>makes it <strong>easy to customize</strong> the system according to specific requirements</li>
+ <li>pluggable event serving policies: <strong>load shedding, throttling, blocking</strong>
+</li>
+</ul><p><strong>Dynamic and loose coupling of S4 applications</strong>:</p>
<ul>
- <li>through a pub-sub mechanism</li>
+<li>through a pub-sub mechanism</li>
<li>makes it easy to:
-** assemble subsystems into larger systems
-** reuse applications
-** separate pre-processing
-** provision, control and update subsystems independently</li>
-</ul>
-
-<p><strong><a href="fault_tolerance">Fault tolerant</a></strong></p>
+ <ul>
+<li>assemble subsystems into larger systems</li>
+ <li>reuse applications</li>
+ <li>separate pre-processing</li>
+ <li>provision, control and update subsystems independently</li>
+ </ul>
+</li>
+</ul><p><strong><a href="fault_tolerance">Fault tolerant</a></strong></p>
<ul>
- <li><em>Fail-over</em> mechanism for high availability</li>
- <li><em>Checkpointing and recovery</em> mechanism for minimizing state loss</li>
-</ul>
-
-<p><strong>Pure Java</strong>: statically typed, easy to understand, to refactor, and to extend</p>
+<li>
+<strong>Fail-over</strong> mechanism for high availability</li>
+ <li>
+<strong>Checkpointing and recovery</strong> mechanism for minimizing state loss</li>
+</ul><p><strong>Pure Java</strong>: statically typed, easy to understand, to refactor, and to extend</p>
<h1 id="how-does-it-work">How does it work?</h1>
@@ -116,51 +117,59 @@
<p><strong>Platform</strong></p>
<ul>
- <li>S4 provides a runtime distributed platform that handles communication, scheduling and distribution across containers.</li>
- <li>Distributed containers are called <em>S4 nodes</em></li>
- <li>S4 nodes are deployed on <em>S4 clusters</em></li>
- <li>S4 clusters define named ensembles of S4 nodes, with a fixed size</li>
- <li>The size of an S4 cluster corresponds to the number of logical <em>partitions</em> (sometimes referred to as <em>tasks</em>)</li>
-</ul>
-
-<p><strong>Applications</strong></p>
+<li>S4 provides a runtime distributed platform that handles communication, scheduling and distribution across containers.</li>
+ <li>Distributed containers are called <strong>S4 nodes</strong>
+</li>
+ <li>S4 nodes are deployed on <strong>S4 clusters</strong>
+</li>
+ <li>S4 clusters define named ensembles of S4 nodes
+ <ul>
+<li>by default, the size of the cluster is fixed</li>
+ <li>the size of an S4 cluster corresponds to the number of logical <strong>partitions</strong> (sometimes referred to as <strong>tasks</strong>)</li>
+ <li>an ongoing integration with <a href="http://helix.apache.org">Apache Helix</a> removes these limitations and allows a variable number of nodes and a rebalancing of partitions</li>
+ </ul>
+</li>
+</ul><p><strong>Applications</strong></p>
+
+<ul>
+<li>Users develop applications and deploy them on S4 clusters</li>
+ <li>Applications are built as a graph of:
+ <ul>
+<li>
+<strong>Processing elements</strong> (PEs)</li>
+ <li>
+<strong>Streams</strong> that interconnect PEs</li>
+ </ul>
+</li>
+ <li>PEs communicate asynchronously by sending <strong>events</strong> on streams.</li>
+ <li>Events are dispatched to nodes according to their <strong>key</strong>
+</li>
+</ul><p><strong>External streams</strong> are a special kind of stream that:</p>
<ul>
- <li>Users develop applications and deploy them on S4 clusters</li>
- <li>
- <p>Applications are built from:
-** <em>Processing elements</em> (PEs)
-** <em>Streams</em> that interconnect PEs</p>
- </li>
- <li>PEs communicate asynchronously by sending <em>events</em> on streams.</li>
- <li>Events are dispatched to nodes according to their key</li>
-</ul>
-
-<p><strong>External streams</strong> are a special kind of stream that:</p>
-
-<ul>
- <li>send events outside of the application</li>
+<li>send events outside of the application</li>
<li>receive events from external sources</li>
<li>can interconnect and assemble applications into larger systems.</li>
-</ul>
-
-<p><strong>Adapters</strong> are S4 applications that can convert external streams into streams of S4 events. Since adapters are also S4 applications, they can be scaled easily.</p>
+</ul><p><strong>Adapters</strong> are S4 applications that can convert external streams into streams of S4 events. Since adapters are also S4 applications, they can be scaled easily.</p>
<h2 id="a-hierarchical-perspective-on-s4">A hierarchical perspective on S4</h2>
<p>The following diagram sums-up the key concepts in a hierarchical fashion:</p>
-<p><img src="/images/doc/0.6.0/S4_hierarchical_archi.png" alt="image" /></p>
+<p><img src="/images/doc/0.6.0/S4_hierarchical_archi.png" alt="image"></p>
<h1 id="where-can-i-find-more-information">Where can I find more information?</h1>
<ul>
- <li><a href="http://incubator.apache.org/s4/">The website</a> is a good starting point.</li>
- <li><a href="https://cwiki.apache.org/confluence/display/S4/">The wiki</a> currently contains the most up-to-date information: general information (this page), configuration, examples.</li>
- <li>Questions can be asked through the <a href="https://cwiki.apache.org/confluence/display/S4/S4+Apache+mailing+lists">mailing lists</a></li>
+<li>
+<a href="http://incubator.apache.org/s4/">The website</a> is a good starting point.</li>
+ <li>
+<a href="https://cwiki.apache.org/confluence/display/S4/">The wiki</a> currently contains the most up-to-date information: general information (this page), configuration, examples.</li>
+ <li>Questions can be asked through the <a href="https://cwiki.apache.org/confluence/display/S4/S4+Apache+mailing+lists">mailing lists</a>
+</li>
<li>The source code is available throught [git](https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git], <a href="http://incubator.apache.org/s4/contrib/">here</a> are instructions for fetching the code.</li>
<li>A nice set of <a href="http://www.slideshare.net/leoneu/20111104-s4-overview">slides</a> was used for a presentation at Stanford in November 2011.</li>
- <li>The driving ideas are detailed in a <a href="http://www.4lunas.org/pub/2010-s4.pdf">conference publication</a> from KDCloud’11 (joint workshop with ICDM’11)</li>
+ <li>The driving ideas are detailed in a <a href="http://www.4lunas.org/pub/2010-s4.pdf">conference publication</a> from KDCloudâ11 (joint workshop with ICDMâ11)</li>
</ul></div>
</div>
<div id='footer'>
Modified: incubator/s4/site/doc/0.6.0/walkthrough/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/walkthrough/index.html?rev=1450917&r1=1450916&r2=1450917&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/walkthrough/index.html (original)
+++ incubator/s4/site/doc/0.6.0/walkthrough/index.html Wed Feb 27 18:59:51 2013
@@ -22,6 +22,8 @@
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
+ <link href='/style/coderay.css' rel='stylesheet' type='text/css' />
+ <link href='/style/pygmentize.css' rel='stylesheet' type='text/css' />
</head>
<body>
<div id='header'>
@@ -66,21 +68,18 @@
<p>There are 2 ways:</p>
<ul>
- <li><a href="http://incubator.apache.org/s4/download/">Download</a> the 0.6.0 release </li>
-</ul>
-
-<blockquote>
- <p>We recommend getting the “source” release and building it</p>
+<li>
+<a href="http://incubator.apache.org/s4/download/">Download</a> the 0.6.0 release </li>
+</ul><blockquote>
+ <p>We recommend getting the âsourceâ release and building it</p>
</blockquote>
<ul>
- <li>or checkout from the Apache git repository, by following the <a href="/contrib">instructions</a>. The 0.6.0 tag corresponds to the current release.</li>
-</ul>
-
-<p>If you get the binary release, s4 scripts are immediately available. Otherwise you must build the project:</p>
+<li>or checkout from the Apache git repository, by following the <a href="/contrib">instructions</a>. The 0.6.0 tag corresponds to the current release.</li>
+</ul><p>If you get the binary release, s4 scripts are immediately available. Otherwise you must build the project:</p>
<ul>
- <li>
+<li>
<p>Compile and install S4 in the local maven repository: (you can also let the tests run without the -DskipTests option)</p>
<pre><code> S4:incubator-s4$ ./gradlew install -DskipTests
@@ -95,18 +94,14 @@
...:s4-tools:installApp
</code></pre>
</li>
-</ul>
+</ul><hr><h1 id="start-a-new-application">Start a new application</h1>
-<hr />
-
-<h1 id="start-a-new-application">Start a new application</h1>
-
-<p>S4 provides some scripts in order to simplify development and testing of applications. Let’s see how to create a new project and start a sample application.</p>
+<p>S4 provides some scripts in order to simplify development and testing of applications. Letâs see how to create a new project and start a sample application.</p>
<h2 id="create-a-new-project">Create a new project</h2>
<ul>
- <li>
+<li>
<p>Create a new application template (here, we create it in the /tmp directory):</p>
<pre><code> S4:incubator-s4$ ./s4 newApp myApp -parentDir=/tmp
@@ -122,80 +117,69 @@
src/ --> sources (maven-like structure)
</code></pre>
</li>
-</ul>
-
-<h2 id="have-a-look-at-the-sample-project-content">Have a look at the sample project content</h2>
+</ul><h2 id="have-a-look-at-the-sample-project-content">Have a look at the sample project content</h2>
<p>The src/main/java/hello directory contains 3 files:</p>
<ul>
- <li>
- <p>HelloPE.java : a very simple PE that simply prints the name contained in incoming events
- // ProcessingElement provides integration with the S4 platform
- public class HelloPE extends ProcessingElement {</p>
-
- <pre><code> // you should define downstream streams here and inject them in the app definition
+<li>HelloPE.java : a very simple PE that simply prints the name contained in incoming events</li>
+</ul><pre><code class="language-java"><span class="c1">// ProcessingElement provides integration with the S4 platform</span>
+<span class="kd">public</span> <span class="kd">class</span> <span class="nc">HelloPE</span> <span class="kd">extends</span> <span class="n">ProcessingElement</span> <span class="o">{</span>
+ <span class="c1">// you should define downstream streams here and inject them in the app definition</span>
+
+ <span class="c1">// PEs can maintain some state</span>
+ <span class="kt">boolean</span> <span class="n">seen</span> <span class="o">=</span> <span class="kc">false</span><span class="o">;</span>
+
+ <span class="c1">// This method is called upon a new Event on an incoming stream.</span>
+ <span class="c1">// You may overload it for handling instances of your own specialized subclasses of Event</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onEvent</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Hello "</span> <span class="o">+</span> <span class="o">(</span><span class="n">seen</span> <span class="o">?</span> <span class="s">"again "</span> <span class="o">:</span> <span class="s">""</span><span class="o">)</span> <span class="o">+</span> <span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span> <span class="o">+</span> <span class="s">"!"</span><span class="o">);</span>
+ <span class="n">seen</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="c1">// skipped remaining methods</span></code></pre>
- // PEs can maintain some state
- boolean seen = false;
-
- // This method is called upon a new Event on an incoming stream.
- // You may overload it for handling instances of your own specialized subclasses of Event
- public void onEvent(Event event) {
- System.out.println("Hello " + (seen ? "again " : "") + event.get("name") + "!");
- seen = true;
- }
- // skipped remaining methods
-</code></pre>
- </li>
- <li>
- <p>HelloApp.java: defines a simple application: exposes an input stream (“names”), connected to the HelloPE. See <a href="event_dispatch">the event dispatch configuration page</a> for more information about how events are dispatched.
+<ul>
+<li>HelloApp.java: defines a simple application: exposes an input stream (ânamesâ), connected to the HelloPE. See <a href="event_dispatch">the event dispatch configuration page</a> for more information about how events are dispatched.
// App parent class provides integration with the S4 platform
- public class HelloApp extends App {</p>
-
- <pre><code> @Override
- protected void onStart() {
- }
-
- @Override
- protected void onInit() {
- // That's where we define PEs and streams
- // create a prototype
- HelloPE helloPE = createPE(HelloPE.class);
- // Create a stream that listens to the "lines" stream and passes events to the helloPE instance.
- createInputStream("names", new KeyFinder<Event>() {
- // the KeyFinder is used to identify keys
- @Override
- public List<String> get(Event event) {
- return Arrays.asList(new String[] { event.get("name") });
- }
- }, helloPE);
- } // skipped remaining methods
-</code></pre>
- </li>
- <li>
- <p>HelloInputAdapter is a simple adapter that reads character lines from a socket, converts them into events, and sends the events to interested S4 apps, through the “names” stream</p>
- </li>
-</ul>
+ public class HelloApp extends App {</li>
+</ul><pre><code class="language-java"><span class="nd">@Override</span>
+<span class="kd">protected</span> <span class="kt">void</span> <span class="nf">onStart</span><span class="o">()</span> <span class="o">{</span>
+<span class="o">}</span>
+
+<span class="nd">@Override</span>
+<span class="kd">protected</span> <span class="kt">void</span> <span class="nf">onInit</span><span class="o">()</span> <span class="o">{</span>
+ <span class="c1">// That's where we define PEs and streams</span>
+ <span class="c1">// create a prototype</span>
+ <span class="n">HelloPE</span> <span class="n">helloPE</span> <span class="o">=</span> <span class="n">createPE</span><span class="o">(</span><span class="n">HelloPE</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
+ <span class="c1">// Create a stream that listens to the "lines" stream and passes events to the helloPE instance.</span>
+ <span class="n">createInputStream</span><span class="o">(</span><span class="s">"names"</span><span class="o">,</span> <span class="k">new</span> <span class="n">KeyFinder</span><span class="o"><</span><span class="n">Event</span><span class="o">>()</span> <span class="o">{</span>
+ <span class="c1">// the KeyFinder is used to identify keys</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="nf">get</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="k">new</span> <span class="n">String</span><span class="o">[]</span> <span class="o">{</span> <span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span> <span class="o">});</span>
+ <span class="o">}</span>
+ <span class="o">},</span> <span class="n">helloPE</span><span class="o">);</span>
+<span class="o">}</span>
+<span class="c1">// skipped remaining methods</span></code></pre>
-<h2 id="run-the-sample-app">Run the sample app</h2>
+<ul>
+<li>HelloInputAdapter is a simple adapter that reads character lines from a socket, converts them into events, and sends the events to interested S4 apps, through the ânamesâ stream</li>
+</ul><h2 id="run-the-sample-app">Run the sample app</h2>
<p>In order to run an S4 application, you need :</p>
<ul>
- <li>to set-up a cluster: provision a cluster and start S4 nodes for that cluster</li>
+<li>to set-up a cluster: provision a cluster and start S4 nodes for that cluster</li>
<li>to package the app</li>
<li>to publish the app on the cluster</li>
-</ul>
-
-<h1 id="set-up-the-cluster">Set-up the cluster:</h1>
+</ul><h1 id="set-up-the-cluster">Set-up the cluster:</h1>
<ul>
- <li>
+<li>
<p>In 2 steps:</p>
<ol>
- <li>
+<li>
<p>Start a Zookeeper server instance (-clean option removes previous ZooKeeper data, if any):</p>
<pre><code> S4:incubator-s4$ ./s4 zkServer - clean
@@ -204,44 +188,46 @@
[main] INFO org.apache.s4.tools.ZKServer - cleaning existing data in [/var/folders/8V/8VdgKWU3HCiy2yV4dzFpDk+++TI/-Tmp-/tmp/zookeeper/data] and [/var/folders/8V/8VdgKWU3HCiy2yV4dzFpDk+++TI/-Tmp-/tmp/zookeeper/log]
</code></pre>
</li>
- <li>
- <p>Define a new cluster. Say a cluster named “cluster1” with 2 partitions, nodes listening to ports starting from 12000:</p>
+ </ol>
+</li>
+</ul><ol>
+<li>
+ <p>Define a new cluster. Say a cluster named âcluster1â with 2 partitions, nodes listening to ports starting from 12000:</p>
- <pre><code> S4:myApp$ ./s4 newCluster -c=cluster1 -nbTasks=2 -flp=12000
+ <pre><code> S4:myApp$ ./s4 newCluster -c=cluster1 -nbTasks=2 -flp=12000
calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
[main] INFO org.apache.s4.tools.DefineCluster - preparing new cluster [cluster1] with [2] node(s)
[main] INFO org.apache.s4.tools.DefineCluster - New cluster configuration uploaded into zookeeper
</code></pre>
- </li>
- </ol>
</li>
- <li>
- <p>Alternatively you may combine these two steps into a single one, by passing the cluster configuration inline with the <code>zkServer</code> command:</p>
+</ol><ul>
+<li>Alternatively you may combine these two steps into a single one, by passing the cluster configuration inline with the <code>zkServer</code> command:</li>
+</ul><pre><code class="language-bash">S4:incubator-s4<span class="nv">$ </span>./s4 zkServer -clusters<span class="o">=</span><span class="nv">c</span><span class="o">=</span>cluster1:flp<span class="o">=</span>12000:nbTasks<span class="o">=</span>2 -clean</code></pre>
- <pre><code> S4:incubator-s4$ ./s4 zkServer -clusters=c=cluster1:flp=12000:nbTasks=2 -clean
-</code></pre>
- </li>
- <li>
- <p>Start 2 S4 nodes with the default configuration, and attach them to cluster “cluster1” :</p>
+<ul>
+<li>Start 2 S4 nodes with the default configuration, and attach them to cluster âcluster1â :</li>
+</ul><pre><code class="language-bash">S4:myApp<span class="nv">$ </span>./s4 node -c<span class="o">=</span>cluster1
+calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
+15:50:18.996 <span class="o">[</span>main<span class="o">]</span> INFO org.apache.s4.core.Main - Initializing S4 node with :
+- comm module class <span class="o">[</span>org.apache.s4.comm.DefaultCommModule<span class="o">]</span>
+- comm configuration file <span class="o">[</span>default.s4.comm.properties from classpath<span class="o">]</span>
+- core module class <span class="o">[</span>org.apache.s4.core.DefaultCoreModule<span class="o">]</span>
+- core configuration file<span class="o">[</span>default.s4.core.properties from classpath<span class="o">]</span>
+-extra modules: <span class="o">[]</span>
+<span class="o">[</span>main<span class="o">]</span> INFO org.apache.s4.core.Main - Starting S4 node.</code></pre>
- <pre><code> S4:myApp$ ./s4 node -c=cluster1
- calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
- 15:50:18.996 [main] INFO org.apache.s4.core.Main - Initializing S4 node with :
- - comm module class [org.apache.s4.comm.DefaultCommModule]
- - comm configuration file [default.s4.comm.properties from classpath]
- - core module class [org.apache.s4.core.DefaultCoreModule]
- - core configuration file[default.s4.core.properties from classpath]
- -extra modules: []
- [main] INFO org.apache.s4.core.Main - Starting S4 node. This node will automatically download applications published for the cluster it belongs to and again (maybe in another shell):
-
- S4:myApp$ ./s4 node -c=cluster1
+<p>This node will automatically download applications published for the cluster it belongs to
+and again (maybe in another shell):</p>
+
+<pre><code>S4:myApp$ ./s4 node -c=cluster1
</code></pre>
- </li>
- <li>Build, package and publish the app to cluster1:
+
+<ul>
+<li>Build, package and publish the app to cluster1:
<ul>
- <li>This is done in 2 separate steps:
+<li>This is done in 2 separate steps:
<ol>
- <li>
+<li>
<p>Create an s4r archive. The following creates an archive named myApp.s4r (here you may specify an arbitrary name) in build/libs.
Again specifying the app class is optional : </p>
@@ -255,9 +241,9 @@ Again specifying the app class is option
</code></pre>
</li>
</ol>
- </li>
+</li>
</ul>
- </li>
+</li>
<li>
<p>S4 nodes will detect the new application, download it, load it and start it. You will get something like:</p>
@@ -281,29 +267,25 @@ Again specifying the app class is option
[ZkClient-EventThread-15-localhost:2181] INFO org.apache.s4.core.App - Init prototype [hello.HelloPE].
</code></pre>
</li>
-</ul>
-
-<p>Great! The application is now deployed on 2 S4 nodes.</p>
+</ul><p>Great! The application is now deployed on 2 S4 nodes.</p>
-<p>You can check the status of the application, nodes and streams with the “status” command:</p>
+<p>You can check the status of the application, nodes and streams with the âstatusâ command:</p>
<pre><code>./s4 status
</code></pre>
<p>Now what we need is some input!</p>
-<p>We can get input through an adapter, i.e. an S4 app that converts an external stream into S4 events, and injects the events into S4 clusters. In the sample application, the adapter is a very basic class, that extends App, listens to an input socket on port 15000, and converts each received line of characters into a generic S4 event, in which the line data is kept in a “name” field. We specify :</p>
+<p>We can get input through an adapter, i.e. an S4 app that converts an external stream into S4 events, and injects the events into S4 clusters. In the sample application, the adapter is a very basic class, that extends App, listens to an input socket on port 15000, and converts each received line of characters into a generic S4 event, in which the line data is kept in a ânameâ field. We specify :</p>
<ul>
- <li>the adapter class</li>
+<li>the adapter class</li>
<li>the name of the output stream</li>
<li>the cluster where to deploy this app</li>
-</ul>
-
-<p>For easy testing, we provide a facility to start a node with an adapter app without having to package the adapter app.</p>
+</ul><p>For easy testing, we provide a facility to start a node with an adapter app without having to package the adapter app.</p>
<ul>
- <li>
+<li>
<p>First, we need to define a new S4 subcluster for that app:</p>
<pre><code> S4:myApp$ ./s4 newCluster -c=cluster2 -nbTasks=1 -flp=13000
@@ -311,16 +293,16 @@ Again specifying the app class is option
</li>
<li>Then we configure the application:
<ul>
- <li>we specify the adapter class (app class)
- * we use “names” for identifying the output stream (this is the same name used as input by the myApp app)</li>
+<li>we specify the adapter class (app class)
+ * we use ânamesâ for identifying the output stream (this is the same name used as input by the myApp app)</li>
<li>
- <p><em>there is also a -s4r parameter, indicating where to fetch the application package from. We don’t need it here, since we skip that step and use a special “adapter” tool</em></p>
+ <p><em>there is also a -s4r parameter, indicating where to fetch the application package from. We donât need it here, since we skip that step and use a special âadapterâ tool</em></p>
<pre><code> ./s4 deploy -appClass=hello.HelloInputAdapter -p=s4.adapter.output.stream=names -c=cluster2 -appName=adapter
</code></pre>
</li>
</ul>
- </li>
+</li>
<li>
<p>Then we simply start the adapter (there is no packaging and copying of the S4R package)
> The adapter command must be run from the root of your S4 project (myApp dir in our case).</p>
@@ -329,7 +311,7 @@ Again specifying the app class is option
</code></pre>
</li>
<li>
- <p>Now let’s just provide some data to the external stream (our adapter is listening to port 15000):</p>
+ <p>Now letâs just provide some data to the external stream (our adapter is listening to port 15000):</p>
<pre><code> S4:~$ echo "Bob" | nc localhost 15000
</code></pre>
@@ -340,53 +322,45 @@ Again specifying the app class is option
<pre><code> Hello Bob!
</code></pre>
</li>
-</ul>
-
-<blockquote>
- <p>If you keep sending messages, nodes will alternatively display the “hello” messages because the adapter app sends keyless events on the “names” stream in a round-robin fashion by default.</p>
+</ul><blockquote>
+ <p>If you keep sending messages, nodes will alternatively display the âhelloâ messages because the adapter app sends keyless events on the ânamesâ stream in a round-robin fashion by default.</p>
</blockquote>
<h2 id="what-happened">What happened?</h2>
<p>The following figures illustrate the various steps we have taken. The local file system is used as the S4 application repository in our example.</p>
-<p><img src="/images/doc/0.6.0/sampleAppDeployment.png" alt="image" /></p>
+<p><img src="/images/doc/0.6.0/sampleAppDeployment.png" alt="image"></p>
-<hr />
+<hr><h1 id="run-the-twitter-trending-example">Run the Twitter trending example</h1>
-<h1 id="run-the-twitter-trending-example">Run the Twitter trending example</h1>
-
-<p>Let’s have a look at another application, that computes trendy Twitter topics by listening to the spritzer stream from the Twitter API. This application was adapted from a previous example in S4 0.3.</p>
+<p>Letâs have a look at another application, that computes trendy Twitter topics by listening to the spritzer stream from the Twitter API. This application was adapted from a previous example in S4 0.3.</p>
<h2 id="overview">Overview</h2>
<p>This application is divided into:</p>
<ul>
- <li>twitter-counter , in test-apps/twitter-counter/ : extracts topics from tweets and maintains a count of the most popular ones, periodically dumped to disk</li>
- <li>twitter-adapter, in test-apps/twitter-adapter/ : listens to the feed from Twitter, converts status text into S4 events, and passes them to the “RawStatus” stream</li>
-</ul>
-
-<p>Have a look at the code in these directories. You’ll note that:</p>
+<li>twitter-counter , in test-apps/twitter-counter/ : extracts topics from tweets and maintains a count of the most popular ones, periodically dumped to disk</li>
+ <li>twitter-adapter, in test-apps/twitter-adapter/ : listens to the feed from Twitter, converts status text into S4 events, and passes them to the âRawStatusâ stream</li>
+</ul><p>Have a look at the code in these directories. Youâll note that:</p>
<ul>
- <li>the build.gradle file must be tailored to include new dependencies (twitter4j libs in twitter-adapter)</li>
+<li>the build.gradle file must be tailored to include new dependencies (twitter4j libs in twitter-adapter)</li>
<li>events are partitioned through various keys</li>
-</ul>
-
-<h2 id="run-it">Run it!</h2>
+</ul><h2 id="run-it">Run it!</h2>
<blockquote>
<p>Note: You need a twitter4j.properties file in your home directory with the following content (debug is optional):</p>
</blockquote>
-<pre><code> debug=true
- user=<a twitter username>
- password=<matching password>
+<pre><code>debug=true
+user=<a twitter username>
+password=<matching password>
</code></pre>
<ul>
- <li>
+<li>
<p>Start a Zookeeper instance. From the S4 base directory, do:</p>
<pre><code> ./s4 zkServer
@@ -418,7 +392,7 @@ Again specifying the app class is option
</code></pre>
</li>
<li>
- <p>Deploy twitter-adapter app. In this example, we don’t directly specify the app class of the adapter, we use the deployment approach for apps (remember, the adapter is also an app).</p>
+ <p>Deploy twitter-adapter app. In this example, we donât directly specify the app class of the adapter, we use the deployment approach for apps (remember, the adapter is also an app).</p>
<pre><code> ./s4 deploy -appName=twitter-adapter -c=cluster2 -b=`pwd`/test-apps/twitter-adapter/build.gradle
</code></pre>
@@ -435,23 +409,19 @@ Again specifying the app class is option
<pre><code> ./s4 status
</code></pre>
</li>
-</ul>
-
-<hr />
-
-<h1 id="what-next">What next?</h1>
+</ul><hr><h1 id="what-next">What next?</h1>
<p>You have now seen some basics applications, and you know how to run them, and how to get events into the system. You may now try to code your own apps with your own data.</p>
<p><a href="../application_dependencies">This page</a> will help for specifying your own dependencies.</p>
-<p>There are more parameters available for the scripts (typing the name of the task will list the options). In particular, if you want distributed deployments, you’ll need to pass the Zookeeper connection strings when you start the nodes.</p>
+<p>There are more parameters available for the scripts (typing the name of the task will list the options). In particular, if you want distributed deployments, youâll need to pass the Zookeeper connection strings when you start the nodes.</p>
<p>You may also customize the communication and the core layers of S4 by tweaking configuration files and modules.</p>
<p>Last, the <a href="http://people.apache.org/~mmorel/apache-s4-0.6.0-incubating-doc/javadoc/">javadoc</a> will help you when writing applications.</p>
-<p>We hope this will help you start rapidly, and remember: we’re happy to help!</p></div>
+<p>We hope this will help you start rapidly, and remember: weâre happy to help!</p></div>
</div>
<div id='footer'>
<div class='container'>
Modified: incubator/s4/site/download/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/download/index.html?rev=1450917&r1=1450916&r2=1450917&view=diff
==============================================================================
--- incubator/s4/site/download/index.html (original)
+++ incubator/s4/site/download/index.html Wed Feb 27 18:59:51 2013
@@ -20,7 +20,8 @@
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
- </script>
+ </script><link href="../style/coderay.css" rel="stylesheet" type="text/css">
+<link href="../style/pygmentize.css" rel="stylesheet" type="text/css">
</head>
<body>
<div id="header">
Added: incubator/s4/site/images/doc/0.6.0/executors.png
URL: http://svn.apache.org/viewvc/incubator/s4/site/images/doc/0.6.0/executors.png?rev=1450917&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/s4/site/images/doc/0.6.0/executors.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/s4/site/images/doc/0.6.0/sources/executors.odg
URL: http://svn.apache.org/viewvc/incubator/s4/site/images/doc/0.6.0/sources/executors.odg?rev=1450917&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/s4/site/images/doc/0.6.0/sources/executors.odg
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/s4/site/images/doc/0.6.0/sources/queues.odg
URL: http://svn.apache.org/viewvc/incubator/s4/site/images/doc/0.6.0/sources/queues.odg?rev=1450917&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/s4/site/images/doc/0.6.0/sources/queues.odg
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: incubator/s4/site/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/index.html?rev=1450917&r1=1450916&r2=1450917&view=diff
==============================================================================
--- incubator/s4/site/index.html (original)
+++ incubator/s4/site/index.html Wed Feb 27 18:59:51 2013
@@ -20,7 +20,8 @@
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
- </script>
+ </script><link href="style/coderay.css" rel="stylesheet" type="text/css">
+<link href="style/pygmentize.css" rel="stylesheet" type="text/css">
</head>
<body>
<div id="header">