You are viewing a plain text version of this content. The canonical link for it is here.
Posted to s4-commits@incubator.apache.org by mm...@apache.org on 2013/03/10 20:11:54 UTC
svn commit: r1454912 - in /incubator/s4/site/doc/0.6.0:
application_dependencies/index.html event_injection/
event_injection/index.html index.html overview/index.html
walkthrough/index.html
Author: mmorel
Date: Sun Mar 10 20:11:54 2013
New Revision: 1454912
URL: http://svn.apache.org/r1454912
Log:
doc updates for 0.6.0 release
Added:
incubator/s4/site/doc/0.6.0/event_injection/
incubator/s4/site/doc/0.6.0/event_injection/index.html
Modified:
incubator/s4/site/doc/0.6.0/application_dependencies/index.html
incubator/s4/site/doc/0.6.0/index.html
incubator/s4/site/doc/0.6.0/overview/index.html
incubator/s4/site/doc/0.6.0/walkthrough/index.html
Modified: incubator/s4/site/doc/0.6.0/application_dependencies/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/application_dependencies/index.html?rev=1454912&r1=1454911&r2=1454912&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/application_dependencies/index.html (original)
+++ incubator/s4/site/doc/0.6.0/application_dependencies/index.html Sun Mar 10 20:11:54 2013
@@ -74,9 +74,9 @@
<pre><code> project.ext["libraries"] = [
      twitter4j_core:   'org.twitter4j:twitter4j-core:2.2.5',
      twitter4j_stream:  'org.twitter4j:twitter4j-stream:2.2.5',
- Â Â Â Â Â Â s4_base:Â Â Â Â Â Â 'org.apache.s4:s4-base:0.5.0',
- Â Â Â Â Â Â s4_comm:Â Â Â Â Â Â 'org.apache.s4:s4-comm:0.5.0',
- Â Â Â Â Â Â s4_core:Â Â Â Â Â Â 'org.apache.s4:s4-core:0.5.0'
+ Â Â Â Â Â Â s4_base:Â Â Â Â Â Â 'org.apache.s4:s4-base:0.6.0',
+ Â Â Â Â Â Â s4_comm:Â Â Â Â Â Â 'org.apache.s4:s4-comm:0.6.0',
+ Â Â Â Â Â Â s4_core:Â Â Â Â Â Â 'org.apache.s4:s4-core:0.6.0'
    ]
</code></pre>
</li>
Added: incubator/s4/site/doc/0.6.0/event_injection/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/event_injection/index.html?rev=1454912&view=auto
==============================================================================
--- incubator/s4/site/doc/0.6.0/event_injection/index.html (added)
+++ incubator/s4/site/doc/0.6.0/event_injection/index.html Sun Mar 10 20:11:54 2013
@@ -0,0 +1,94 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
+<title>S4: Inject events into S4</title>
+<meta content="A general-purpose distributed stream computing platform" name="description">
+<link href="../../../style/screen.css" media="screen" rel="stylesheet" type="text/css">
+<link href="../../../style/print.css" media="print" rel="stylesheet" type="text/css">
+<!--[if lt IE 9]>
+ <link href="../../../style/ie.css" media="screen" rel="stylesheet" type="text/css">
+ <![endif]--><link href="../../../style/style.css" rel="stylesheet" type="text/css">
+<link href="../../../style/nav.css" rel="stylesheet" type="text/css">
+<script type="text/javascript">
+ var _gaq = _gaq || [];
+ _gaq.push(['_setAccount', 'UA-19490961-1']);
+ _gaq.push(['_setDomainName', 'incubator.apache.org']);
+ _gaq.push(['_trackPageview']);
+ (function() {
+ var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
+ ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
+ var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
+ })();
+ </script><link href="../../../style/coderay.css" rel="stylesheet" type="text/css">
+<link href="../../../style/pygmentize.css" rel="stylesheet" type="text/css">
+</head>
+<body>
+ <div id="header">
+ <div class="container">
+ <div id="logo">
+ <a href="../../../">
+ <img src="../../../images/s4_test.png"></a>
+ </div>
+ <div id="navbar"><ul id="nav">
+<li>
+ <a href="../../../">home</a>
+ </li>
+ <li>
+ <a href="https://cwiki.apache.org/confluence/display/S4/S4+Wiki">doc [0.5]</a>
+ </li>
+ <li>
+ <a href="http://git-wip-us.apache.org/repos/asf?p=incubator-s4.git" onclick="_gaq.push(['_trackEvent', 'External', 'Apache Git', 'http://github.com/s4']);">code</a>
+ </li>
+ <li>
+ <a href="http://people.apache.org/~mmorel/apache-s4-0.5.0-incubating-doc/javadoc/">API</a>
+ </li>
+ <li>
+ <a href="../../../contrib">get involved</a>
+ </li>
+ <li>
+ <a href="../../../team">team</a>
+ </li>
+ <li>
+ <a href="../../../download">download</a>
+ </li>
+</ul></div>
+ </div>
+ </div>
+ <div id="wrapper">
+ <div class="container" id="container">
+<blockquote>
+ <p>How do we inject events into S4?</p>
+</blockquote>
+
+<h1 id="problem-statement">Problem statement</h1>
+<p>There is a data stream somewhere. We want to inject that data into our S4 stream processing application.</p>
+
+<p>The data stream can be almost anything: user clicks in a web application, alarms in a monitoring system, stock trade operations, even static data from a database or a file server.</p>
+
+<h1 id="requirements">Requirements</h1>
+<ul>
+<li>We must be able to connect to that stream of data</li>
+ <li>We must be able to chunk the data and isolate events out of it</li>
+</ul>
+<h1 id="solutions">Solutions</h1>
+<p>Currently S4 does not provide a specific abstraction for a source of event or an external data stream. This means that the connection to an external data source can be implemented arbitrarily. Typically, connections to external data sources are initiated in the App object itself, in the <code>start()</code> method. </p>
+
+<h2 id="in-a-single-application">In a single application</h2>
+<p>Let us consider a single S4 application. The App class is instantiated on each node (for instantiating the local topology), and this has to be taken in consideration.
+For instance, if we get the information from the twitter sprinker stream, given that the stream is provided through a single socket connection, we should connect only from one node. To ensure that, leader election through ZooKeeper is an option, but a much simpler one is simply to base this decision on the id of the partition, provided in the <code>App</code> class by <code>getReceiver().getPartitionId()</code>. In our example, we would initiate the connection only if we are in partition 0.</p>
+
+<pre><code class="language-java"></code></pre>
+
+<h2 id="using-an-adapter-application">Using an adapter application</h2>
+
+<p>In some cases the above solution is not practical: it introduces some load imbalance between the partitions, and the extraction of the</p>
+</div>
+ </div>
+ <div id="footer">
+ <div class="container">
+ <span class="copyright">Apache S4 - Copyright 2013 The Apache Software Foundation</span>
+ </div>
+ </div>
+ </body>
+</html>
Modified: incubator/s4/site/doc/0.6.0/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/index.html?rev=1454912&r1=1454911&r2=1454912&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/index.html (original)
+++ incubator/s4/site/doc/0.6.0/index.html Sun Mar 10 20:11:54 2013
@@ -79,6 +79,8 @@
</li>
<li>
<a href="event_dispatch">Dispatch events </a> within an application and between applications</li>
+ <li>
+<a href="event_injection">Inject events</a> into S4 applications</li>
</ul>
<h2 id="running-s4">Running S4</h2>
<ul>
Modified: incubator/s4/site/doc/0.6.0/overview/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/overview/index.html?rev=1454912&r1=1454911&r2=1454912&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/overview/index.html (original)
+++ incubator/s4/site/doc/0.6.0/overview/index.html Sun Mar 10 20:11:54 2013
@@ -67,10 +67,8 @@
<ul>
<li>
-<strong>performance improvements</strong>: stream throughput improved by 1000 % (200k+ messages / s / stream)</li>
- <li>improved [configurability](S4:Configuration - 0.6.0], for both the S4 platform and deployed applications</li>
- <li>
-<strong>elasticity</strong> and fine partition tuning, through an integration with Apache Helix</li>
+<strong>major performance improvements</strong>: stream throughput improved by 1000 % (200k+ messages / s / stream)</li>
+ <li>major <a href="../configuration">configurability</a> and usability improvements, for both the S4 platform and deployed applications</li>
</ul>
<h1 id="what-are-the-cool-features">What are the cool features?</h1>
@@ -78,9 +76,8 @@
<ul>
<li>Application packages are standard jar files (suffixed <code>.s4r</code>)</li>
+ <li>Platform modules for customizing the platform are standard jar files</li>
<li>By default keys are homogeneously sparsed over the cluster: helps balance the load, especially for fine grained partitioning</li>
- <li>S4 also provides fine control over the partitioning (with Apache Helix)</li>
- <li>Semi-automatic Rebalancing</li>
</ul>
<p><strong>Modular design</strong>:</p>
@@ -103,7 +100,7 @@
</ul>
</li>
</ul>
-<p><strong><a href="fault_tolerance">Fault tolerant</a></strong></p>
+<p><strong><a href="../fault_tolerance">Fault tolerant</a></strong></p>
<ul>
<li>
@@ -129,10 +126,13 @@
<ul>
<li>by default, the size of the cluster is fixed</li>
<li>the size of an S4 cluster corresponds to the number of logical <strong>partitions</strong> (sometimes referred to as <strong>tasks</strong>)</li>
- <li>an ongoing integration with <a href="http://helix.apache.org">Apache Helix</a> removes these limitations and allows a variable number of nodes and a rebalancing of partitions</li>
</ul>
</li>
</ul>
+<blockquote>
+ <p>an ongoing integration with <a href="http://helix.apache.org">Apache Helix</a> will these limitations and allows a variable number of nodes and a rebalancing of partitions</p>
+</blockquote>
+
<p><strong>Applications</strong></p>
<ul>
@@ -163,10 +163,6 @@
<p>The following diagram sums-up the key concepts in a hierarchical fashion:</p>
<p><img src="../../../images/doc/0.6.0/S4_hierarchical_archi.png" alt="image"></p>
-
-<h1 id="where-can-i-find-more-information">Where can I find more information?</h1>
-
-<p>See the <a href="resources">resources</a> page.</p>
</div>
</div>
<div id="footer">
Modified: incubator/s4/site/doc/0.6.0/walkthrough/index.html
URL: http://svn.apache.org/viewvc/incubator/s4/site/doc/0.6.0/walkthrough/index.html?rev=1454912&r1=1454911&r2=1454912&view=diff
==============================================================================
--- incubator/s4/site/doc/0.6.0/walkthrough/index.html (original)
+++ incubator/s4/site/doc/0.6.0/walkthrough/index.html Sun Mar 10 20:11:54 2013
@@ -130,6 +130,7 @@
</ul>
<pre><code class="language-java"><span class="c1">// ProcessingElement provides integration with the S4 platform</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">HelloPE</span> <span class="kd">extends</span> <span class="n">ProcessingElement</span> <span class="o">{</span>
+
<span class="c1">// you should define downstream streams here and inject them in the app definition</span>
<span class="c1">// PEs can maintain some state</span>
@@ -141,7 +142,7 @@
<span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Hello "</span> <span class="o">+</span> <span class="o">(</span><span class="n">seen</span> <span class="o">?</span> <span class="s">"again "</span> <span class="o">:</span> <span class="s">""</span><span class="o">)</span> <span class="o">+</span> <span class="n">event</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span> <span class="o">+</span> <span class="s">"!"</span><span class="o">);</span>
<span class="n">seen</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
<span class="o">}</span>
-<span class="c1">// skipped remaining methods</span></code></pre>
+ <span class="c1">// skipped remaining methods</span></code></pre>
<ul>
<li>HelloApp.java: defines a simple application: exposes an input stream (ânamesâ), connected to the HelloPE. See <a href="event_dispatch">the event dispatch configuration page</a> for more information about how events are dispatched.</li>