You are viewing a plain text version of this content. The canonical link for it is here.
Posted to s4-commits@incubator.apache.org by mm...@apache.org on 2013/06/01 16:35:48 UTC
git commit: Update website for 0.6.0 release
Updated Branches:
refs/heads/dev 2f474503e -> 2548710e1
Update website for 0.6.0 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-s4/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-s4/commit/2548710e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-s4/tree/2548710e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-s4/diff/2548710e
Branch: refs/heads/dev
Commit: 2548710e1f8fcd135ab2b84372ece5f4842bc036
Parents: 2f47450
Author: Matthieu Morel <mm...@apache.org>
Authored: Sat Jun 1 18:35:05 2013 +0200
Committer: Matthieu Morel <mm...@apache.org>
Committed: Sat Jun 1 18:35:05 2013 +0200
----------------------------------------------------------------------
website/README.markdown | 6 +-
website/content/doc/0.6.0/event_injection.md | 20 ++--
website/content/doc/0.6.0/walkthrough.md | 4 +-
website/content/download.haml | 11 ++-
website/content/index.haml | 17 ++--
website/content/style/coderay.scss | 119 ---------------------
website/layouts/nav.haml | 4 +-
7 files changed, 36 insertions(+), 145 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/2548710e/website/README.markdown
----------------------------------------------------------------------
diff --git a/website/README.markdown b/website/README.markdown
index e1f1c67..2cacf6e 100644
--- a/website/README.markdown
+++ b/website/README.markdown
@@ -1,12 +1,14 @@
-Code for the S4 website
+Code for the S4 website.
-This website uses the [nanoc](http://nanoc.stoneship.org/) static website generator
+This website uses the [nanoc](http://nanoc.stoneship.org/) static website generator.
Entry pages are written with haml and the documentation is written with markdown.
# To compile the site:
+You need ruby 1.8+ and gem 2.0.3+
+
* Install nanoc: `gem install nanoc`
* `nanoc compile`
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/2548710e/website/content/doc/0.6.0/event_injection.md
----------------------------------------------------------------------
diff --git a/website/content/doc/0.6.0/event_injection.md b/website/content/doc/0.6.0/event_injection.md
index 685ff86..70688a5 100644
--- a/website/content/doc/0.6.0/event_injection.md
+++ b/website/content/doc/0.6.0/event_injection.md
@@ -7,7 +7,7 @@ title: Inject events into S4
# Problem statement
There is a data stream somewhere. We want to inject that data into our S4 stream processing application.
-The data stream can be almost anything: user clicks in a web application, alarms in a monitoring system, stock trade operations, even static data from a database or a file server.
+The data stream can be almost anything: user clicks in a web application, alarms in a monitoring system, stock trade operations, or data from a database or a file system.
# Requirements
* We must be able to connect to that stream of data
@@ -17,20 +17,24 @@ The data stream can be almost anything: user clicks in a web application, alarms
Currently S4 does not provide a specific abstraction for a source of event or an external data stream. This means that the connection to an external data source can be implemented arbitrarily. Typically, connections to external data sources are initiated in the App object itself, in the `start()` method.
## In a single application
-Let us consider a single S4 application. The App class is instantiated on each node (for instantiating the local topology), and this has to be taken in consideration.
-For instance, if we get the information from the twitter sprinker stream, given that the stream is provided through a single socket connection, we should connect only from one node. To ensure that, leader election through ZooKeeper is an option, but a much simpler one is simply to base this decision on the id of the partition, provided in the `App` class by `getReceiver().getPartitionId()`. In our example, we would initiate the connection only if we are in partition 0.
+Let us consider a single S4 application. Remember, the App class is instantiated on each node (for instantiating the local topology). This has some implications. For instance, if we get the information from the twitter sprinker stream, given that the stream is provided through a single socket connection, we should connect only from one node. To ensure that, leader election through ZooKeeper could be an option, but a much simpler one is simply to base this decision on the id of the partition, provided in the `App` class by `getReceiver().getPartitionId()`. In our example, we would initiate the connection only if we are in partition 0.
-~~~
-#!java
+>The code from the twitter adapter app example could actually be factored into the twitter-counter example App class itself, by listening to the sprinkler stream from partition 0 (for instance)
+## Using an adapter application
-~~~
+In some cases the above solution is not practical: it may introduce some load imbalance between the partitions, the extraction of the events out of an external source may be complex, and we might want to enrich the events with other information. In that case, it could be useful to delegate these tasks to a separate application: the adapter application.
+Adapter apps can take advantage of the AdapterApp class that automatically creates an output stream. One can define an arbitrary complex graph of PEs and simply use the output stream (`getOutputStream`) to pass events to downstream S4 apps.
-## Using an adapter application
+S4 applications communicate by matching names of input and output streams. The current mechanism is very simple : when an application creates an output stream, it is exposed in ZooKeeper and other applications that define input streams with the same name are automatically connected, through that matching stream.
+
+By default, events are dispatched in a round-robin fashion between S4 apps.
+
+The [twitter example](../twitter_trending_example) uses an adapter application (though a very basic one).
-In some cases the above solution is not practical: it introduces some load imbalance between the partitions, and the extraction of the
+More info on event dispatch is also available [here](../event_dispatch).
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/2548710e/website/content/doc/0.6.0/walkthrough.md
----------------------------------------------------------------------
diff --git a/website/content/doc/0.6.0/walkthrough.md b/website/content/doc/0.6.0/walkthrough.md
index 60715ec..96363f9 100644
--- a/website/content/doc/0.6.0/walkthrough.md
+++ b/website/content/doc/0.6.0/walkthrough.md
@@ -131,9 +131,7 @@ In order to run an S4 application, you need :
[main] INFO org.apache.s4.tools.ZKServer - Starting zookeeper server on port [2181]
[main] INFO org.apache.s4.tools.ZKServer - cleaning existing data in [/var/folders/8V/8VdgKWU3HCiy2yV4dzFpDk+++TI/-Tmp-/tmp/zookeeper/data] and [/var/folders/8V/8VdgKWU3HCiy2yV4dzFpDk+++TI/-Tmp-/tmp/zookeeper/log]
-
-
-1. Define a new cluster. Say a cluster named "cluster1" with 2 partitions, nodes listening to ports starting from 12000:
+ 1. Define a new cluster. Say a cluster named "cluster1" with 2 partitions, nodes listening to ports starting from 12000:
S4:myApp$ ./s4 newCluster -c=cluster1 -nbTasks=2 -flp=12000
calling referenced s4 script : /Users/S4/tmp/incubator-s4/s4
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/2548710e/website/content/download.haml
----------------------------------------------------------------------
diff --git a/website/content/download.haml b/website/content/download.haml
index ec45db4..1b88744 100644
--- a/website/content/download.haml
+++ b/website/content/download.haml
@@ -5,11 +5,20 @@ title: Download
%h1{:class=>"titleheader"}= "Downloading S4"
- %p= "The current stable version is 0.5.0-incubating. You can verify your download by following <a href='http://www.apache.org/info/verification.html'>these procedures</a> and using <a href='http://www.apache.org/dist/incubator/s4/KEYS'>these keys</a>."
+ %p= "The current stable version is 0.6.0-incubating. You can verify your download by following <a href='http://www.apache.org/info/verification.html'>these procedures</a> and using <a href='http://www.apache.org/dist/incubator/s4/KEYS'>these keys</a>."
+
+ / 0.6.0
+ %h2{:class=>"secheader"}= "0.6.0 release"
+
+ %p Focus on configurability and performance.
+ %ul
+ %li <a href='http://www.apache.org/dist/incubator/s4/s4-0.6.0-incubating/RELEASE_NOTES.html'>Release notes</a>
+ %li Source release: <a href='http://www.apache.org/dist/incubator/s4/s4-6.0-incubating/apache-s4-0.6.0-incubating-src.zip'>s4-0.6.0-incubating-src.zip</a> (<a href='http://www.apache.org/dist/incubator/s4/s4-0.6.0-incubating/apache-s4-0.6.0-incubating-src.zip.asc'>asc</a>, <a href='http://www.apache.org/dist/incubator/s4/s4-0.6.0-incubating/apache-s4-0.6.0-incubating-src.zip.md5'>md5</a>)
/ 0.5.0
%h2{:class=>"secheader"}= "0.5.0 release"
+ %p Complete refactoring aiming at a cleaner API and codebase, and introducing new tools for easing development and deployment
%ul
%li <a href='http://www.apache.org/dist/incubator/s4/s4-0.5.0-incubating/RELEASE_NOTES.html'>Release notes</a>
%li Source release: <a href='http://www.apache.org/dist/incubator/s4/s4-0.5.0-incubating/apache-s4-0.5.0-incubating-src.zip'>s4-0.5.0-incubating-src.zip</a> (<a href='http://www.apache.org/dist/incubator/s4/s4-0.5.0-incubating/apache-s4-0.5.0-incubating-src.zip.asc'>asc</a>, <a href='http://www.apache.org/dist/incubator/s4/s4-0.5.0-incubating/apache-s4-0.5.0-incubating-src.zip.md5'>md5</a>)
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/2548710e/website/content/index.haml
----------------------------------------------------------------------
diff --git a/website/content/index.haml b/website/content/index.haml
index b288a3b..1543584 100644
--- a/website/content/index.haml
+++ b/website/content/index.haml
@@ -5,13 +5,10 @@ title: Distributed Stream Computing Platform
#about{:class=>"span-18 last"}
#notice{:class=>"span-17 last"}
- #notice_text= "March 2013: <b>S4 0.6.0 Release Candidate</b> <a href='http://people.apache.org/~mmorel/s4-0.6.0-incubating-release-candidate-3/'>available for download</a>. Check the <a href='/doc/0.6.0'>improved documentation</a>!."
+ #notice_text= "June 2013: <b>S4 0.6.0 Released</b> Get it <a href='/download'>here</a>!"
#notice{:class=>"span-17 last"}
- #notice_text= "August 2012: <b>S4 0.5.0 has been released!</b> Get it <a href='/download'>here</a>!."
-
- #notice{:class=>"span-17 last"}
- #notice_text= "S4 0.5.0 \"Piper\" is a complete refactoring of the previous version of S4. It provides a clearer API and a more robust implementation. It features TCP based communications, state recovery, and a new set of tools. More information is available in an <a href='https://cwiki.apache.org/confluence/display/S4/S4+0.5.0+Overview'>overview</a> and you may also check the <a href='http://www.apache.org/dist/incubator/s4/s4-0.5.0-incubating/RELEASE_NOTES.html'>release notes</a>."
+ #notice_text= "S4 0.6.0 focuses on configurability and performance. More information is available in an <a href='http://incubator.apache.org/s4/doc/0.6.0/overview/'>overview</a> and you may also check the <a href='http://www.apache.org/dist/incubator/s4/s4-0.6.0-incubating/RELEASE_NOTES.html'>release notes</a>."
%p{:class=>'large'}= "S4 is a <span class='high'>general-purpose</span>, <span class='high'>distributed</span>, <span class='high'>scalable</span>, <span class='high'>fault-tolerant</span>, <span class='high'>pluggable</span> platform that allows programmers to easily develop applications for processing continuous unbounded streams of data."
@@ -50,7 +47,7 @@ title: Distributed Stream Computing Platform
#ov_4{:class=>"span-6"}
%h4{:class=>"subsecheader"}= "extensible"
- %p= "Applications can easily be written and deployed using a simple API. Many basic applications for stream processing are available out of the box and more are being written."
+ %p= "Applications can easily be written and deployed using a simple API. Building blocks of the platform (message queues and processors, serializer, checkpointing backend) can be replaced by custom implementations."
#ov_5{:class=>"span-6"}
%h4{:class=>"subsecheader"}= "cluster management"
%p= "S4 hides all cluster management tasks using a communication layer built on top of <a href='http://hadoop.apache.org/zookeeper'>ZooKeeper</a>, a distributed, open-source coordination service for distributed applications."
@@ -68,13 +65,13 @@ title: Distributed Stream Computing Platform
#sidebar{:class=>"span-5 last"}
#download
#latest_head
- %span= "S4 Piper release"
+ %span= "S4 release"
%br
- %span{:style=>"font-size: small;"}= "v0.5.0"
+ %span{:style=>"font-size: small;"}= "v0.6.0"
#latest_link
- %a{:href=>"https://cwiki.apache.org/confluence/display/S4/S4+piper+walkthrough", :style=>"border: none;"}= "Get Started"
+ %a{:href=>"http://incubator.apache.org/s4/doc/0.6.0/", :style=>"border: none;"}= "Get Started"
#disclaimer
- %p= "This is an alpha version. There's no guarantee of backwards-compatibility until the 1.0 release."
+ %p= "There is no guarantee of backwards-compatibility until the 1.0 release."
#participate
%h2{:class=>"secheader"}= "participate"
%ul{:class=>'large greybar'}
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/2548710e/website/content/style/coderay.scss
----------------------------------------------------------------------
diff --git a/website/content/style/coderay.scss b/website/content/style/coderay.scss
deleted file mode 100644
index 82d083a..0000000
--- a/website/content/style/coderay.scss
+++ /dev/null
@@ -1,119 +0,0 @@
-.CodeRay {
- background-color: hsl(0,0%,95%);
- border: 1px solid silver;
- color: black;
-}
-.CodeRay pre {
- margin: 0px;
-}
-
-span.CodeRay { white-space: pre; border: 0px; padding: 2px; }
-
-table.CodeRay { border-collapse: collapse; width: 100%; padding: 2px; }
-table.CodeRay td { padding: 2px 4px; vertical-align: top; }
-
-.CodeRay .line-numbers {
- color: gray;
- text-align: right;
- -webkit-user-select: none;
- -moz-user-select: none;
- user-select: none;
-}
-.CodeRay .line-numbers a {
- color: gray !important;
- text-decoration: none !important;
- padding-right: 5px;
-}
-.CodeRay .line-numbers a:target
-.CodeRay .line-numbers .highlighted
-.CodeRay .line-numbers .highlighted a
-.CodeRay span.line-numbers { padding: 0px 4px; }
-.CodeRay .line { display: block; float: left; width: 100%; }
-.CodeRay .code { width: 100%; }
-
-.CodeRay .debug { color: white !important; background: blue !important; }
-
-.CodeRay .annotation { color:#007 }
-.CodeRay .attribute-name { color:#b48 }
-.CodeRay .attribute-value { color:#700 }
-.CodeRay .binary { color:#509 }
-.CodeRay .char .content { color:#D20 }
-.CodeRay .char .delimiter { color:#710 }
-.CodeRay .char { color:#D20 }
-.CodeRay .class { color:#B06; font-weight:bold }
-.CodeRay .class-variable { color:#369 }
-.CodeRay .color { color:#0A0 }
-.CodeRay .comment { color:#777 }
-.CodeRay .comment .char { color:#444 }
-.CodeRay .comment .delimiter { color:#444 }
-.CodeRay .complex { color:#A08 }
-.CodeRay .constant { color:#036; font-weight:bold }
-.CodeRay .decorator { color:#B0B }
-.CodeRay .definition { color:#099; font-weight:bold }
-.CodeRay .delimiter { color:black }
-.CodeRay .directive { color:#088; font-weight:bold }
-.CodeRay .doc { color:#970 }
-.CodeRay .doc-string { color:#D42; font-weight:bold }
-.CodeRay .doctype { color:#34b }
-.CodeRay .entity { color:#800; font-weight:bold }
-.CodeRay .error { color:#F00; background-color:#FAA }
-.CodeRay .escape { color:#666 }
-.CodeRay .exception { color:#C00; font-weight:bold }
-.CodeRay .float { color:#60E }
-.CodeRay .function { color:#06B; font-weight:bold }
-.CodeRay .global-variable { color:#d70 }
-.CodeRay .hex { color:#02b }
-.CodeRay .imaginary { color:#f00 }
-.CodeRay .include { color:#B44; font-weight:bold }
-.CodeRay .inline { background-color: hsla(0,0%,0%,0.07); color: black }
-.CodeRay .inline-delimiter { font-weight: bold; color: #666 }
-.CodeRay .instance-variable { color:#33B }
-.CodeRay .integer { color:#00D }
-.CodeRay .key .char { color: #60f }
-.CodeRay .key .delimiter { color: #404 }
-.CodeRay .key { color: #606 }
-.CodeRay .keyword { color:#080; font-weight:bold }
-.CodeRay .label { color:#970; font-weight:bold }
-.CodeRay .local-variable { color:#963 }
-.CodeRay .namespace { color:#707; font-weight:bold }
-.CodeRay .octal { color:#40E }
-.CodeRay .operator { }
-.CodeRay .predefined { color:#369; font-weight:bold }
-.CodeRay .predefined-constant { color:#069 }
-.CodeRay .predefined-type { color:#0a5; font-weight:bold }
-.CodeRay .preprocessor { color:#579 }
-.CodeRay .pseudo-class { color:#00C; font-weight:bold }
-.CodeRay .regexp .content { color:#808 }
-.CodeRay .regexp .delimiter { color:#404 }
-.CodeRay .regexp .modifier { color:#C2C }
-.CodeRay .regexp { background-color:hsla(300,100%,50%,0.06); }
-.CodeRay .reserved { color:#080; font-weight:bold }
-.CodeRay .shell .content { color:#2B2 }
-.CodeRay .shell .delimiter { color:#161 }
-.CodeRay .shell { background-color:hsla(120,100%,50%,0.06); }
-.CodeRay .string .char { color: #b0b }
-.CodeRay .string .content { color: #D20 }
-.CodeRay .string .delimiter { color: #710 }
-.CodeRay .string .modifier { color: #E40 }
-.CodeRay .string { background-color:hsla(0,100%,50%,0.05); }
-.CodeRay .symbol .content { color:#A60 }
-.CodeRay .symbol .delimiter { color:#630 }
-.CodeRay .symbol { color:#A60 }
-.CodeRay .tag { color:#070 }
-.CodeRay .type { color:#339; font-weight:bold }
-.CodeRay .value { color: #088; }
-.CodeRay .variable { color:#037 }
-
-.CodeRay .insert { background: hsla(120,100%,50%,0.12) }
-.CodeRay .delete { background: hsla(0,100%,50%,0.12) }
-.CodeRay .change { color: #bbf; background: #007; }
-.CodeRay .head { color: #f8f; background: #505 }
-.CodeRay .head .filename { color: white; }
-
-.CodeRay .delete .eyecatcher { background-color: hsla(0,100%,50%,0.2); border: 1px solid hsla(0,100%,45%,0.5); margin: -1px; border-bottom: none; border-top-left-radius: 5px; border-top-right-radius: 5px; }
-.CodeRay .insert .eyecatcher { background-color: hsla(120,100%,50%,0.2); border: 1px solid hsla(120,100%,25%,0.5); margin: -1px; border-top: none; border-bottom-left-radius: 5px; border-bottom-right-radius: 5px; }
-
-.CodeRay .insert .insert { color: #0c0; background:transparent; font-weight:bold }
-.CodeRay .delete .delete { color: #c00; background:transparent; font-weight:bold }
-.CodeRay .change .change { color: #88f }
-.CodeRay .head .head { color: #f4f }
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/2548710e/website/layouts/nav.haml
----------------------------------------------------------------------
diff --git a/website/layouts/nav.haml b/website/layouts/nav.haml
index a8229b3..0ff05b2 100644
--- a/website/layouts/nav.haml
+++ b/website/layouts/nav.haml
@@ -2,11 +2,11 @@
%li
%a{:href=>"/"}= "home"
%li
- %a{:href=>"https://cwiki.apache.org/confluence/display/S4/S4+Wiki"}= "doc [0.5]"
+ %a{:href=>"/doc/0.6.0"}= "doc"
%li
%a{:href=>"http://git-wip-us.apache.org/repos/asf?p=incubator-s4.git", :onClick=> google_analytics_event("External", "Apache Git", "http://github.com/s4")}= "code"
%li
- %a{:href=>"http://people.apache.org/~mmorel/apache-s4-0.5.0-incubating-doc/javadoc/"}="API"
+ %a{:href=>"http://people.apache.org/~mmorel/apache-s4-0.6.0-incubating-doc/javadoc/"}="API"
%li
%a{:href=>"/contrib"}= "get involved"
%li