You are viewing a plain text version of this content. The canonical link for it is here.
Posted to s4-commits@incubator.apache.org by ki...@apache.org on 2012/12/20 19:06:49 UTC
[13/19] git commit: Update README.md
Update README.md
Project: http://git-wip-us.apache.org/repos/asf/incubator-s4/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-s4/commit/d8e2d89b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-s4/tree/d8e2d89b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-s4/diff/d8e2d89b
Branch: refs/heads/S4-110
Commit: d8e2d89be6d2ce55988cbe7af2d54897ff280273
Parents: 9fddde7
Author: Kishore Gopalakrishna <g....@gmail.com>
Authored: Wed Nov 28 01:06:09 2012 -0800
Committer: Kishore Gopalakrishna <g....@gmail.com>
Committed: Wed Nov 28 01:06:09 2012 -0800
----------------------------------------------------------------------
README.md | 52 ++++++++++++++++++++++++++++++----------------------
1 files changed, 30 insertions(+), 22 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/d8e2d89b/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 5f133b9..4a287c1 100644
--- a/README.md
+++ b/README.md
@@ -6,12 +6,16 @@ Integration with Helix
Goal is to provide better partition management, fault tolerance and automatic rebalancing during cluster expansion.
-Currently S4 has the limitation that the number of partitions is always dependent on the number of nodes. In other words
- * If the stream is already partitioned upstream or in a pub-sub system but number of nodes in s4 cluster is different, then the stream needs to be re-partitioned which results in additional hop. In cases where multiple streams that are already partitioned needs to be joined in S4, it requires re-hashing both the streams.
+Limitation in S4
+ * Tasks are always partitioned based on the number of nodes
+ * If the stream is already partitioned outside of s4 or in a pub-sub system but number of nodes in s4 cluster is different, then the stream needs to be re-partitioned which results in additional hop. In cases where multiple streams that are already partitioned outside of S4 but needs to be joined in S4, it requires re-hashing both the streams.
* When the system needs to scale, adding new nodes mean the number of partitions change. This results in lot of data shuffling and possibly losing all the state that is stored.
- * Also the fault tolerance is achieved by having stand alone nodes that remain idle and become active when a node fails. This results in inefficient use of hardware resources.
+ * Also the fault tolerance is achieved by having stand alone nodes that remain idle and becomes active when a node fails. This results in inefficient use of hardware resources.
-Integrating with Apache Helix, allows one to partition the task processing differently for each stream. Allows adding new nodes without having to change the number of partitions. Allow the fault tolerance can be at a partition level which means all nodes can be active and when a node fails its load can be equally balanced amongst the remaining active nodes.
+Integrating with Apache Helix,
+ * Allows one to partition the task processing differently for each stream.
+ * Allows adding new nodes without having to change the number of partitions.
+ * Fault tolerance can be achieved at a partition level which means all nodes can be active and when a node fails its load can be equally balanced among the remaining active nodes.
This is still in prototype mode.
@@ -28,38 +32,42 @@ Checkout the S4 integration with Helix code
git clone git://github.com/kishoreg/incubator-s4.git
-Setup eclipse, sorry command line tools will be fixed soon.
+Build S4.
- ./gradlew eclipse
+ ./gradlew install
+ ./gradlew s4-tools:installApp
+
+Start zookeeper
-The following can be directly run from eclipse.
+ ./s4 zkServer
-Create the cluster, -nbTasks is just the number of nodes.This will create two nodes localhost_12000 and localhost_120
-01
+Create the cluster, -nbNodes is just the number of s4 nodes that will be run. This will create two nodes localhost_12000 and localhost_12001
- DefineCluster -c=cluster1 -nbTasks=2 -flp=12000
+ ./s4 newCluster -c=cluster1 -nbNodes=2 -flp=12000
Create a task that processes events from stream(names). -id can be anything but should be unique, for now id and stream name needs to be the same. p is the number of partitions, so in this case it distributes 4 partitions among two nodes. -r is the number of replica/standby needed for each partition. Note that, when a node fails its load would be distributed among remaining nodes. So even though theoretically its possible to have number of standby's as the number of nodes, the performance would be horrible. In general this can be decided based on the head room available in the cluster.
- CreateTask -zk localhost:2181 -c cluster1 -id names -t consumer -p 4 -r 1 -s names
-
-Deploy the App by providing the s4r. See [s4 walk through](https://cwiki.apache.org/confluence/display/S4/S4+piper+walkthrough) instruction on how to generate this s4r. Once can optionally provide the list of nodes where this App has to be deployed.
+ ./s4 createTask -zk localhost:2181 -c cluster1 -id names -t consumer -p 4 -r 1 -s names
- DeployApp -c cluster1 -s4r <incubator-s4>/myApp/build/libs/myApp.s4r -appName HelloApp -zk localhost:2181
+Generate a HelloWorld App
-Start the s4 node, note we now need to specify the node id while starting. This gives predictability in the sense which nodes owns the task. After starting this node, you will see that since this is the only node up, it will own all the 4 partitions of the stream.
+ ./s4 newApp myApp -parentDir=/tmp
+ cd /tmp/myApp
+ ./s4 s4r -a=hello.HelloApp -b=/tmp/build.gradle myApp
+
+Deploy the App by providing the s4r. One can optionally provide the list of nodes where this App has to be deployed.
- Main -c=cluster1 -zk=localhost:2181 -id=localhost_12000
-
-Send some events to names stream. All events go to the localhost_12000 since other node is down.
+ ./s4 deployApp -c cluster1 -s4r /tmp/myApp/build/libs/myApp.s4r -appName myApp -zk localhost:2181
- GenericEventAdapter
+Start the two s4 nodes in two separate windows. Note we now need to specify the node id while starting. This allows nodes to associated with same partitions when they are started.
-Now start another node. Helix automatically detects the new node and assigns 2 out of 4 partitions to the new node. There is a hand off that happens during this part where localhost_12000 can save its state to a common location and localhost_12001 can restore from that state before it accepts new events. This state transfer is not implemented yet.
+ ./s4 node -c=cluster1 -zk=localhost:2181 -id=localhost_12000
+ ./s4 node -c=cluster1 -zk=localhost:2181 -id=localhost_12001
+
- Main -c=cluster1 -zk=localhost:2181 -id=localhost_12001
+Send some events to names stream. Notice that the partitions are divided among two nodes and each event is routed to appropriate node.
-You will see that 50% of the events now go to new node localhost_12001
+ ./s4 adapter -c=cluster1 -zk=localhost:2181 -s=names
Overview