[16/19] git commit: Adding instructions to try the integration

[ki...@apache.org]

Adding instructions to try the integration


Project: http://git-wip-us.apache.org/repos/asf/incubator-s4/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-s4/commit/62cdd2a0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-s4/tree/62cdd2a0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-s4/diff/62cdd2a0

Branch: refs/heads/S4-110
Commit: 62cdd2a05575a349d0f098d603de8bc6f063381e
Parents: 5e797c8
Author: Kishore Gopalakrishna <g....@gmail.com>
Authored: Mon Nov 26 16:44:15 2012 -0800
Committer: Kishore Gopalakrishna <g....@gmail.com>
Committed: Mon Nov 26 16:44:15 2012 -0800

----------------------------------------------------------------------
 README.md |   75 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 75 insertions(+), 0 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-s4/blob/62cdd2a0/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 2d6b3ef..8202e07 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,81 @@
 Apache S4
 =========
 
+Integration with Helix
+----------------------
+
+Goal is to provide better partition management, fault tolerance and automatic rebalancing during cluster expansion.
+
+Currently S4 has the limitation that the number of partitions is always dependent on the number of nodes. In other words 
+   * If the stream is already partitioned upstream and if the processing has to be done in S4 but only 5 nodes are sufficient, the stream needs to be re-partitioned which results in additional hop.
+   * When the system needs to scale, adding new nodes mean the number of partitions change. This results in lot of data shuffling and possibly losing all the state that is stored.
+   * Also the fault tolerance is achieved by having stand alone nodes that remain idle and become active when a node fails. This results in inefficient use of hardware resources.
+   
+Integrating with Apache Helix, allows one to partition the task processing differently for each stream. Allows adding new nodes without having to change the number of partitions. Allow the fault tolerance can be at a partition level which means all nodes can be active and when a node fails its load can be equally balanced amongst the remaining active nodes.
+
+
+This is still in prototype mode.
+
+To try it, 
+
+# This will install the helix jars into local repo
+   git clone git://github.com/apache/incubator-helix.git
+   ./build or mvn clean install -Dmaven.test.exec.skip=true
+
+# Checkout the integration with Helix code
+   git clone git://github.com/kishoreg/incubator-s4.git
+
+The following things can be directly run from eclipse.
+ 
+#Create the cluster, -nbTasks is just the number of nodes.
+# This will create two nodes localhost_12000 and localhost_12001
+   DefineCluster -c=cluster1 -nbTasks=2 -flp=12000
+
+#Create a stream(names) consumer/processor task. Id can be anything but should be unique, for now both needs to be the same,
+#p is the number of partitions, so in this case it distributes 4 partitions among two nodes. -r is the number of replica/standby needed for each partition. 
+
+   CreateTask  -zk localhost:2181 -c cluster1 -id names -t consumer -p 4 -r 1 -s names
+
+# Deploy the name by providing the s4r. See the s4 walk through instruction on how to generate this s4r.
+   DeployApp -c cluster1 -s4r <incubator-s4>/myApp/build/libs/myApp.s4r -appName HelloApp -zk localhost:2181
+
+# Start the s4 node, note we now need to specify the node id while starting. This gives predictability in which nodes owns the task. After starting this node, you will see that since this is the only node up, it will own all the 4 partitions.
+   Main -c=cluster1 -zk=localhost:2181 -id=localhost_12000
+   
+#Send some events to cluster1. This sends events to the localhost_12000
+  GenericEventAdapter 
+
+# Now start another node. Helix automatically detects the new node and assigns 2 out of 4 partitions to the new node. There is a hand off that happens during this part where localhost_12000 can save its state to a common location and localhost_12001 can restore from that state before it accepts new events. This state transfer is not implemented yet.
+   Main -c=cluster1 -zk=localhost:2181 -id=localhost_12001
+
+You will see that 50% of the events now go new node localhost_12001
+
+
+TODO: Add new node.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 Overview
 --------
 S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable