You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@helix.apache.org by ki...@apache.org on 2013/05/24 22:27:23 UTC
git commit: Applying additional changes to website from rms(bob)
Updated Branches:
refs/heads/master b162f1bec -> 3aa36efb5
Applying additional changes to website from rms(bob)
Project: http://git-wip-us.apache.org/repos/asf/incubator-helix/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-helix/commit/3aa36efb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-helix/tree/3aa36efb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-helix/diff/3aa36efb
Branch: refs/heads/master
Commit: 3aa36efb5fd68ee6a5042de9c819caddd94394f0
Parents: b162f1b
Author: kishoreg <ki...@apache.org>
Authored: Fri May 24 13:27:14 2013 -0700
Committer: kishoreg <ki...@apache.org>
Committed: Fri May 24 13:27:14 2013 -0700
----------------------------------------------------------------------
src/site/markdown/Architecture.md | 91 ++++++++++---------
src/site/markdown/Concepts.md | 83 +++++++++---------
src/site/markdown/Quickstart.md | 114 +++++++++++++++++++++---
src/site/markdown/index.md | 2 +
src/site/markdown/tutorial_participant.md | 4 +-
src/site/markdown/tutorial_spectator.md | 8 +--
src/site/markdown/tutorial_throttling.md | 1 -
7 files changed, 196 insertions(+), 107 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-helix/blob/3aa36efb/src/site/markdown/Architecture.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/Architecture.md b/src/site/markdown/Architecture.md
index bf7b4d0..7acf590 100644
--- a/src/site/markdown/Architecture.md
+++ b/src/site/markdown/Architecture.md
@@ -18,11 +18,11 @@ under the License.
-->
-Helix aims to provide the following abilities to a distributed system
+Helix aims to provide the following abilities to a distributed system:
-* Auto management of a cluster hosting partitioned, replicated resources
+* Automatic management of a cluster hosting partitioned, replicated resources.
* Soft and hard failure detection and handling.
-* Automatic load balancing via smart placement of resources on servers(nodes) based on server capacity and resource profile (size of partition, access patterns, etc)
+* Automatic load balancing via smart placement of resources on servers(nodes) based on server capacity and resource profile (size of partition, access patterns, etc).
* Centralized config management and self discovery. Eliminates the need to modify config on each node.
* Fault tolerance and optimized rebalancing during cluster expansion.
* Manages entire operational lifecycle of a node. Addition, start, stop, enable/disable without downtime.
@@ -57,10 +57,10 @@ We have divided Helix in 3 logical components based on their responsibility
3. CONTROLLER: The controller observes and controls the PARTICIPANT nodes. It is responsible for coordinating all transitions in the cluster and ensuring that state constraints are satisfied and cluster stability is maintained.
-These are simply logical components and can be deployed as per the system requirements. For example.
+These are simply logical components and can be deployed as per the system requirements. For example:
-1. Controller can be deployed as a separate service.
-2. Controller can be deployed along with Participant but only one Controller will be active at any given time.
+1. Controller can be deployed as a separate service
+2. Controller can be deployed along with a Participant but only one Controller will be active at any given time.
Both have pros and cons, which will be discussed later and one can chose the mode of deployment as per system needs.
@@ -69,13 +69,13 @@ Both have pros and cons, which will be discussed later and one can chose the mod
We need a distributed store to maintain the state of the cluster and a notification system to notify if there is any change in the cluster state. Helix uses Zookeeper to achieve this functionality.
-Zookeeper provides
+Zookeeper provides:
* A way to represent PERSISTENT state which basically remains until its deleted.
* A way to represent TRANSIENT/EPHEMERAL state which vanishes when the process that created the STATE dies.
* Notification mechanism when there is a change in PERSISTENT/EPHEMERAL STATE
-The name space provided by ZooKeeper is much like that of a standard file system. A name is a sequence of path elements separated by a slash (/). Every node[ZNODE] in ZooKeeper's name space is identified by a path.
+The namespace provided by ZooKeeper is much like that of a standard file system. A name is a sequence of path elements separated by a slash (/). Every node[ZNODE] in ZooKeeper\'s namespace is identified by a path.
More info on Zookeeper can be found here http://zookeeper.apache.org
@@ -83,14 +83,14 @@ More info on Zookeeper can be found here http://zookeeper.apache.org
Even though the concept of Resource, Partition, Replicas is common to most distributed systems, one thing that differentiates one distributed system from another is the way each partition is assigned a state and the constraints on each state.
-For example,
+For example:
-1. If a system is serving READ ONLY data then all partitions replicas are equal and they can either be ONLINE or OFFLINE.
-2. If a system takes BOTH READ and WRITES but ensure that WRITES go through only one partition then the states will be MASTER,SLAVE and OFFLINE. Writes go through the MASTER and is replicated to the SLAVES. Optionally, READS can go through SLAVES
+1. If a system is serving READ ONLY data then all partition\'s replicas are equal and they can either be ONLINE or OFFLINE.
+2. If a system takes BOTH READ and WRITES but ensure that WRITES go through only one partition then the states will be MASTER, SLAVE and OFFLINE. Writes go through the MASTER and is replicated to the SLAVES. Optionally, READS can go through SLAVES.
-Apart from defining State for each partition, the transition path to each STATE can be application specific. For example, in order to become master it might be a requirement to first become a SLAVE. This ensures that if the SLAVE does not have the data as part of OFFLINE-SLAVE transition it can bootstrap data from other nodes in the system.
+Apart from defining STATE for each partition, the transition path to each STATE can be application specific. For example, in order to become MASTER it might be a requirement to first become a SLAVE. This ensures that if the SLAVE does not have the data as part of OFFLINE-SLAVE transition it can bootstrap data from other nodes in the system.
-Helix provides a way to configure an application specific state machine along with constraints on each state. Along with constraints on State, Helix also provides a way to specify constraints on transitions.(More on this later)
+Helix provides a way to configure an application specific state machine along with constraints on each state. Along with constraints on STATE, Helix also provides a way to specify constraints on transitions. (More on this later.)
```
OFFLINE | SLAVE | MASTER
@@ -113,25 +113,25 @@ MASTER | SLAVE | SLAVE | N/A |
The following terminologies are used in Helix to model a state machine.
-* IDEALSTATE:The state in which we need the cluster to be in if all nodes are up and running. In other words, all state constraints are satisfied.
-* CURRENSTATE:Represents the current state of each node in the cluster
-* EXTERNALVIEW:Represents the combined view of CURRENTSTATE of all nodes.
+* IDEALSTATE: The state in which we need the cluster to be in if all nodes are up and running. In other words, all state constraints are satisfied.
+* CURRENTSTATE: Represents the current state of each node in the cluster
+* EXTERNALVIEW: Represents the combined view of CURRENTSTATE of all nodes.
-Goal of Helix is always to make the CURRENTSTATE of the system same as the IDEALSTATE. Some scenarios where this may not be true are
+The goal of Helix is always to make the CURRENTSTATE of the system same as the IDEALSTATE. Some scenarios where this may not be true are:
* When all nodes are down
* When one or more nodes fail
-* New nodes are added and the partitions need to be reassigned.
+* New nodes are added and the partitions need to be reassigned
### IDEALSTATE
-Helix lets the application define the IdealState on a resource basis which basically consists of
+Helix lets the application define the IdealState on a resource basis which basically consists of:
-* List of partitions. Example 64
-* Number of replicas for each partition. Example 3
+* List of partitions. Example: 64
+* Number of replicas for each partition. Example: 3
* Node and State for each replica.
-Example
+Example:
* Partition-1, replica-1, Master, Node-1
* Partition-1, replica-2, Slave, Node-2
@@ -144,7 +144,7 @@ Helix comes with various algorithms to automatically assign the partitions to no
### CURRENTSTATE
-Every Instance in the cluster hosts one or more partitions of a resource. Each of the partitions has a State associated with it.
+Every instance in the cluster hosts one or more partitions of a resource. Each of the partitions has a State associated with it.
Example Node-1
@@ -169,20 +169,19 @@ External clients needs to know the state of each partition in the cluster and th
Mode of operation in a cluster
-A node process can be one of the following
+A node process can be one of the following:
-* PARTICIPANT: The process registers itself in the cluster and acts on the messages received in its queue and updates the current state. Example:Storage Node
-* SPECTATOR: The process is simply interested in the changes in the externalView. Router is a spectator of Storage cluster.
-* CONTROLLER:This process actively controls the cluster by reacting to changes in Cluster State and sending messages to PARTICIPANTS
+* PARTICIPANT: The process registers itself in the cluster and acts on the messages received in its queue and updates the current state. Example: Storage Node
+* SPECTATOR: The process is simply interested in the changes in the Externalview. The Router is a spectator of the Storage cluster.
+* CONTROLLER: This process actively controls the cluster by reacting to changes in Cluster State and sending messages to PARTICIPANTS.
### Participant Node Process
-
* When Node starts up, it registers itself under LIVEINSTANCES
* After registering, it waits for new Messages in the message queue
-* When it receives a message, it will perform the required task as indicated in the message.
-* After the task is completed, depending on the task outcome it updates the CURRENTSTATE.
+* When it receives a message, it will perform the required task as indicated in the message
+* After the task is completed, depending on the task outcome it updates the CURRENTSTATE
### Controller Process
@@ -192,10 +191,10 @@ A node process can be one of the following
### Spectator Process
-* When the process starts it asks cluster manager agent to be notified of changes in ExternalView
-* When ever it receives a notification it reads the externalView and performs required duties. For Router, it updates its routing table.
+* When the process starts, it asks cluster manager agent to be notified of changes in ExternalView
+* Whenever it receives a notification, it reads the Externalview and performs required duties. For the Router, it updates its routing table.
-h4. Interaction between controller, participant and spectator
+#### Interaction between controller, participant and spectator
The following picture shows how controllers, participants and spectators interact with each other.
@@ -203,22 +202,22 @@ The following picture shows how controllers, participants and spectators interac
## Core algorithm
-
-* Controller gets the Ideal State and the Current State of active storage nodes from ZK
-* Compute the delta between Ideal State and Current State for each partition across all participant nodes
-* For each partition compute tasks based on State Machine Table. Its possible to configure priority on the state Transition. For example in case of Master Slave:
- * Attempt Mastership transfer if possible without violating constraint.
+* Controller gets the IdealState and the CurrentState of active storage nodes from Zookeeper
+* Compute the delta between IdealState and CurrentState for each partition across all participant nodes
+* For each partition compute tasks based on the State Machine Table. It\'s possible to configure priority on the state Transition. For example, in case of Master-Slave:
+ * Attempt mastership transfer if possible without violating constraint.
* Partition Addition
* Drop Partition
-* Add the tasks in parallel if possible to respective queue for each storage node keeping in mind
-* The tasks added are mutually independent.
-* If a task is dependent on another task being completed do not add that task.
-* After any task is completed by Participant, Controllers gets notified of the change and State Transition algorithm is re-run until the current state is same as Ideal State.
+* Add the tasks in parallel if possible to the respective queue for each storage node (if the tasks added are mutually independent)
+* If a task is dependent on another task being completed, do not add that task
+* After any task is completed by a Participant, Controllers gets notified of the change and the State Transition algorithm is re-run until the CurrentState is same as IdealState.
## Helix znode layout
Helix organizes znodes under clusterName in multiple levels.
-The top level (under clusterName) znodes are all Helix defined and in upper case
+
+The top level (under clusterName) znodes are all Helix defined and in upper case:
+
* PROPERTYSTORE: application property store
* STATEMODELDEFES: state model definitions
* INSTANCES: instance runtime information including current state and messages
@@ -229,15 +228,17 @@ The top level (under clusterName) znodes are all Helix defined and in upper case
* CONTROLLER: cluster controller runtime information
Under INSTANCES, there are runtime znodes for each instance. An instance organizes znodes as follows:
+
* CURRENTSTATES
- * sessionId
- * resourceName
+ * sessionId
+ * resourceName
* ERRORS
* STATUSUPDATES
* MESSAGES
* HEALTHREPORT
Under CONFIGS, there are different scopes of configurations:
+
* RESOURCE: contains resource scope configurations
* CLUSTER: contains cluster scope configurations
* PARTICIPANT: contains participant scope configurations
http://git-wip-us.apache.org/repos/asf/incubator-helix/blob/3aa36efb/src/site/markdown/Concepts.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/Concepts.md b/src/site/markdown/Concepts.md
index 9fb8eb7..91e84b5 100644
--- a/src/site/markdown/Concepts.md
+++ b/src/site/markdown/Concepts.md
@@ -17,16 +17,16 @@ specific language governing permissions and limitations
under the License.
-->
-Helix is based on the simple fact that a given task has the following attributes associated with it
+Helix is based on the idea that a given task has the following attributes associated with it:
-* Location of the task, for example it runs on Node N1
-* State, for examples its running, stopped etc.
+* _Location of the task_. For example it runs on Node N1
+* _State_. For example, it is running, stopped etc.
-A task is referred to as a 'resource'.
+In Helix terminology, a task is referred to as a _resource_.
-### IDEALSTATE
+### IdealState
-Ideal state simply allows one to map tasks to location and state. A standard way of expressing this in Helix is
+IdealState simply allows one to map tasks to location and state. A standard way of expressing this in Helix:
```
"TASK_NAME" : {
@@ -34,7 +34,7 @@ Ideal state simply allows one to map tasks to location and state. A standard way
}
```
-Consider a simple case where you want to launch a task 'myTask' on node 'N1'. The idealstate for this can be expressed as follows
+Consider a simple case where you want to launch a task \'myTask\' on node \'N1\'. The IdealState for this can be expressed as follows:
```
{
@@ -46,11 +46,11 @@ Consider a simple case where you want to launch a task 'myTask' on node 'N1'. Th
}
}
```
-#### PARTITION
+### Partition
-If this task get too big to fit on one box, you might want to divide it into subTasks. Each subTask is referred to as 'partition' in Helix. Lets say you want to divide the task into 3 subTasks/partitions, the idealstate can be changed as shown below.
+If this task get too big to fit on one box, you might want to divide it into subTasks. Each subTask is referred to as a _partition_ in Helix. Let\'s say you want to divide the task into 3 subTasks/partitions, the IdealState can be changed as shown below.
-'myTask_0', 'myTask_1', 'myTask_2' are logical names representing the partitions of myTask. Each tasks runs on N1,N2 and N3 respectively.
+\'myTask_0\', \'myTask_1\', \'myTask_2\' are logical names representing the partitions of myTask. Each tasks runs on N1, N2 and N3 respectively.
```
{
@@ -72,13 +72,13 @@ If this task get too big to fit on one box, you might want to divide it into sub
}
```
-#### REPLICA
+### Replica
-Partitioning allows one to split the data/task into multiple subparts. But lets say the request rate each partition increases. The common solution is to have multiple copies for each partition. Helix refers to the copy of a partition as 'replica'. Adding replica also increases the availability of the system during failures. One can see this methodology employed often in Search systems. The index is divided into shard and each shard has multiple copies.
+Partitioning allows one to split the data/task into multiple subparts. But let\'s say the request rate each partition increases. The common solution is to have multiple copies for each partition. Helix refers to the copy of a partition as a _replica_. Adding a replica also increases the availability of the system during failures. One can see this methodology employed often in Search systems. The index is divided into shards, and each shard has multiple copies.
-Lets say you want to add one additional replica for each task. The idealstate can simply be changed as shown below.
+Let\'s say you want to add one additional replica for each task. The IdealState can simply be changed as shown below.
-For increasing the availability of the system, its better to place replica of given partition on different nodes.
+For increasing the availability of the system, it\'s better to place the replica of a given partition on different nodes.
```
{
@@ -104,11 +104,11 @@ For increasing the availability of the system, its better to place replica of gi
}
```
-#### STATE
+### State
-Now lets take a slightly complicated scenario where a task represents a database. Unlike an index which is in general read only, database supports both reads and write. Keeping the data consistent among the replica is crucial in distributed data stores. One commonly applied technique is to assign one replica as MASTER and remaining as SLAVE. All writes go to MASTER and are then replicated to SLAVE.
+Now let\'s take a slightly complicated scenario where a task represents a database. Unlike an index which is in general read-only, a database supports both reads and writes. Keeping the data consistent among the replicas is crucial in distributed data stores. One commonly applied technique is to assign one replica as MASTER and remaining replicas as SLAVE. All writes go to the MASTER and are then replicated to the SLAVE replicas.
-Helix allows one to assign different states to each replica. Lets say you have two mysql instances N1 and N2 where one will serve as MASTER and another as SLAVE. The ideal state can be changed to
+Helix allows one to assign different states to each replica. Let\'s say you have two MySQL instances N1 and N2, where one will serve as MASTER and another as SLAVE. The IdealState can be changed to:
```
{
@@ -128,18 +128,17 @@ Helix allows one to assign different states to each replica. Lets say you have t
```
-### STATE MACHINE and TRANSITIONS
+### State Machine and Transitions
-Idealstate allows one to exactly specify the desired state of the cluster. Given an idealstate, Helix takes up the responsibility of ensuring that cluster reaches idealstate. Helix CONTROLLER reads the idealstate and then commands the PARTICIPANT to take appropriate actions to move from one state to another until it matches Idealstate. These actions are referred to as 'transitions' in Helix.
+IdealState allows one to exactly specify the desired state of the cluster. Given an IdealState, Helix takes up the responsibility of ensuring that the cluster reaches the IdealState. The Helix _controller_ reads the IdealState and then commands the Participant to take appropriate actions to move from one state to another until it matches the IdealState. These actions are referred to as _transitions_ in Helix.
-Next logical question is, how does the CONTROLLER compute the transitions required to get to idealstate. This is where finite state machine concept comes in. Helix allows applications to plug in FSM. A state machine consists of the following
+The next logical question is: how does the _controller_ compute the transitions required to get to IdealState? This is where the finite state machine concept comes in. Helix allows applications to plug in a finite state machine. A state machine consists of the following:
-* STATE : Describes the role of a replica
-* TRANSITION: An action that allows a replica to move from one STATE to another, thus changing its role.
+* State: Describes the role of a replica
+* Transition: An action that allows a replica to move from one State to another, thus changing its role.
Here is an example of MASTERSLAVE state machine,
-
```
OFFLINE | SLAVE | MASTER
_____________________________
@@ -155,7 +154,7 @@ MASTER | SLAVE | SLAVE | N/A |
```
-Helix allows each resource to be associated with one state machine. This means you can have one resource as a index and another as database in the same cluster. One can associate each resource with a state machine as follows
+Helix allows each resource to be associated with one state machine. This means you can have one resource as an index and another as a database in the same cluster. One can associate each resource with a state machine as follows:
```
{
@@ -175,12 +174,12 @@ Helix allows each resource to be associated with one state machine. This means y
```
-### CURRENT STATE
+### Current State
-Currentstate of a resource simply represents its actual state at a PARTICIPANT. In the below example,
+CurrentState of a resource simply represents its actual state at a PARTICIPANT. In the below example:
-* 'INSTANCE_NAME' : Unique name representing the process.
-* 'SESSION_ID': Id that is automatically assigned every time a process joins the cluster.
+* INSTANCE_NAME: Unique name representing the process
+* SESSION_ID: ID that is automatically assigned every time a process joins the cluster
```
{
@@ -203,11 +202,11 @@ Currentstate of a resource simply represents its actual state at a PARTICIPANT.
}
}
```
-Each node in the cluster has its own Current state.
+Each node in the cluster has its own CurrentState.
-### EXTERNAL VIEW
+### External View
-In order to communicate with the PARTICIPANTs, external clients need to know the current state of each of the PARTICIPANT. The external clients are referred to as SPECTATORS. In order to make the life of SPECTATOR simple, Helix provides EXTERNALVIEW that is an aggregated view of the current state across all nodes. The EXTERNALVIEW has similar format as IDEALSTATE.
+In order to communicate with the PARTICIPANTs, external clients need to know the current state of each of the PARTICIPANTs. The external clients are referred to as SPECTATORS. In order to make the life of SPECTATOR simple, Helix provides an EXTERNALVIEW that is an aggregated view of the current state across all nodes. The EXTERNALVIEW has a similar format as IDEALSTATE.
```
{
@@ -232,29 +231,29 @@ In order to communicate with the PARTICIPANTs, external clients need to know the
}
```
-### REBALANCER
+### Rebalancer
-The core component of Helix is the CONTROLLER which runs the REBALANCER algorithm on every cluster event. Cluster event can be one of the following
+The core component of Helix is the CONTROLLER which runs the REBALANCER algorithm on every cluster event. Cluster events can be one of the following:
* Nodes start/stop and soft/hard failures
* New nodes are added/removed
* Ideal state changes
-There are few more like config changes etc but the key point to take away is there are many ways to trigger the rebalancer.
+There are few more such as config changes, etc. The key takeaway: there are many ways to trigger the rebalancer.
-When a rebalancer is run it simply does the following
+When a rebalancer is run it simply does the following:
-* Compares the idealstate and current state
-* Computes the transitions required to reach the idealstate.
-* Issues the transitions to PARTICIPANT
+* Compares the IdealState and current state
+* Computes the transitions required to reach the IdealState
+* Issues the transitions to each PARTICIPANT
-The above steps happen for every change in the system. Once current state matches the idealstate the system is considered stable which implies 'IDEALSTATE = CURRENTSTATE = EXTERNALVIEW'
+The above steps happen for every change in the system. Once the current state matches the IdealState, the system is considered stable which implies \'IDEALSTATE = CURRENTSTATE = EXTERNALVIEW\'
-### DYNAMIC IDEALSTATE
+### Dynamic IdealState
-One of the things that makes Helix powerful is that idealstate can be changed dynamically. This means one can listen to cluster events like node failures and dynamically change the ideal state. Helix will then take care of triggering the respective transitions in the system.
+One of the things that makes Helix powerful is that IdealState can be changed dynamically. This means one can listen to cluster events like node failures and dynamically change the ideal state. Helix will then take care of triggering the respective transitions in the system.
-Helix comes with few algorithms to automatically compute the idealstate based on the constraints. For e.g. if you have a resource 3 partitions and 2 replicas, Helix can automatically compute the idealstate based on the nodes that are currently active. See features page to find out more about various execution modes of Helix like AUTO_REBALANCE, AUTO and CUSTOM.
+Helix comes with few algorithms to automatically compute the IdealState based on the constraints. For example, if you have a resource of 3 partitions and 2 replicas, Helix can automatically compute the IdealState based on the nodes that are currently active. See the [tutorial](./tutorial_rebalance.html) to find out more about various execution modes of Helix like AUTO_REBALANCE, AUTO and CUSTOM.
http://git-wip-us.apache.org/repos/asf/incubator-helix/blob/3aa36efb/src/site/markdown/Quickstart.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/Quickstart.md b/src/site/markdown/Quickstart.md
index 6807c77..65410c3 100644
--- a/src/site/markdown/Quickstart.md
+++ b/src/site/markdown/Quickstart.md
@@ -264,6 +264,36 @@ ExternalView for myDB:
{
"id" : "myDB",
"mapFields" : {
+ "myDB_0" : {
+ "localhost_12913" : "SLAVE",
+ "localhost_12914" : "MASTER",
+ "localhost_12915" : "SLAVE"
+ },
+ "myDB_1" : {
+ "localhost_12913" : "SLAVE",
+ "localhost_12914" : "SLAVE",
+ "localhost_12915" : "MASTER"
+ },
+ "myDB_2" : {
+ "localhost_12913" : "MASTER",
+ "localhost_12914" : "SLAVE",
+ "localhost_12915" : "SLAVE"
+ },
+ "myDB_3" : {
+ "localhost_12913" : "SLAVE",
+ "localhost_12914" : "SLAVE",
+ "localhost_12915" : "MASTER"
+ },
+ "myDB_4" : {
+ "localhost_12913" : "MASTER",
+ "localhost_12914" : "SLAVE",
+ "localhost_12915" : "SLAVE"
+ },
+ "myDB_5" : {
+ "localhost_12913" : "SLAVE",
+ "localhost_12914" : "MASTER",
+ "localhost_12915" : "SLAVE"
+ }
},
"listFields" : {
},
@@ -286,6 +316,14 @@ Next, we\'ll show how Helix does the work that you\'d otherwise have to build in
./helix-admin.sh --zkSvr localhost:2199 --addNode MYCLUSTER localhost:12917
./helix-admin.sh --zkSvr localhost:2199 --addNode MYCLUSTER localhost:12918
+And start up these instances:
+
+ # start up each instance. These are mock implementations that are actively managed by Helix
+ ./start-helix-participant.sh --zkSvr localhost:2199 --cluster MYCLUSTER --host localhost --port 12916 --stateModelType MasterSlave 2>&1 > /tmp/participant_12916.log
+ ./start-helix-participant.sh --zkSvr localhost:2199 --cluster MYCLUSTER --host localhost --port 12917 --stateModelType MasterSlave 2>&1 > /tmp/participant_12917.log
+ ./start-helix-participant.sh --zkSvr localhost:2199 --cluster MYCLUSTER --host localhost --port 12918 --stateModelType MasterSlave 2>&1 > /tmp/participant_12918.log
+
+
And now, let Helix do the work for you. To shift the work, simply rebalance. After the rebalance, each node will have one master and two slaves.
./helix-admin.sh --zkSvr localhost:2199 --rebalance MYCLUSTER myDB 3
@@ -354,6 +392,36 @@ ExternalView for myDB:
{
"id" : "myDB",
"mapFields" : {
+ "myDB_0" : {
+ "localhost_12913" : "SLAVE",
+ "localhost_12914" : "SLAVE",
+ "localhost_12917" : "MASTER"
+ },
+ "myDB_1" : {
+ "localhost_12916" : "SLAVE",
+ "localhost_12917" : "SLAVE",
+ "localhost_12918" : "MASTER"
+ },
+ "myDB_2" : {
+ "localhost_12913" : "MASTER",
+ "localhost_12917" : "SLAVE",
+ "localhost_12918" : "SLAVE"
+ },
+ "myDB_3" : {
+ "localhost_12915" : "MASTER",
+ "localhost_12917" : "SLAVE",
+ "localhost_12918" : "SLAVE"
+ },
+ "myDB_4" : {
+ "localhost_12916" : "MASTER",
+ "localhost_12917" : "SLAVE",
+ "localhost_12918" : "SLAVE"
+ },
+ "myDB_5" : {
+ "localhost_12913" : "SLAVE",
+ "localhost_12914" : "MASTER",
+ "localhost_12915" : "SLAVE"
+ }
},
"listFields" : {
},
@@ -369,9 +437,7 @@ Mission accomplished. The partitions are nicely balanced.
Building a fault tolerant system isn\'t trivial, but with Helix, it\'s easy. Helix detects a failed instance, and triggers mastership transfer automatically.
-First, let's fail an instance:
-
-_KILL A NODE (need to find the pid via listInstanceInfo)_
+First, let's fail an instance. In this example, we\'ll kill localhost:12918 to simulate a failure.
We lost localhost:12918, so myDB_1 lost its MASTER. Helix can fix that, it will transfer mastership to a healthy node that is currently a SLAVE, say localhost:12197. Helix balances the load as best as it can, given there are 6 partitions on 5 nodes. Let\'s see:
@@ -390,23 +456,23 @@ IdealState for myDB:
},
"myDB_1" : {
"localhost_12916" : "SLAVE",
- "localhost_12917" : "MASTER"
- "localhost_12918" : "OFFLINE"
+ "localhost_12917" : "SLAVE",
+ "localhost_12918" : "MASTER"
},
"myDB_2" : {
"localhost_12913" : "MASTER",
"localhost_12917" : "SLAVE",
- "localhost_12918" : "OFFLINE"
+ "localhost_12918" : "SLAVE"
},
"myDB_3" : {
"localhost_12915" : "MASTER",
"localhost_12917" : "SLAVE",
- "localhost_12918" : "OFFLINE"
+ "localhost_12918" : "SLAVE"
},
"myDB_4" : {
"localhost_12916" : "MASTER",
"localhost_12917" : "SLAVE",
- "localhost_12918" : "OFFLINE"
+ "localhost_12918" : "SLAVE"
},
"myDB_5" : {
"localhost_12913" : "SLAVE",
@@ -416,9 +482,9 @@ IdealState for myDB:
},
"listFields" : {
"myDB_0" : [ "localhost_12917", "localhost_12913", "localhost_12914" ],
- "myDB_1" : [ "localhost_12917", "localhost_12916", "localhost_12918" ],
- "myDB_2" : [ "localhost_12913", "localhost_12917", "localhost_12918" ],
- "myDB_3" : [ "localhost_12915", "localhost_12917", "localhost_12918" ],
+ "myDB_1" : [ "localhost_12918", "localhost_12917", "localhost_12916" ],
+ "myDB_2" : [ "localhost_12913", "localhost_12918", "localhost_12917" ],
+ "myDB_3" : [ "localhost_12915", "localhost_12918", "localhost_12917" ],
"myDB_4" : [ "localhost_12916", "localhost_12917", "localhost_12918" ],
"myDB_5" : [ "localhost_12914", "localhost_12915", "localhost_12913" ]
},
@@ -435,6 +501,32 @@ ExternalView for myDB:
{
"id" : "myDB",
"mapFields" : {
+ "myDB_0" : {
+ "localhost_12913" : "SLAVE",
+ "localhost_12914" : "SLAVE",
+ "localhost_12917" : "MASTER"
+ },
+ "myDB_1" : {
+ "localhost_12916" : "SLAVE",
+ "localhost_12917" : "MASTER"
+ },
+ "myDB_2" : {
+ "localhost_12913" : "MASTER",
+ "localhost_12917" : "SLAVE"
+ },
+ "myDB_3" : {
+ "localhost_12915" : "MASTER",
+ "localhost_12917" : "SLAVE"
+ },
+ "myDB_4" : {
+ "localhost_12916" : "MASTER",
+ "localhost_12917" : "SLAVE"
+ },
+ "myDB_5" : {
+ "localhost_12913" : "SLAVE",
+ "localhost_12914" : "MASTER",
+ "localhost_12915" : "SLAVE"
+ }
},
"listFields" : {
},
http://git-wip-us.apache.org/repos/asf/incubator-helix/blob/3aa36efb/src/site/markdown/index.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/index.md b/src/site/markdown/index.md
index cbcb6ba..f07c9cb 100644
--- a/src/site/markdown/index.md
+++ b/src/site/markdown/index.md
@@ -44,6 +44,8 @@ Navigating the Documentation
[Service discovery](./recipes/service_discovery.html)
+[Distributed Task DAG Execution](./task_dag_execution.html)
+
What Is Helix
--------------
http://git-wip-us.apache.org/repos/asf/incubator-helix/blob/3aa36efb/src/site/markdown/tutorial_participant.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/tutorial_participant.md b/src/site/markdown/tutorial_participant.md
index 49d0701..19e6f98 100644
--- a/src/site/markdown/tutorial_participant.md
+++ b/src/site/markdown/tutorial_participant.md
@@ -41,7 +41,9 @@ After the Helix manager instance is created, only thing that needs to be registe
The methods of the State Model will be called when controller sends transitions to the Participant. In this example, we'll use the OnlineOffline factory. Other options include:
* MasterSlaveStateModelFactory
-* _NEED TO COMPLETE THIS LIST_
+* LeaderStandbyStateModelFactory
+* BootstrapHandler
+
```
manager = HelixManagerFactory.getZKHelixManager(clusterName,
http://git-wip-us.apache.org/repos/asf/incubator-helix/blob/3aa36efb/src/site/markdown/tutorial_spectator.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/tutorial_spectator.md b/src/site/markdown/tutorial_spectator.md
index 03fc45a..a5b9a0e 100644
--- a/src/site/markdown/tutorial_spectator.md
+++ b/src/site/markdown/tutorial_spectator.md
@@ -36,10 +36,7 @@ It requires the following parameters:
* ADMIN: To carry out system admin actions.
* zkConnectString: Connection string to Zookeeper. This is of the form host1:port1,host2:port2,host3:port3.
-After the Helix manager instance is created, only thing that needs to be registered is the state model factory.
-The methods of the State Model will be called when controller sends transitions to the Participant. In this example, we'll use the OnlineOffline factory. Other options:
-
-_NEED TO ADD OTHER OPTIONS HERE_
+After the Helix manager instance is created, only thing that needs to be registered is the listener. When the ExternalView changes, the listener is notified.
### Spectator Code
@@ -73,6 +70,3 @@ result = theInstance.sendRequest(yourApplicationRequest, responseObject);
When the external view changes, the application needs to react by sending requests to a different instance.
-_SHOULD WE ADD AN EXAMPLE OF THIS? IT WOULD MAKE THINGS MORE COMPLICATED, BECAUSE THE APP SHOULD HAVE A LOCAL COPY OF THE ROUTING TABLE AND UPDATE IT ONLY WHEN THE EXTERNAL VIEW CHANGES, THAT\'S HOW I'D IMPLEMENT IT._
-
-
http://git-wip-us.apache.org/repos/asf/incubator-helix/blob/3aa36efb/src/site/markdown/tutorial_throttling.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/tutorial_throttling.md b/src/site/markdown/tutorial_throttling.md
index 0fcc9bc..5002156 100644
--- a/src/site/markdown/tutorial_throttling.md
+++ b/src/site/markdown/tutorial_throttling.md
@@ -32,4 +32,3 @@ Helix allows applications to set a threshold on transitions. The threshold can b
* Resource e.g database
* Node i.e per-node maximum transitions in parallel
-_I can\'t find this code._ See HelixManager.getHelixAdmin.addMessageConstraint()