[jira] [Commented] (HELIX-54) make helix-admin operations atomic

["dafu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HELIX-54?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721284#comment-13721284 ] 

dafu commented on HELIX-54:
---------------------------

Current helix admin api's are not atomic: i) no sync on invoking from multiple processes, and ii) not handling zk session expiry (due to java gc).

To improve it, we have two options:

a) Making use of zookeeper transaction: http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction()
This requires upgrade zookeeper jar to 3.4.0+ and rewrite all helix admin api's using zookeeper#transaction()

b) Implement distributed lock
This is a more general solution but has some limitations. The basic idea is:

1) the 1st process creates a permanent znode representing holding the lock; all other processes wait on the removal of the znode
2) the 1st process performs the helix operation
3) the 1st process removes the znode.
4) all other processes get notified and proceed

Note that we can't use an ephemeral znode, since if the process creating the ephemeral znode experiences a long gc, the ephemeral node will be removed. some other process will create a new ephemeral node and grabs the lock. at this time, if the formal process recovers from gc, it would think it has the lock also.

Based on distributed-lock, all helix admin api that needs sync on multiple processes could use the following idiom:

DistributedLock lock = new DistributedLock(String namespace);
if (lock.tryLock(timeout-value, time-unit)  // wait until locking successfully or timeout
{
  if (zookeeper content shows operation not successfully performed) {
    // clean up if necessary
    // call helix admin api
  }
} finally {
  lock.unlock();
}
 
if (zookeeper content shows operation not successfully performed) {
  // fail permanently
}
 

a) Handling multi-process: the helix admin operation is sync on the lock, only the 1st process can grab the lock and perform the admin operation.
b) Handling zk session expiry (due to java gc): if the 1st process holds the lock and gets a zk session expiry, then all other processes will wait on the lock until timeout. If zk session expiry recovers before timeout, the 1st process will continue finish the helix operation, otherwise fails permanently.

Note that the distributed-lock solution is not fault-tolerant. if the 1st process crashes or experience a long gc, the helix admin operation will fail permanently. if this happens, we also need a way to clean orphan znodes.

                
> make helix-admin operations atomic
> ----------------------------------
>
>                 Key: HELIX-54
>                 URL: https://issues.apache.org/jira/browse/HELIX-54
>             Project: Apache Helix
>          Issue Type: Bug
>    Affects Versions: 0.6.0-incubating
>         Environment: make helix-admin operations e.g. addCluster() multi-process safe. 
>            Reporter: dafu
>            Assignee: dafu
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira