[jira] [Commented] (ACCUMULO-1572) single node zookeeper failure kills connected accumulo servers

["C Drew Thornton (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ACCUMULO-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13709824#comment-13709824 ] 

C Drew Thornton commented on ACCUMULO-1572:
-------------------------------------------

There are two configurations where I have experienced this:

1) The Zookeeper leader is removed from the ensemble and a new leader is elected.
  Result: Master immediately goes down along with the tablet servers.

2) According to http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html, the leader should not serve clients in 5+ node ensemble (zoo.cfg - leaderServes=no). Accumulo does not seem to anticipate this situation, so perhaps when a node is removed, the clients attempt to reestablish connection at the leader, who is not serving, and they fail. This may appear as two failures in the ensemble.
  Result: Tablet servers die who were clients of the removed node. If the master was one of the clients, then the cluster goes down.

Versions:
Zookeeper 3.4.5 - 5 nodes with 4 processors and 6GB system memory (-Xmx4096m -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC)
CDH 4.3 - MRv1, no YARN - no special configuration
Accumulo 1.5.0 - adjusted for system memory
                
> single node zookeeper failure kills connected accumulo servers
> --------------------------------------------------------------
>
>                 Key: ACCUMULO-1572
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1572
>             Project: Accumulo
>          Issue Type: Bug
>          Components: master, tserver
>    Affects Versions: 1.5.0
>            Reporter: Eric Newton
>            Priority: Blocker
>             Fix For: 1.5.1
>
>
> Drew Thornton writes on the user mailing list:
> {quote}
> If one zookeeper node is shutdown/fails/whatever and the rest of the ensemble stays up, the tablet servers attached as clients to the shutdown node immediately fail. If one of the clients happens to be the master, the cluster goes down.
> Accumulo does not seem to be failing over to the remaining zookeeper nodes, and this causes me to restart the individual tablet servers again.
> The zookeeper ensemble is very stable and has plenty of bandwidth/memory/processing, so taking one node down out of five doesn't crash the zookeepers, just the tablet servers...
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira