[jira] (ZOOKEEPER-2678) Large databases take a long time to regain a quorum

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847250#comment-15847250 ] 

ASF GitHub Bot commented on ZOOKEEPER-2678:
-------------------------------------------

Github user hanm commented on a diff in the pull request:

    https://github.com/apache/zookeeper/pull/157#discussion_r98734336
  
    --- Diff: src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java ---
    @@ -507,9 +507,12 @@ public synchronized void shutdown() {
             if (firstProcessor != null) {
                 firstProcessor.shutdown();
             }
    -        if (zkDb != null) {
    -            zkDb.clear();
    -        }
    +
    +        // There is no need to clear the database
    +        //  * When a new quorum is established we can still apply the diff
    +        //    on top of the same zkDb data
    +        //  * If we fetch a new snapshot from leader, the zkDb will be
    +        //    cleared anyway before loading the snapshot
     
    --- End diff --
    
    There is one case we may still want to clear db here - when one of the ZooKeeper critical threads (such as * processors, session trackers) fail, ZooKeeper server will shutdown (see runFromConfig) and consequently invoke ZooKeeper#shutdown. In this case, I don't see a particular reason not to clear the db, though not doing it might be fine (as one could argue the server will be dead anyway), but I tend to lean towards the safe side on cleaning the db. One way to conditionally do that is to add a Boolean parameter to ZooKeeper#shutdown so we can have fine grained control over when to clear db in what code path.


> Large databases take a long time to regain a quorum
> ---------------------------------------------------
>
>                 Key: ZOOKEEPER-2678
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2678
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.4.9, 3.5.2
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>
> I know this is long but please here me out.
> I recently inherited a massive zookeeper ensemble.  The snapshot is 3.4 GB on disk.  Because of its massive size we have been running into a number of issues. There are lots of problems that we hope to fix with tuning GC etc, but the big one right now that is blocking us making a lot of progress on the rest of them is that when we lose a quorum because the leader left, for what ever reason, it can take well over 5 mins for a new quorum to be established.  So we cannot tune the leader without risking downtime.
> We traced down where the time was being spent and found that each server was clearing the database so it would be read back in again before leader election even started.  Then as part of the sync phase each server will write out a snapshot to checkpoint the progress it made as part of the sync.
> I will be putting up a patch shortly with some proposed changes in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)