[jira] [Created] (ZOOKEEPER-3513) Zookeeper upgrade fails due to missing snapshots

["Stephan Huttenhuis (Jira)" <ji...@apache.org>]

Stephan Huttenhuis created ZOOKEEPER-3513:
---------------------------------------------

             Summary: Zookeeper upgrade fails due to missing snapshots
                 Key: ZOOKEEPER-3513
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3513
             Project: ZooKeeper
          Issue Type: Bug
          Components: server
    Affects Versions: 3.5.4, 3.6.0
            Reporter: Stephan Huttenhuis


In ZOOKEEPER-2325 a check was added that requires a snapshot when loading data. We have been running 3-node ensembles on Zookeeper 3.4.13 for about 5 months for use with Solr Cloud. During this time some ensembles created a few snapshots but other didn't generate any. Because of this upgrading to e.g. 3.5.5 fails.

Either it is perfectly possible for Zookeeper data to have no snapshots or something is going wrong with generating snapshots. The ensembles are straightforward.
 - The following stack occurs:
{noformat}
java.io.IOException: No snapshot found, but there are log entries. Something is broken!
	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:211)
	at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
	at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
	at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
	at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
	at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
	at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
	at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
{noformat}

 - The zoo.cfg
{noformat}
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/data/zookeeper/data
# the port at which the clients will connect
clientPort=2181

server.1=myserver1:2888:3888
server.2=myserver1:2888:3888
server.3=myserver1:2888:3888
{noformat}
 

 - The contents of /data/zookeeper/data/version-2
{noformat}
-rw-r--r-- 1 zookeeper zookeeper    1 Aug  7 21:50 acceptedEpoch
-rw-r--r-- 1 zookeeper zookeeper    1 Aug  8 20:38 currentEpoch
-rw-r--r-- 1 zookeeper zookeeper  65M Apr  1 14:44 log.1
-rw-r--r-- 1 zookeeper zookeeper  65M May 15 23:30 log.100000001
-rw-r--r-- 1 zookeeper zookeeper  65M Jul  3 23:21 log.100001645
-rw-r--r-- 1 zookeeper zookeeper  65M Aug  8 20:37 log.300000802
-rw-r--r-- 1 zookeeper zookeeper  65M Aug 20 13:58 log.70000062a
-rw-r--r-- 1 zookeeper zookeeper  65M Apr  4 21:22 log.f0
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)