You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "Aldan Brito (Jira)" <ji...@apache.org> on 2020/05/12 09:06:00 UTC
[jira] [Updated] (ZOOKEEPER-3826) upgrade from 3.4.x to 3.5.x
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aldan Brito updated ZOOKEEPER-3826:
-----------------------------------
Description:
upgrade of zookeeper from 3.4.14 to 3.5.7
We faced the snapshot issue which is described in https://issues.apache.org/jira/browse/ZOOKEEPER-3056
After setting the property "snapshot.trust.empty=true" the upgrade was successful.
while reverting the "snapshot.trust.empty=false" flag and restart of the zookeeper pods, one of the zookeeper server is failing with the similar stack trace no snapshot found.
{code:java}
{"type":"log", "host":"zk-testzk-0", "level":"ERROR", "neid":"zookeeper-4636c00bfc3849e0be179bc71cef17f8", "system":"zookeeper", "time":"2020-05-12T08:32:17.685Z", "timezone":"UTC", "log":{"message":"main - org.apache.zookeeper.server.quorum.QuorumPeer - Unable to load database on disk"}}
java.io.IOException: No snapshot found, but there are log entries. Something is broken!
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:901)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:887)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:205)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
{"type":"log", "host":"zk-testzk-0", "level":"ERROR", "neid":"zookeeper-4636c00bfc3849e0be179bc71cef17f8", "system":"zookeeper", "time":"2020-05-12T08:32:17.764Z", "timezone":"UTC", "log":{"message":"main - org.apache.zookeeper.server.quorum.QuorumPeerMain - Unexpected exception, exiting abnormally"}}
java.lang.RuntimeException: Unable to run quorum server
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:938)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:887)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:205)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.io.IOException: No snapshot found, but there are log entries. Something is broken!
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:901)
{code}
was:
upgrade of zookeeper from 3.4.14 to 3.5.7
We faced the snapshot issue which is described in https://issues.apache.org/jira/browse/ZOOKEEPER-3056
After setting the property "snapshot.trust.empty=true" the upgrade was successful.
while reverting the "snapshot.trust.empty=false" flag one of the zookeeper server is failing with the similar stack trace no snapshot found.
{code:java}
{"type":"log", "host":"zk-testzk-0", "level":"ERROR", "neid":"zookeeper-4636c00bfc3849e0be179bc71cef17f8", "system":"zookeeper", "time":"2020-05-12T08:32:17.685Z", "timezone":"UTC", "log":{"message":"main - org.apache.zookeeper.server.quorum.QuorumPeer - Unable to load database on disk"}}
java.io.IOException: No snapshot found, but there are log entries. Something is broken!
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:901)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:887)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:205)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
{"type":"log", "host":"zk-testzk-0", "level":"ERROR", "neid":"zookeeper-4636c00bfc3849e0be179bc71cef17f8", "system":"zookeeper", "time":"2020-05-12T08:32:17.764Z", "timezone":"UTC", "log":{"message":"main - org.apache.zookeeper.server.quorum.QuorumPeerMain - Unexpected exception, exiting abnormally"}}
java.lang.RuntimeException: Unable to run quorum server
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:938)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:887)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:205)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.io.IOException: No snapshot found, but there are log entries. Something is broken!
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:901)
{code}
> upgrade from 3.4.x to 3.5.x
> ---------------------------
>
> Key: ZOOKEEPER-3826
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3826
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.5.7
> Environment: Kuberenetes
> Reporter: Aldan Brito
> Priority: Critical
>
> upgrade of zookeeper from 3.4.14 to 3.5.7
> We faced the snapshot issue which is described in https://issues.apache.org/jira/browse/ZOOKEEPER-3056
> After setting the property "snapshot.trust.empty=true" the upgrade was successful.
> while reverting the "snapshot.trust.empty=false" flag and restart of the zookeeper pods, one of the zookeeper server is failing with the similar stack trace no snapshot found.
> {code:java}
> {"type":"log", "host":"zk-testzk-0", "level":"ERROR", "neid":"zookeeper-4636c00bfc3849e0be179bc71cef17f8", "system":"zookeeper", "time":"2020-05-12T08:32:17.685Z", "timezone":"UTC", "log":{"message":"main - org.apache.zookeeper.server.quorum.QuorumPeer - Unable to load database on disk"}}
> java.io.IOException: No snapshot found, but there are log entries. Something is broken!
> at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
> at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
> at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:901)
> at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:887)
> at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:205)
> at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
> at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
> {"type":"log", "host":"zk-testzk-0", "level":"ERROR", "neid":"zookeeper-4636c00bfc3849e0be179bc71cef17f8", "system":"zookeeper", "time":"2020-05-12T08:32:17.764Z", "timezone":"UTC", "log":{"message":"main - org.apache.zookeeper.server.quorum.QuorumPeerMain - Unexpected exception, exiting abnormally"}}
> java.lang.RuntimeException: Unable to run quorum server
> at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:938)
> at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:887)
> at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:205)
> at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
> at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
> Caused by: java.io.IOException: No snapshot found, but there are log entries. Something is broken!
> at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
> at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
> at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:901)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)