You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Manikumar <ma...@gmail.com> on 2019/12/07 18:16:51 UTC
Data loss after upgrading ZooKeeper 3.4.14 to 3.5.6
Hi all,
I tried to upgrade single node ZK server from 3.4.14 to 3.5.6. There were
no snapshot files, so as suggested in ZOOKEEPER-3056
<https://issues.apache.org/jira/browse/ZOOKEEPER-3056>,
I have set snapshot.trust.empty to true. After server startup, when I tried
to list the znodes, I found that znodes are missing.
Looking into the source code,
https://github.com/apache/zookeeper/blob/release-3.5.6/zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileTxnSnapLog.java#L224
looks like we are not reading transaction log if there are no snapshot
files and snapshot.trust.empty is set to true.
Appreciate any thoughts on this?
ZK 3.5.6 logs:
[2019-12-07 12:13:35,007] INFO Created server with tickTime 3000
minSessionTimeout 6000 maxSessionTimeout 60000 datadir
/var/lib/zookeeper/version-2 snapdir /var/lib/zookeeper/version-2
(org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-07 12:13:35,012] INFO Using
org.apache.zookeeper.server.NIOServerCnxnFactory as server connection
factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2019-12-07 12:13:35,014] INFO Configuring NIO connection handler with 10s
sessionless connection timeout, 1 selector thread(s), 12 worker threads,
and 64 kB direct buffers. (org.apache
.zookeeper.server.NIOServerCnxnFactory)
[2019-12-07 12:13:35,017] INFO binding to port 0.0.0.0/0.0.0.0:2181
(org.apache.zookeeper.server.NIOServerCnxnFactory)
[2019-12-07 12:13:35,027] INFO zookeeper.snapshotSizeFactor = 0.33
(org.apache.zookeeper.server.ZKDatabase)
[2019-12-07 12:13:35,029] DEBUG Created new input stream
/var/lib/zookeeper/version-2/log.1
(org.apache.zookeeper.server.persistence.FileTxnLog)
[2019-12-07 12:13:35,031] DEBUG Created new input archive
/var/lib/zookeeper/version-2/log.1
(org.apache.zookeeper.server.persistence.FileTxnLog)
[2019-12-07 12:13:35,035] DEBUG EOF exception java.io.EOFException: Failed
to read /var/lib/zookeeper/version-2/log.1
(org.apache.zookeeper.server.persistence.FileTxnLog)
[2019-12-07 12:13:35,035] WARN No snapshot found, but there are log
entries. This should only be allowed during upgrading.
(org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2019-12-07 12:13:35,035] INFO Snapshotting: 0x0 to
/var/lib/zookeeper/version-2/snapshot.0
(org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2019-12-07 12:13:35,036] INFO Snapshotting: 0x0 to
/var/lib/zookeeper/version-2/snapshot.0
(org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2019-12-07 12:13:35,050] INFO Using checkIntervalMs=60000
maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)
[2019-12-07 12:15:07,137] DEBUG Accepted socket connection from /
127.0.0.1:38888 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2019-12-07 12:15:07,144] DEBUG Session establishment request from client /
127.0.0.1:38888 client's lastZxid is 0x0
(org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-07 12:15:07,145] DEBUG Adding session 0x100006e15fb0000
(org.apache.zookeeper.server.SessionTrackerImpl)
[2019-12-07 12:15:07,148] TRACE SessionTrackerImpl --- Adding session
0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
[2019-12-07 12:15:07,149] DEBUG Client attempting to establish new session:
session = 0x100006e15fb0000, zxid = 0x0, timeout = 30000, address = /
127.0.0.1:38888 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-07 12:15:07,155] TRACE :Psessionid:0x100006e15fb0000
type:createSession cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown
reqpath:n/a (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-12-07 12:15:07,155] TRACE SessionTrackerImpl --- Existing session
0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
[2019-12-07 12:15:07,155] INFO Creating new log file: log.1
(org.apache.zookeeper.server.persistence.FileTxnLog)
[2019-12-07 12:15:07,170] DEBUG Processing request::
sessionid:0x100006e15fb0000 type:createSession cxid:0x0 zxid:0x1
txntype:-10 reqpath:n/a (org.apache.zookeeper.server.FinalRequestProcessor)
Thanks,
Manikumar
Re: Data loss after upgrading ZooKeeper 3.4.14 to 3.5.6
Posted by Manikumar <ma...@gmail.com>.
Filed a JIRA here : https://issues.apache.org/jira/browse/ZOOKEEPER-3644
Thanks,
Manikumar
On Sat, Dec 7, 2019 at 11:46 PM Manikumar <ma...@gmail.com> wrote:
> Hi all,
>
> I tried to upgrade single node ZK server from 3.4.14 to 3.5.6. There were
> no snapshot files, so as suggested in ZOOKEEPER-3056
> <https://issues.apache.org/jira/browse/ZOOKEEPER-3056>,
> I have set snapshot.trust.empty to true. After server startup, when I
> tried to list the znodes, I found that znodes are missing.
>
> Looking into the source code,
>
> https://github.com/apache/zookeeper/blob/release-3.5.6/zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileTxnSnapLog.java#L224
>
> looks like we are not reading transaction log if there are no snapshot
> files and snapshot.trust.empty is set to true.
>
> Appreciate any thoughts on this?
>
> ZK 3.5.6 logs:
> [2019-12-07 12:13:35,007] INFO Created server with tickTime 3000
> minSessionTimeout 6000 maxSessionTimeout 60000 datadir
> /var/lib/zookeeper/version-2 snapdir /var/lib/zookeeper/version-2
> (org.apache.zookeeper.server.ZooKeeperServer)
> [2019-12-07 12:13:35,012] INFO Using
> org.apache.zookeeper.server.NIOServerCnxnFactory as server connection
> factory (org.apache.zookeeper.server.ServerCnxnFactory)
> [2019-12-07 12:13:35,014] INFO Configuring NIO connection handler with 10s
> sessionless connection timeout, 1 selector thread(s), 12 worker threads,
> and 64 kB direct buffers. (org.apache
> .zookeeper.server.NIOServerCnxnFactory)
> [2019-12-07 12:13:35,017] INFO binding to port 0.0.0.0/0.0.0.0:2181
> (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2019-12-07 12:13:35,027] INFO zookeeper.snapshotSizeFactor = 0.33
> (org.apache.zookeeper.server.ZKDatabase)
> [2019-12-07 12:13:35,029] DEBUG Created new input stream
> /var/lib/zookeeper/version-2/log.1
> (org.apache.zookeeper.server.persistence.FileTxnLog)
> [2019-12-07 12:13:35,031] DEBUG Created new input archive
> /var/lib/zookeeper/version-2/log.1
> (org.apache.zookeeper.server.persistence.FileTxnLog)
> [2019-12-07 12:13:35,035] DEBUG EOF exception java.io.EOFException: Failed
> to read /var/lib/zookeeper/version-2/log.1
> (org.apache.zookeeper.server.persistence.FileTxnLog)
> [2019-12-07 12:13:35,035] WARN No snapshot found, but there are log
> entries. This should only be allowed during upgrading.
> (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
> [2019-12-07 12:13:35,035] INFO Snapshotting: 0x0 to
> /var/lib/zookeeper/version-2/snapshot.0
> (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
> [2019-12-07 12:13:35,036] INFO Snapshotting: 0x0 to
> /var/lib/zookeeper/version-2/snapshot.0
> (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
> [2019-12-07 12:13:35,050] INFO Using checkIntervalMs=60000
> maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)
> [2019-12-07 12:15:07,137] DEBUG Accepted socket connection from /
> 127.0.0.1:38888 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2019-12-07 12:15:07,144] DEBUG Session establishment request from client /
> 127.0.0.1:38888 client's lastZxid is 0x0
> (org.apache.zookeeper.server.ZooKeeperServer)
> [2019-12-07 12:15:07,145] DEBUG Adding session 0x100006e15fb0000
> (org.apache.zookeeper.server.SessionTrackerImpl)
> [2019-12-07 12:15:07,148] TRACE SessionTrackerImpl --- Adding session
> 0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
> [2019-12-07 12:15:07,149] DEBUG Client attempting to establish new
> session: session = 0x100006e15fb0000, zxid = 0x0, timeout = 30000, address
> = /127.0.0.1:38888 (org.apache.zookeeper.server.ZooKeeperServer)
> [2019-12-07 12:15:07,155] TRACE :Psessionid:0x100006e15fb0000
> type:createSession cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown
> reqpath:n/a (org.apache.zookeeper.server.PrepRequestProcessor)
> [2019-12-07 12:15:07,155] TRACE SessionTrackerImpl --- Existing session
> 0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
> [2019-12-07 12:15:07,155] INFO Creating new log file: log.1
> (org.apache.zookeeper.server.persistence.FileTxnLog)
> [2019-12-07 12:15:07,170] DEBUG Processing request::
> sessionid:0x100006e15fb0000 type:createSession cxid:0x0 zxid:0x1
> txntype:-10 reqpath:n/a (org.apache.zookeeper.server.FinalRequestProcessor)
>
>
> Thanks,
> Manikumar
>