You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "Manikumar (Jira)" <ji...@apache.org> on 2019/12/10 15:39:00 UTC

[jira] [Updated] (ZOOKEEPER-3644) Data loss after upgrading 3.4.14 to 3.5.6 with snapshot.trust.empty=true

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Manikumar updated ZOOKEEPER-3644:
---------------------------------
    Description: 
We have tried to upgrade single node ZK server from 3.4.14 to 3.5.6.  There were no snapshot files, so as suggested in ZOOKEEPER-3056,
 I have set snapshot.trust.empty to true. After server startup, when I tried to list the znodes, I found that znodes are missing.

Steps to reproduce:
 # Start a single node ZK 3.4.14 server and create few znodes
 # Upgrade the server to 3.5.6 with  snapshot.trust.empty=true config
 # try to list the znodes using zkShell

Looking into the [source code|https://github.com/apache/zookeeper/blob/release-3.5.6/zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileTxnSnapLog.java#L224],
 looks like we are not reading transaction log if there are no snapshot files and snapshot.trust.empty is set to true.

ZK 3.5.6 logs:

{quote}
 [2019-12-07 12:13:35,007] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /var/lib/zookeeper/version-2 snapdir /var/lib/zookeeper/version-2
  (org.apache.zookeeper.server.ZooKeeperServer)
 [2019-12-07 12:13:35,012] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
 [2019-12-07 12:13:35,014] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 12 worker threads, and 64 kB direct buffers. (org.apache
 .zookeeper.server.NIOServerCnxnFactory)
 [2019-12-07 12:13:35,017] INFO binding to port [0.0.0.0/0.0.0.0:2181|http://0.0.0.0/0.0.0.0:2181] (org.apache.zookeeper.server.NIOServerCnxnFactory)
 [2019-12-07 12:13:35,027] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
 [2019-12-07 12:13:35,029] DEBUG Created new input stream /var/lib/zookeeper/version-2/log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
 [2019-12-07 12:13:35,031] DEBUG Created new input archive /var/lib/zookeeper/version-2/log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
 [2019-12-07 12:13:35,035] DEBUG EOF exception java.io.EOFException: Failed to read /var/lib/zookeeper/version-2/log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
 [2019-12-07 12:13:35,035] WARN No snapshot found, but there are log entries. This should only be allowed during upgrading. (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
 [2019-12-07 12:13:35,035] INFO Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
 [2019-12-07 12:13:35,036] INFO Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
 [2019-12-07 12:13:35,050] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)
 [2019-12-07 12:15:07,137] DEBUG Accepted socket connection from /[127.0.0.1:38888|http://127.0.0.1:38888/] (org.apache.zookeeper.server.NIOServerCnxnFactory)
 [2019-12-07 12:15:07,144] DEBUG Session establishment request from client /[127.0.0.1:38888|http://127.0.0.1:38888/] client's lastZxid is 0x0 (org.apache.zookeeper.server.ZooKeeperServer)
 [2019-12-07 12:15:07,145] DEBUG Adding session 0x100006e15fb0000 (org.apache.zookeeper.server.SessionTrackerImpl)
 [2019-12-07 12:15:07,148] TRACE SessionTrackerImpl — Adding session 0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
 [2019-12-07 12:15:07,149] DEBUG Client attempting to establish new session: session = 0x100006e15fb0000, zxid = 0x0, timeout = 30000, address = /[127.0.0.1:38888|http://127.0.0.1:38888/] (org.apache.zookeeper.server.ZooKeeperServer)
 [2019-12-07 12:15:07,155] TRACE :Psessionid:0x100006e15fb0000 type:createSession cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a (org.apache.zookeeper.server.PrepRequestProcessor)
 [2019-12-07 12:15:07,155] TRACE SessionTrackerImpl — Existing session 0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
 [2019-12-07 12:15:07,155] INFO Creating new log file: log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
 [2019-12-07 12:15:07,170] DEBUG Processing request:: sessionid:0x100006e15fb0000 type:createSession cxid:0x0 zxid:0x1 txntype:-10 reqpath:n/a (org.apache.zookeeper.server.FinalRequestProcessor)

{quote}

  was:
We have tried to upgrade single node ZK server from 3.4.14 to 3.5.6.  There were no snapshot files, so as suggested in ZOOKEEPER-3056,
 I have set snapshot.trust.empty to true. After server startup, when I tried to list the znodes, I found that znodes are missing.

Steps to reproduce:
 # Start a single node ZK 3.4.14 server and create few znodes
 # Upgrade the server to 3.5.6 with  snapshot.trust.empty=true config
 # try to list the znodes using zkShell

Looking into the [source code|https://github.com/apache/zookeeper/blob/release-3.5.6/zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileTxnSnapLog.java#L224],
 looks like we are not reading transaction log if there are no snapshot files and snapshot.trust.empty is set to true.

ZK 3.5.6 logs:
 [2019-12-07 12:13:35,007] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /var/lib/zookeeper/version-2 snapdir /var/lib/zookeeper/version-2
  (org.apache.zookeeper.server.ZooKeeperServer)
 [2019-12-07 12:13:35,012] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
 [2019-12-07 12:13:35,014] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 12 worker threads, and 64 kB direct buffers. (org.apache
 .zookeeper.server.NIOServerCnxnFactory)
 [2019-12-07 12:13:35,017] INFO binding to port [0.0.0.0/0.0.0.0:2181|http://0.0.0.0/0.0.0.0:2181] (org.apache.zookeeper.server.NIOServerCnxnFactory)
 [2019-12-07 12:13:35,027] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
 [2019-12-07 12:13:35,029] DEBUG Created new input stream /var/lib/zookeeper/version-2/log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
 [2019-12-07 12:13:35,031] DEBUG Created new input archive /var/lib/zookeeper/version-2/log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
 [2019-12-07 12:13:35,035] DEBUG EOF exception java.io.EOFException: Failed to read /var/lib/zookeeper/version-2/log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
 [2019-12-07 12:13:35,035] WARN No snapshot found, but there are log entries. This should only be allowed during upgrading. (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
 [2019-12-07 12:13:35,035] INFO Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
 [2019-12-07 12:13:35,036] INFO Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
 [2019-12-07 12:13:35,050] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)
 [2019-12-07 12:15:07,137] DEBUG Accepted socket connection from /[127.0.0.1:38888|http://127.0.0.1:38888/] (org.apache.zookeeper.server.NIOServerCnxnFactory)
 [2019-12-07 12:15:07,144] DEBUG Session establishment request from client /[127.0.0.1:38888|http://127.0.0.1:38888/] client's lastZxid is 0x0 (org.apache.zookeeper.server.ZooKeeperServer)
 [2019-12-07 12:15:07,145] DEBUG Adding session 0x100006e15fb0000 (org.apache.zookeeper.server.SessionTrackerImpl)
 [2019-12-07 12:15:07,148] TRACE SessionTrackerImpl — Adding session 0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
 [2019-12-07 12:15:07,149] DEBUG Client attempting to establish new session: session = 0x100006e15fb0000, zxid = 0x0, timeout = 30000, address = /[127.0.0.1:38888|http://127.0.0.1:38888/] (org.apache.zookeeper.server.ZooKeeperServer)
 [2019-12-07 12:15:07,155] TRACE :Psessionid:0x100006e15fb0000 type:createSession cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a (org.apache.zookeeper.server.PrepRequestProcessor)
 [2019-12-07 12:15:07,155] TRACE SessionTrackerImpl — Existing session 0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
 [2019-12-07 12:15:07,155] INFO Creating new log file: log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
 [2019-12-07 12:15:07,170] DEBUG Processing request:: sessionid:0x100006e15fb0000 type:createSession cxid:0x0 zxid:0x1 txntype:-10 reqpath:n/a (org.apache.zookeeper.server.FinalRequestProcessor)


> Data loss after upgrading 3.4.14 to 3.5.6 with snapshot.trust.empty=true
> ------------------------------------------------------------------------
>
>                 Key: ZOOKEEPER-3644
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3644
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.5.6
>            Reporter: Manikumar
>            Priority: Major
>
> We have tried to upgrade single node ZK server from 3.4.14 to 3.5.6.  There were no snapshot files, so as suggested in ZOOKEEPER-3056,
>  I have set snapshot.trust.empty to true. After server startup, when I tried to list the znodes, I found that znodes are missing.
> Steps to reproduce:
>  # Start a single node ZK 3.4.14 server and create few znodes
>  # Upgrade the server to 3.5.6 with  snapshot.trust.empty=true config
>  # try to list the znodes using zkShell
> Looking into the [source code|https://github.com/apache/zookeeper/blob/release-3.5.6/zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileTxnSnapLog.java#L224],
>  looks like we are not reading transaction log if there are no snapshot files and snapshot.trust.empty is set to true.
> ZK 3.5.6 logs:
> {quote}
>  [2019-12-07 12:13:35,007] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /var/lib/zookeeper/version-2 snapdir /var/lib/zookeeper/version-2
>   (org.apache.zookeeper.server.ZooKeeperServer)
>  [2019-12-07 12:13:35,012] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
>  [2019-12-07 12:13:35,014] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 12 worker threads, and 64 kB direct buffers. (org.apache
>  .zookeeper.server.NIOServerCnxnFactory)
>  [2019-12-07 12:13:35,017] INFO binding to port [0.0.0.0/0.0.0.0:2181|http://0.0.0.0/0.0.0.0:2181] (org.apache.zookeeper.server.NIOServerCnxnFactory)
>  [2019-12-07 12:13:35,027] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
>  [2019-12-07 12:13:35,029] DEBUG Created new input stream /var/lib/zookeeper/version-2/log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
>  [2019-12-07 12:13:35,031] DEBUG Created new input archive /var/lib/zookeeper/version-2/log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
>  [2019-12-07 12:13:35,035] DEBUG EOF exception java.io.EOFException: Failed to read /var/lib/zookeeper/version-2/log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
>  [2019-12-07 12:13:35,035] WARN No snapshot found, but there are log entries. This should only be allowed during upgrading. (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
>  [2019-12-07 12:13:35,035] INFO Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
>  [2019-12-07 12:13:35,036] INFO Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
>  [2019-12-07 12:13:35,050] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)
>  [2019-12-07 12:15:07,137] DEBUG Accepted socket connection from /[127.0.0.1:38888|http://127.0.0.1:38888/] (org.apache.zookeeper.server.NIOServerCnxnFactory)
>  [2019-12-07 12:15:07,144] DEBUG Session establishment request from client /[127.0.0.1:38888|http://127.0.0.1:38888/] client's lastZxid is 0x0 (org.apache.zookeeper.server.ZooKeeperServer)
>  [2019-12-07 12:15:07,145] DEBUG Adding session 0x100006e15fb0000 (org.apache.zookeeper.server.SessionTrackerImpl)
>  [2019-12-07 12:15:07,148] TRACE SessionTrackerImpl — Adding session 0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
>  [2019-12-07 12:15:07,149] DEBUG Client attempting to establish new session: session = 0x100006e15fb0000, zxid = 0x0, timeout = 30000, address = /[127.0.0.1:38888|http://127.0.0.1:38888/] (org.apache.zookeeper.server.ZooKeeperServer)
>  [2019-12-07 12:15:07,155] TRACE :Psessionid:0x100006e15fb0000 type:createSession cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a (org.apache.zookeeper.server.PrepRequestProcessor)
>  [2019-12-07 12:15:07,155] TRACE SessionTrackerImpl — Existing session 0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
>  [2019-12-07 12:15:07,155] INFO Creating new log file: log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
>  [2019-12-07 12:15:07,170] DEBUG Processing request:: sessionid:0x100006e15fb0000 type:createSession cxid:0x0 zxid:0x1 txntype:-10 reqpath:n/a (org.apache.zookeeper.server.FinalRequestProcessor)
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)