[jira] Commented: (ZOOKEEPER-582) ZooKeeper can revert to old data when a snapshot is created outside of normal processing

["Patrick Hunt (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779149#action_12779149 ] 

Patrick Hunt commented on ZOOKEEPER-582:
----------------------------------------

As Ben mentioned we will never see this situation during normal operation of ZK.

The case where we did see this was a result of a user running the migration tool that we provide to upgrade from version 2 to version 3 of ZooKeeper. The tool migrates the data by writing a single snapshot file where the zxid is maintained (it does not write a log file). As a result of the scenario Ben mentioned (snap with no associated log file) this could cause this bug to occur. If you have run the migration tool, documented here:
http://hadoop.apache.org/zookeeper/docs/r3.0.0/releasenotes.html#migration_data
you can verify whether or not you have this situation by looking at your ZooKeeper datadirectory

Here's an example

-rw-r--r--  1 root search 67108880 Nov 17 19:31 log.300022b61
-rw-r--r--  1 root search 67108880 Nov 17 19:38 log.3000292d0
-rw-r--r--  1 root search  3646608 Nov  5 12:13 snapshot.1db5df6e2d6
-rw-r--r--  1 root search  3616579 Nov 17 19:31 snapshot.3000292c9
-rw-r--r--  1 root search  3616708 Nov 17 19:38 snapshot.300038d32

where the files are of the form <file>.<epoch><xid> 
epoch and xid both being 4 byte values represented as hex

Notice that the snapshot.1db5df6e2d6 has epoch of 0x1db, while the other
files have epoch of 0x3, this is the scenario described in the description of this
JIRA. (there is no log file associated with epoch 0x1db)

If you see this in your datadir - a snapshot with an epoch where there are no log files with
this same epoch, then this bug pertains.  If you see snapshots of a particular epoch
and log files with the same epoch then this bug does NOT pertain.


> ZooKeeper can revert to old data when a snapshot is created outside of normal processing
> ----------------------------------------------------------------------------------------
>
>                 Key: ZOOKEEPER-582
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-582
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.1.1, 3.2.1
>            Reporter: Benjamin Reed
>            Priority: Blocker
>             Fix For: 3.2.2, 3.1.2
>
>
> when zookeeper starts up it will restore the most recent state (latest zxid) it finds in the data directory. unfortunately, in the quorum version of zookeeper updates are logged using an epoch based on the latest log file in a directory. if there is a snapshot with a higher epoch than the log files, the zookeeper server will start logging using an epoch one higher than the highest log file.
> so if a data directory has a snapshot with an epoch of 27 and there are no log files, zookeeper will start logging changes using epoch 1. if the cluster restarts the state will be restored from the snapshot with the epoch of 27, which in effect, restores old data.
> normal operation of zookeeper will never result in this situation.
> this does not effect standalone zookeeper.
> a fix should make sure to use an epoch one higher than the current state, whether it comes from the snapshot or log, and should include a sanity check to make sure that a follower never connects to a leader that has a lower epoch than its own.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.