[jira] [Created] (ZOOKEEPER-3490) Zookeeper follower's not reflecting writes (after months)

["Matthew Hertz (JIRA)" <ji...@apache.org>]

Matthew Hertz created ZOOKEEPER-3490:
----------------------------------------

             Summary: Zookeeper follower's not reflecting writes (after months)
                 Key: ZOOKEEPER-3490
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3490
             Project: ZooKeeper
          Issue Type: Bug
    Affects Versions: 3.4.13
            Reporter: Matthew Hertz


Hi,

We have a 3 node Zookeeper cluster. There are a number of znode's on the leader that are not visible on the followers.
{code:java}
$ zkCli -server <server 1> (follower) 
[zk: <server 1>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
abcd.company.com:<built-in function getpid>
cZxid = 0xf00061d68
ctime = Thu Nov 01 12:40:33 GMT 2018
mZxid = 0xf00061d68
mtime = Thu Nov 01 12:40:33 GMT 2018
pZxid = 0xf00061d68
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x500be5318d60407
dataLength = 58
numChildren = 0
{code}
{code:java}
$ zkCli -server <server 2> (leader)
[zk: <server2>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040 Node does not exist: /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
{code}
{code:java}
$ zkCli -server <server 3> (follower)
[zk: <server3>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
abcd.company.com:<built-in function getpid>
cZxid = 0xf00061d68
ctime = Thu Nov 01 12:40:33 GMT 2018
mZxid = 0xf00061d68
mtime = Thu Nov 01 12:40:33 GMT 2018
pZxid = 0xf00061d68
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x500be5318d60407
dataLength = 58
numChildren = 0
{code}
These nodes are ephemeral nodes. The sessions no longer exist. There are 6 znodes in this 'inconsistent' state. The cluster is currently connected - there are no networking partitions currently.

We're at a loss for how to both debug and fix this. Restarting the Zookeeper followers presumably will not help? Are all nodes ever force-synced from the leader?

Help would be appreciated.

Thanks

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)