[jira] [Created] (ZOOKEEPER-3906) Data Inconsistency Between Zookeeper Leader and zookeeper Followers

["Gangadhar (Jira)" <ji...@apache.org>]

Gangadhar created ZOOKEEPER-3906:
------------------------------------

             Summary: Data Inconsistency Between Zookeeper Leader and zookeeper Followers
                 Key: ZOOKEEPER-3906
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3906
             Project: ZooKeeper
          Issue Type: Bug
          Components: quorum
    Affects Versions: 3.5.7
            Reporter: Gangadhar


Issue: Data Inconsistency Between Zookeeper Leader and zookeeper Followers. When we try to do the topic lookup for one of the topics I got broker not part of the cluster and verified below things as part of troubleshooting.

Steps followed as part of troubleshooting:

We have 5 zookeeper cluster.

*Step1:*  verified all zookeepers are following the leader or not?. As per below information its following all 4 zookeepers to zookeeper leader

zk_version      3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/11/2020 11:30 GMT
zk_avg_latency  0
zk_max_latency  823
zk_min_latency  0
zk_packets_received     30214264
zk_packets_sent 32424272
zk_num_alive_connections        7
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count  75190
zk_watch_count  21394
zk_ephemerals_count     793
zk_approximate_data_size        24706628
zk_open_file_descriptor_count   281
zk_max_file_descriptor_count    4096
zk_followers    4
zk_synced_followers     4
zk_pending_syncs        0
zk_last_proposal_size   166
zk_max_proposal_size    121947
zk_min_proposal_size    32


*Step 2:* Verified namespace bundle in all the zookeepers using the below command. We have received information from all zookeepers. except for Leader zookeeper.

./pulsar zookeeper-shell  get /namespace/$tenant/$Namespace/$Bubdle


*Step 3:* Try to delete the Namespace/$Bubdle to own the topic to another broker.

./pulsar zookeeper-shell  deleteall /namespace/$tenant/$Namespace/$Bubdle

*Error:* 
14:04:54.769 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
14:04:54.775 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
14:04:54.824 [main-SendThread(11.111.226.146:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 11-111-226-146.ebiz.verizon.com/11.111.226.146:2181. Will not attempt to authenticate using SASL (unknown error)
14:04:54.831 [main-SendThread(11.111.226.146:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /11.111.225.75:38804, server: 11-111-226-146.ebiz.verizon.com/11.111.20.146:2181
14:04:54.835 [main-SendThread(11.111.226.146:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server 11-111-226-146.ebiz.verizon.com/11.111.226.146:2181, sessionid = 0x500001bbbeb0651, negotiated timeout = 20000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
*Node does not exist: /namespace/$tenant/$Namespace/$Bubdle*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)