You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "Bozhen Liu (Jira)" <ji...@apache.org> on 2020/05/06 15:28:00 UTC
[jira] [Created] (ZOOKEEPER-3819) Potential Races on DataTree
Bozhen Liu created ZOOKEEPER-3819:
-------------------------------------
Summary: Potential Races on DataTree
Key: ZOOKEEPER-3819
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3819
Project: ZooKeeper
Issue Type: Bug
Components: server
Affects Versions: 3.6.1, 3.5.4
Reporter: Bozhen Liu
We run our static race detector on ZooKeeper and discovered three methods in class org/apache/zookeeper/server/DataTree may have concurrent accesses on ephemerals without proper lock protection. The three methods are createNode(), killSession() and deserialize(). They all read and write the ephemerals in the similar way without common lock from get() to put():
{code:java}
HashSet<String> list = ephemerals.get(eowner);
if (list == null){
list = new HashSet<String>();
ephemerals.put(eowner, list);
}
list.add(path);
{code}
This is similar to ZOOKEEPER-3102.
The possible traces that lead to the race are (v3.5.4, probably the same in v3.6.1):
=> Race: org/apache/zookeeper/server/DataTree.java/util/Map (org/apache/zookeeper/server/DataTree:478 (TID: 2190) , org/apache/zookeeper/server/DataTree:1215 (TID: 9592) )
Trace 1st node: (TID: 2190)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig (line 200)
-> Call java/lang/Thread.start from org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from java/lang/Thread.start (line -1)
*> Thread (2190) created by Application,Lorg/apache/zookeeper/server/quorum/QuorumPeer>.run (line 1167)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.run from java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.pRequest from org/apache/zookeeper/server/PrepRequestProcessor.run (line 145)
-> Call org/apache/zookeeper/server/FinalRequestProcessor.processRequest from org/apache/zookeeper/server/PrepRequestProcessor.pRequest (line 906)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from org/apache/zookeeper/server/FinalRequestProcessor.processRequest (line 104)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1190)
-> Call org/apache/zookeeper/server/ZKDatabase.processTxn from org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1199)
-> Call org/apache/zookeeper/server/DataTree.processTxn from org/apache/zookeeper/server/ZKDatabase.processTxn (line 434)
-> Call org/apache/zookeeper/server/DataTree.createNode from org/apache/zookeeper/server/DataTree.processTxn (line 786)
=> Read on org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in org/apache/zookeeper/server/DataTree.createNode (line 478)
Trace 2st node: (TID: 9592)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig (line 200)
-> Call java/lang/Thread.start from org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/quorum/Leader.lead from org/apache/zookeeper/server/quorum/QuorumPeer.run (line 1227)
*> Thread (6402) created by Application,Lorg/apache/zookeeper/server/quorum/Leader>.lead (line 448)
-> Call org/apache/zookeeper/server/quorum/Leader$LearnerCnxAcceptor.run from java/lang/Thread.start (line -1)
*> Thread (7676) created by Application,Lorg/apache/zookeeper/server/quorum/Leader$LearnerCnxAcceptor>.run (line 386)
-> Call org/apache/zookeeper/server/quorum/LearnerHandler.run from java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/quorum/Leader.processAck from org/apache/zookeeper/server/quorum/LearnerHandler.run (line 559)
-> Call org/apache/zookeeper/server/quorum/Leader.tryToCommit from org/apache/zookeeper/server/quorum/Leader.processAck (line 863)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.processReconfig from org/apache/zookeeper/server/quorum/Leader.tryToCommit (line 779)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.restartLeaderElection from org/apache/zookeeper/server/quorum/QuorumPeer.processReconfig (line 1871)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.startLeaderElection from org/apache/zookeeper/server/quorum/QuorumPeer.restartLeaderElection (line 1530)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.getLastLoggedZxid from org/apache/zookeeper/server/quorum/QuorumPeer.startLeaderElection (line 917)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.loadDataBase from org/apache/zookeeper/server/quorum/QuorumPeer.getLastLoggedZxid (line 1009)
-> Call org/apache/zookeeper/server/ZKDatabase.loadDataBase from org/apache/zookeeper/server/quorum/QuorumPeer.loadDataBase (line 867)
-> Call org/apache/zookeeper/server/persistence/FileTxnSnapLog.restore from org/apache/zookeeper/server/ZKDatabase.loadDataBase (line 240)
-> Call org/apache/zookeeper/server/persistence/FileSnap.deserialize from org/apache/zookeeper/server/persistence/FileTxnSnapLog.restore (line 200)
-> Call org/apache/zookeeper/server/persistence/FileSnap.deserialize from org/apache/zookeeper/server/persistence/FileSnap.deserialize (line 87)
-> Call org/apache/zookeeper/server/util/SerializeUtils.deserializeSnapshot from org/apache/zookeeper/server/persistence/FileSnap.deserialize (line 122)
-> Call org/apache/zookeeper/server/DataTree.deserialize from org/apache/zookeeper/server/util/SerializeUtils.deserializeSnapshot (line 141)
=> Write to org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in org/apache/zookeeper/server/DataTree.deserialize (line 1215)
--------------------------------------------------------------------------------------------------------------------------------
=> Race: org/apache/zookeeper/server/DataTree.java/util/Map (org/apache/zookeeper/server/DataTree:478 (TID: 2190) , org/apache/zookeeper/server/DataTree:1005 (TID: 2835) )
Trace 1st node: (TID: 2190)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig (line 200)
-> Call java/lang/Thread.start from org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from java/lang/Thread.start (line -1)
*> Thread (2190) created by Application,Lorg/apache/zookeeper/server/quorum/QuorumPeer>.run (line 1167)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.run from java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.pRequest from org/apache/zookeeper/server/PrepRequestProcessor.run (line 145)
-> Call org/apache/zookeeper/server/FinalRequestProcessor.processRequest from org/apache/zookeeper/server/PrepRequestProcessor.pRequest (line 906)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from org/apache/zookeeper/server/FinalRequestProcessor.processRequest (line 104)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1190)
-> Call org/apache/zookeeper/server/ZKDatabase.processTxn from org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1199)
-> Call org/apache/zookeeper/server/DataTree.processTxn from org/apache/zookeeper/server/ZKDatabase.processTxn (line 434)
-> Call org/apache/zookeeper/server/DataTree.createNode from org/apache/zookeeper/server/DataTree.processTxn (line 786)
=> Read on org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in org/apache/zookeeper/server/DataTree.createNode (line 478)
Trace 2st node: (TID: 2835)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig (line 200)
-> Call java/lang/Thread.start from org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from java/lang/Thread.start (line -1)
*> Thread (2190) created by Application,Lorg/apache/zookeeper/server/quorum/QuorumPeer>.run (line 1167)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer$1.run from java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/quorum/ReadOnlyZooKeeperServer.startup from org/apache/zookeeper/server/quorum/QuorumPeer$1.run (line 1157)
-> Call org/apache/zookeeper/server/ZooKeeperServer.startup from org/apache/zookeeper/server/quorum/ReadOnlyZooKeeperServer.startup (line 70)
-> Call org/apache/zookeeper/server/quorum/ReadOnlyZooKeeperServer.setupRequestProcessors from org/apache/zookeeper/server/ZooKeeperServer.startup (line 453)
*> Thread (2835) created by Application,Lorg/apache/zookeeper/server/quorum/ReadOnlyZooKeeperServer>.setupRequestProcessors (line 59)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.run from java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.pRequest from org/apache/zookeeper/server/PrepRequestProcessor.run (line 145)
-> Call org/apache/zookeeper/server/FinalRequestProcessor.processRequest from org/apache/zookeeper/server/PrepRequestProcessor.pRequest (line 906)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from org/apache/zookeeper/server/FinalRequestProcessor.processRequest (line 104)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1190)
-> Call org/apache/zookeeper/server/ZKDatabase.processTxn from org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1199)
-> Call org/apache/zookeeper/server/DataTree.processTxn from org/apache/zookeeper/server/ZKDatabase.processTxn (line 434)
-> Call org/apache/zookeeper/server/DataTree.killSession from org/apache/zookeeper/server/DataTree.processTxn (line 854)
=> Write to org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in org/apache/zookeeper/server/DataTree.killSession (line 1005)
--------------------------------------------------------------------------------------------------------------------------------
=> Race: org/apache/zookeeper/server/DataTree.java/util/Map (org/apache/zookeeper/server/DataTree:478 (TID: 2190) , org/apache/zookeeper/server/DataTree:481 (TID: 13154) )
Trace 1st node: (TID: 2190)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig (line 200)
-> Call java/lang/Thread.start from org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from java/lang/Thread.start (line -1)
*> Thread (2190) created by Application,Lorg/apache/zookeeper/server/quorum/QuorumPeer>.run (line 1167)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.run from java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/PrepRequestProcessor.pRequest from org/apache/zookeeper/server/PrepRequestProcessor.run (line 145)
-> Call org/apache/zookeeper/server/FinalRequestProcessor.processRequest from org/apache/zookeeper/server/PrepRequestProcessor.pRequest (line 906)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from org/apache/zookeeper/server/FinalRequestProcessor.processRequest (line 104)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1190)
-> Call org/apache/zookeeper/server/ZKDatabase.processTxn from org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1199)
-> Call org/apache/zookeeper/server/DataTree.processTxn from org/apache/zookeeper/server/ZKDatabase.processTxn (line 434)
-> Call org/apache/zookeeper/server/DataTree.createNode from org/apache/zookeeper/server/DataTree.processTxn (line 786)
=> Read on org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in org/apache/zookeeper/server/DataTree.createNode (line 478)
Trace 2st node: (TID: 13154)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun from org/apache/zookeeper/server/quorum/QuorumPeerMain.main (line 82)
-> Call org/apache/zookeeper/server/quorum/QuorumPeerMain.runFromConfig from org/apache/zookeeper/server/quorum/QuorumPeerMain.initializeAndRun (line 123)
*> Thread (153) created by Application,Lorg/apache/zookeeper/server/quorum/QuorumPeerMain>.runFromConfig (line 200)
-> Call java/lang/Thread.start from org/apache/zookeeper/server/quorum/QuorumPeer.start (line 862)
-> Call org/apache/zookeeper/server/quorum/QuorumPeer.run from java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/quorum/Observer.observeLeader from org/apache/zookeeper/server/quorum/QuorumPeer.run (line 1201)
-> Call org/apache/zookeeper/server/quorum/Learner.syncWithLeader from org/apache/zookeeper/server/quorum/Observer.observeLeader (line 74)
-> Call org/apache/zookeeper/server/ZooKeeperServer.startup from org/apache/zookeeper/server/quorum/Learner.syncWithLeader (line 559)
-> Call org/apache/zookeeper/server/quorum/ObserverZooKeeperServer.setupRequestProcessors from org/apache/zookeeper/server/ZooKeeperServer.startup (line 453)
*> Thread (8653) created by Application,Lorg/apache/zookeeper/server/quorum/ObserverZooKeeperServer>.setupRequestProcessors (line 97)
-> Call java/lang/Thread.start from org/apache/zookeeper/server/quorum/CommitProcessor.start (line 263)
-> Call org/apache/zookeeper/server/quorum/CommitProcessor.run from java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/quorum/CommitProcessor.sendToNextProcessor from org/apache/zookeeper/server/quorum/CommitProcessor.run (line 180)
-> Call org/apache/zookeeper/server/WorkerService.schedule from org/apache/zookeeper/server/quorum/CommitProcessor.sendToNextProcessor (line 272)
*> Thread (13154) created by Application,Lorg/apache/zookeeper/server/WorkerService>.schedule (line 128)
-> Call org/apache/zookeeper/server/WorkerService$ScheduledWorkRequest.run from java/lang/Thread.start (line -1)
-> Call org/apache/zookeeper/server/quorum/CommitProcessor$CommitWorkRequest.doWork from org/apache/zookeeper/server/WorkerService$ScheduledWorkRequest.run (line 162)
-> Call org/apache/zookeeper/server/FinalRequestProcessor.processRequest from org/apache/zookeeper/server/quorum/CommitProcessor$CommitWorkRequest.doWork (line 297)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from org/apache/zookeeper/server/FinalRequestProcessor.processRequest (line 104)
-> Call org/apache/zookeeper/server/ZooKeeperServer.processTxn from org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1190)
-> Call org/apache/zookeeper/server/ZKDatabase.processTxn from org/apache/zookeeper/server/ZooKeeperServer.processTxn (line 1199)
-> Call org/apache/zookeeper/server/DataTree.processTxn from org/apache/zookeeper/server/ZKDatabase.processTxn (line 434)
-> Call org/apache/zookeeper/server/DataTree.createNode from org/apache/zookeeper/server/DataTree.processTxn (line 786)
=> Write to org/apache/zookeeper/server/DataTree.java/util/Map(ephemerals) in org/apache/zookeeper/server/DataTree.createNode (line 481)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)