You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2021/01/25 23:55:17 UTC
[GitHub] [accumulo] keith-turner edited a comment on pull request #1866: Add a UUID to the server's ZooLock
keith-turner edited a comment on pull request #1866:
URL: https://github.com/apache/accumulo/pull/1866#issuecomment-767188215
@dlmarion I ran an experiment w/ this code locally. First I made the following change on my local copy, to always force the creation of multiple ephemeral nodes.
```diff
diff --git a/core/src/main/java/org/apache/accumulo/fate/zookeeper/ZooLock.java b/core/src/main/java/org/apache/accumulo/fate/zookeeper/ZooLock.java
index f79ca72c50..1f87ee3c3a 100644
--- a/core/src/main/java/org/apache/accumulo/fate/zookeeper/ZooLock.java
+++ b/core/src/main/java/org/apache/accumulo/fate/zookeeper/ZooLock.java
@@ -370,6 +370,11 @@ public class ZooLock implements Watcher {
// except that instead of the ephemeral lock node being of the form guid-lock- use lock-guid-.
// Another deviation from the recipe is that we cleanup any extraneous ephermal nodes that
// were created.
+
+ for(int i = 0; i < 3; i++) {
+ zooKeeper.putEphemeralSequential(lockPathPrefix, data);
+ }
+
final String createPath = zooKeeper.putEphemeralSequential(lockPathPrefix, data);
LOG.trace("[{}] Ephemeral node {} created", this.getZLockPrefix(), createPath);
```
Then I started up Accumulo using these changes, enabled trace logging for ZooLock, and restarted the tserver. After that I saw the following in the tserver logs.
```
2021-01-25T18:32:35,744 [zookeeper.ZooLock] TRACE: [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] Ephemeral node /accumulo/3cad24a9-53a9-4a6a-a846-ba24c5d747b9/tservers/accumulo:9997/zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000007 created
2021-01-25T18:32:35,754 [zookeeper.ZooLock] TRACE: Children nodes
2021-01-25T18:32:35,754 [zookeeper.ZooLock] TRACE: - zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000004
2021-01-25T18:32:35,754 [zookeeper.ZooLock] TRACE: - zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000005
2021-01-25T18:32:35,754 [zookeeper.ZooLock] TRACE: - zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000006
2021-01-25T18:32:35,754 [zookeeper.ZooLock] TRACE: - zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000007
2021-01-25T18:32:35,755 [zookeeper.ZooLock] TRACE: [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] lowest sequential node found: /accumulo/3cad24a9-53a9-4a6a-a846-ba24c5d747b9/tservers/accumulo:9997/zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000004
2021-01-25T18:32:35,755 [zookeeper.ZooLock] INFO : [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] Zookeeper client missed server response, multiple ephemeral child nodes created at /accumulo/3cad24a9-53a9-4a6a-a846-ba24c5d747b9/tservers/accumulo:9997/zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#
2021-01-25T18:32:35,755 [zookeeper.ZooLock] TRACE: [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] higher sequential node found: zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000005, deleting it
2021-01-25T18:32:35,758 [zookeeper.ZooLock] INFO : [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] Zookeeper client missed server response, multiple ephemeral child nodes created at /accumulo/3cad24a9-53a9-4a6a-a846-ba24c5d747b9/tservers/accumulo:9997/zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#
2021-01-25T18:32:35,758 [zookeeper.ZooLock] TRACE: [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] higher sequential node found: zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000006, deleting it
2021-01-25T18:32:35,760 [zookeeper.ZooLock] INFO : [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] Zookeeper client missed server response, multiple ephemeral child nodes created at /accumulo/3cad24a9-53a9-4a6a-a846-ba24c5d747b9/tservers/accumulo:9997/zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#
2021-01-25T18:32:35,760 [zookeeper.ZooLock] TRACE: [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] higher sequential node found: zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000007, deleting it
2021-01-25T18:32:35,761 [zookeeper.ZooLock] TRACE: [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] Setting watcher on /accumulo/3cad24a9-53a9-4a6a-a846-ba24c5d747b9/tservers/accumulo:9997/zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000004
2021-01-25T18:32:35,764 [zookeeper.ZooLock] TRACE: Children nodes
2021-01-25T18:32:35,764 [zookeeper.ZooLock] TRACE: - zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000004
2021-01-25T18:32:35,764 [zookeeper.ZooLock] TRACE: [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] First candidate is my lock, acquiring
2021-01-25T18:32:35,764 [tserver.TabletServer] DEBUG: Obtained tablet server lock /accumulo/3cad24a9-53a9-4a6a-a846-ba24c5d747b9/tservers/accumulo:9997/zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000004
2021-01-25T18:32:35,768 [zookeeper.ZooLock] TRACE: Children nodes
2021-01-25T18:32:35,768 [zookeeper.ZooLock] TRACE: - zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000004
```
Then I did the following in zkCli.
```
[zk: localhost:2181(CONNECTED) 0] addauth digest accumulo:uno
[zk: localhost:2181(CONNECTED) 1] ls /accumulo/
3cad24a9-53a9-4a6a-a846-ba24c5d747b9 instances
[zk: localhost:2181(CONNECTED) 1] ls /accumulo/3cad24a9-53a9-4a6a-a846-ba24c5d747b9/tservers/accumulo:9997
[zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000004]
[zk: localhost:2181(CONNECTED) 2] delete /accumulo/3cad24a9-53a9-4a6a-a846-ba24c5d747b9/tservers/accumulo:9997/zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000004
```
After that I saw the following in the tserver logs and confirmed the tserver was not running.
```
2021-01-25T18:37:43,487 [zookeeper.ZooLock] TRACE: [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#0000000004 was deleted
2021-01-25T18:37:43,489 [tserver.TabletServer] ERROR: Lost tablet server lock (reason = LOCK_DELETED), exiting.
```
Everything worked great. After running this experiment I am thinking the log messages related to creating the lock that are trace could be debug. Locks are not created often, locks are really important, and the information could be useful for debugging. Also I noticed the following log message was emitted multiple times.
```
INFO : [zlock#93852053-1721-4f2a-a8fe-ee4b9398389e#] Zookeeper client missed server response, multiple ephemeral child nodes created at /accumulo/3cad24a9-53a9-4a6a-a846-
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org