You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2021/01/29 15:31:50 UTC
[GitHub] [accumulo] milleruntime opened a new pull request #1890: Check if ZK node exists before doing work
milleruntime opened a new pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890
* Check if the parent node exists before performing a sync and getting
the children in TransactionWatcher, preventing NoNodeExceptions.
Here is the exception that I have seen:
<pre>
2021-01-29T10:02:32,310 [tablet.BulkImportCacheCleaner] DEBUG: Error reading bulk import live transactions {}
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /accumulo/cc552965-f3c2-48b6-b200-9d1f2a7c48c5/bulkTx
at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) ~[zookeeper-3.6.2.jar:3.6.2]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.6.2.jar:3.6.2]
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2746) ~[zookeeper-3.6.2.jar:3.6.2]
at org.apache.accumulo.fate.zookeeper.ZooReader.lambda$getChildren$5(ZooReader.java:87) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.fate.zookeeper.ZooReader.retryLoopMutator(ZooReader.java:165) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.fate.zookeeper.ZooReader.retryLoop(ZooReader.java:144) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.fate.zookeeper.ZooReader.retryLoop(ZooReader.java:131) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.fate.zookeeper.ZooReader.getChildren(ZooReader.java:87) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.server.zookeeper.TransactionWatcher$ZooArbitrator.allTransactionsAlive(TransactionWatcher.java:108) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.BulkImportCacheCleaner.run(BulkImportCacheCleaner.java:50) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
</pre>
I don't understand what the `reader.sync()` method call is doing, so I am not sure if that should be called before `exists()` or not. The `reader.sync()` call is not throwing the exception so it may be OK to call that before checking.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#discussion_r566921562
##########
File path: server/base/src/main/java/org/apache/accumulo/server/zookeeper/TransactionWatcher.java
##########
@@ -104,13 +104,15 @@ public static void cleanup(ServerContext context, String type, long tid)
final ZooReader reader = context.getZooReaderWriter();
final Set<Long> result = new HashSet<>();
final String parent = context.getZooKeeperRoot() + "/" + type;
- reader.sync(parent);
- List<String> children = reader.getChildren(parent);
- for (String child : children) {
- if (child.endsWith("-running")) {
- continue;
+ if (reader.exists(parent)) {
Review comment:
This is the only spot the method is called:
https://github.com/apache/accumulo/blob/3fd5cad92f9b63ac19e4466f3f2d5237b905262c/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/BulkImportCacheCleaner.java#L50
Looking at that method, I think it is OK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#discussion_r566974481
##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/BulkImportCacheCleaner.java
##########
@@ -54,9 +54,13 @@ public void run() {
for (Tablet tablet : server.getOnlineTablets().values()) {
tablet.cleanupBulkLoadedFiles(tids);
}
- } catch (KeeperException | InterruptedException e) {
+ } catch (KeeperException e) {
// we'll just clean it up again later
- log.debug("Error reading bulk import live transactions {}", e);
+ log.debug("Error reading bulk import live transactions {}", tids, e);
+ } catch (InterruptedException e) {
+ // propagate the interrupt status.
+ Thread.currentThread().interrupt();
+ log.debug("Interrupted while reading bulk import live transactions {}", tids, e);
Review comment:
Sorry should have considered this - but might want to consider bumping up the log level - we really shouldn't be seeing interrupts and if the keep reoccurring and are not being handled, then we should dig deeper.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#discussion_r566985271
##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/BulkImportCacheCleaner.java
##########
@@ -54,9 +54,13 @@ public void run() {
for (Tablet tablet : server.getOnlineTablets().values()) {
tablet.cleanupBulkLoadedFiles(tids);
}
- } catch (KeeperException | InterruptedException e) {
+ } catch (KeeperException e) {
// we'll just clean it up again later
- log.debug("Error reading bulk import live transactions {}", e);
+ log.debug("Error reading bulk import live transactions {}", tids, e);
+ } catch (InterruptedException e) {
+ // propagate the interrupt status.
+ Thread.currentThread().interrupt();
+ log.debug("Interrupted while reading bulk import live transactions {}", tids, e);
Review comment:
Bumped to warn in 307bef2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#discussion_r566930844
##########
File path: server/base/src/main/java/org/apache/accumulo/server/zookeeper/TransactionWatcher.java
##########
@@ -104,13 +104,15 @@ public static void cleanup(ServerContext context, String type, long tid)
final ZooReader reader = context.getZooReaderWriter();
final Set<Long> result = new HashSet<>();
final String parent = context.getZooKeeperRoot() + "/" + type;
- reader.sync(parent);
- List<String> children = reader.getChildren(parent);
- for (String child : children) {
- if (child.endsWith("-running")) {
- continue;
+ if (reader.exists(parent)) {
Review comment:
I see that too... already done. I will push up changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#discussion_r566945281
##########
File path: server/base/src/main/java/org/apache/accumulo/server/zookeeper/TransactionWatcher.java
##########
@@ -104,13 +104,15 @@ public static void cleanup(ServerContext context, String type, long tid)
final ZooReader reader = context.getZooReaderWriter();
final Set<Long> result = new HashSet<>();
final String parent = context.getZooKeeperRoot() + "/" + type;
- reader.sync(parent);
- List<String> children = reader.getChildren(parent);
- for (String child : children) {
- if (child.endsWith("-running")) {
- continue;
+ if (reader.exists(parent)) {
Review comment:
Because this is a runnable - we may not want (or be able to throw an exception) - so at a minimum we should be able to reset the interrupt status with something like:
Also, not sure what info is valuable in the debug statement, listing the tids might make sense - unless its a big list... Without understanding the context I cannot say at this point....
```
} catch (KeeperException e) {
// we'll just clean it up again later
log.debug("Error reading bulk import live transactions {}", tids, e);
} catch(InterruptedException e) {
// propagate the interrupt status.
Thread.currentThread().interrupt();
log.debug("Interrupted while reading bulk import live transactions {}", tids, e);
}
```
This comes from [Dealing with InterruptedException](https://www.ibm.com/developerworks/library/j-jtp05236/index.html)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#discussion_r566933693
##########
File path: server/base/src/main/java/org/apache/accumulo/server/zookeeper/TransactionWatcher.java
##########
@@ -104,13 +104,15 @@ public static void cleanup(ServerContext context, String type, long tid)
final ZooReader reader = context.getZooReaderWriter();
final Set<Long> result = new HashSet<>();
final String parent = context.getZooKeeperRoot() + "/" + type;
- reader.sync(parent);
- List<String> children = reader.getChildren(parent);
- for (String child : children) {
- if (child.endsWith("-running")) {
- continue;
+ if (reader.exists(parent)) {
Review comment:
> Another thing is that exception handling is swallowing the interrupt exception.
What are you thinking? Separate catch for the `InterruptedException`, wrap with `RuntimeException` and rethrow?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on a change in pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#discussion_r566959054
##########
File path: server/base/src/main/java/org/apache/accumulo/server/zookeeper/TransactionWatcher.java
##########
@@ -104,13 +104,15 @@ public static void cleanup(ServerContext context, String type, long tid)
final ZooReader reader = context.getZooReaderWriter();
final Set<Long> result = new HashSet<>();
final String parent = context.getZooKeeperRoot() + "/" + type;
- reader.sync(parent);
- List<String> children = reader.getChildren(parent);
- for (String child : children) {
- if (child.endsWith("-running")) {
- continue;
+ if (reader.exists(parent)) {
Review comment:
Updated in 6511c46
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime merged pull request #1890: Improvements to BulkImportCacheCleaner
Posted by GitBox <gi...@apache.org>.
milleruntime merged pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on a change in pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#discussion_r566920399
##########
File path: server/base/src/main/java/org/apache/accumulo/server/zookeeper/TransactionWatcher.java
##########
@@ -104,13 +104,15 @@ public static void cleanup(ServerContext context, String type, long tid)
final ZooReader reader = context.getZooReaderWriter();
final Set<Long> result = new HashSet<>();
final String parent = context.getZooKeeperRoot() + "/" + type;
- reader.sync(parent);
- List<String> children = reader.getChildren(parent);
- for (String child : children) {
- if (child.endsWith("-running")) {
- continue;
+ if (reader.exists(parent)) {
Review comment:
Is it okay to return an empty set if the parent doesn't exist? I'm not sure what the undocumented return value of this method is supposed to represent, so it's not easy to tell if returning an empty set is okay in this case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#discussion_r566926865
##########
File path: server/base/src/main/java/org/apache/accumulo/server/zookeeper/TransactionWatcher.java
##########
@@ -104,13 +104,15 @@ public static void cleanup(ServerContext context, String type, long tid)
final ZooReader reader = context.getZooReaderWriter();
final Set<Long> result = new HashSet<>();
final String parent = context.getZooKeeperRoot() + "/" + type;
- reader.sync(parent);
- List<String> children = reader.getChildren(parent);
- for (String child : children) {
- if (child.endsWith("-running")) {
- continue;
+ if (reader.exists(parent)) {
Review comment:
I was looking at that too - I was thinking it might be okay - the comment in the exception handler says that it will just retry, so it could be the same thing as processing nothing with an empty list. There does not seem to be any difference - it looks like it builds a new list each time run is called and nothing is signalling completion.
Also in that exception handling I noticed (well, the ide did) in BulkImportCacheCleaner:[59]
`log.debug("Error reading bulk import live transactions {}", e);` has mismatched arguments - either {} should be removed or something added.
Another thing is that exception handling is swallowing the interrupt exception.
If you don't want to handle those as part of this PR i'll create an issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
milleruntime commented on pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#issuecomment-769891862
> `sync()` ensures the server's view of ZooKeeper state is immediately up-to-date with the consensus of the ZooKeeper cluster. It normally does this asynchronously, with a callback, but our wrapper method in ZooReader will wait until it is sync'd. It should probably be called before the existence check.
OK I had a hunch that is what it is doing but wasn't positive. I will move the sync before the existence check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#issuecomment-769886922
`sync()` ensures the server's view of ZooKeeper state is immediately up-to-date with the consensus of the ZooKeeper cluster. It normally does this asynchronously, with a callback, but our wrapper method in ZooReader will wait until it is sync'd. It should probably be called before the existence check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on a change in pull request #1890: Check if ZK node exists before doing work
Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #1890:
URL: https://github.com/apache/accumulo/pull/1890#discussion_r566945281
##########
File path: server/base/src/main/java/org/apache/accumulo/server/zookeeper/TransactionWatcher.java
##########
@@ -104,13 +104,15 @@ public static void cleanup(ServerContext context, String type, long tid)
final ZooReader reader = context.getZooReaderWriter();
final Set<Long> result = new HashSet<>();
final String parent = context.getZooKeeperRoot() + "/" + type;
- reader.sync(parent);
- List<String> children = reader.getChildren(parent);
- for (String child : children) {
- if (child.endsWith("-running")) {
- continue;
+ if (reader.exists(parent)) {
Review comment:
Because this is a runnable - we may not want (or be able to throw an exception) - so at a minimum we should be able to reset the interrupt status with something like:
Also, not sure what info is valuable in the debug statement, listing the tids might make sense - unless its a big list... Without understanding the context I cannot say at this point....
` } catch (KeeperException e) {
// we'll just clean it up again later
log.debug("Error reading bulk import live transactions {}", tids, e);
} catch(InterruptedException e) {
// propagate the interrupt status.
Thread.currentThread().interrupt();
log.debug("Interrupted while reading bulk import live transactions {}", tids, e);
}
`
This comes from [Dealing with InterruptedException](https://www.ibm.com/developerworks/library/j-jtp05236/index.html)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org