You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/08/03 16:23:13 UTC
[GitHub] [accumulo] dlmarion opened a new pull request, #2846: Added logging to try and find the cause of the intermittent ZK failures
dlmarion opened a new pull request, #2846:
URL: https://github.com/apache/accumulo/pull/2846
I want to push a commit into `main`, that can be reverted later, to try and get more information about the intermittent ZK failures in the ITs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on pull request #2846: Added logging to try and find the cause of the intermittent ZK failures
Posted by GitBox <gi...@apache.org>.
dlmarion commented on PR #2846:
URL: https://github.com/apache/accumulo/pull/2846#issuecomment-1205350459
I did look down that code path to see if anything was multi-threaded and not waiting for things to complete. I didn't see anything, and the existing code waits for the process to complete (the next line down from your reference). What you are saying is an option though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion merged pull request #2846: Added logging to try and find the cause of the intermittent ZK failures
Posted by GitBox <gi...@apache.org>.
dlmarion merged PR #2846:
URL: https://github.com/apache/accumulo/pull/2846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on pull request #2846: Added logging to try and find the cause of the intermittent ZK failures
Posted by GitBox <gi...@apache.org>.
milleruntime commented on PR #2846:
URL: https://github.com/apache/accumulo/pull/2846#issuecomment-1205159582
FYI only one IT failed from my full run on Github.
<pre>
[ERROR] Errors:
[ERROR] TabletIT>AccumuloClusterHarness.setupCluster:167 ยป IllegalState Unable to find instance id from zookeeper.
[INFO]
[ERROR] Tests run: 35, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[WARNING] Corrupted channel by directly writing to native stream in forked JVM 1. See FAQ web page and the dump file /home/runner/work/accumulo/accumulo/test/target/failsafe-reports/2022-08-03T19-40-17_134-jvmRun1.dumpstream
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on pull request #2846: Added logging to try and find the cause of the intermittent ZK failures
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on PR #2846:
URL: https://github.com/apache/accumulo/pull/2846#issuecomment-1206499320
The advantages of using a separate process for initialization are:
1. It reflects how users initialize, and exercises that process,
2. It avoids tainting any JVM-wide static state from the initialization process in the IT client code, and
3. It avoids treating the IT code as inherently part of the server side of things. Most ITs behave as Accumulo clients, not server-side utilities, and shouldn't be running server code in-process.
I don't see why running initialize in-process would help. It should suffice to wait for the initialize process to complete successfully, and then, if necessary, synchronize the IT with ZK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on pull request #2846: Added logging to try and find the cause of the intermittent ZK failures
Posted by GitBox <gi...@apache.org>.
milleruntime commented on PR #2846:
URL: https://github.com/apache/accumulo/pull/2846#issuecomment-1205337371
> might have to add a retry loop in the case where I'm trying to dump the ZK contents.
I am not sure why we are initializing as a separate process [here](https://github.com/apache/accumulo/blob/5d3f4e57e8c59488bddb5a310cfd4baaa4c5b619/minicluster/src/main/java/org/apache/accumulo/miniclusterImpl/MiniAccumuloClusterImpl.java#L582). The process could be returning a success code but Accumulo hasn't finished initializing. We might want to just call init directly and wait for Accumulo to finish initializing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on pull request #2846: Added logging to try and find the cause of the intermittent ZK failures
Posted by GitBox <gi...@apache.org>.
dlmarion commented on PR #2846:
URL: https://github.com/apache/accumulo/pull/2846#issuecomment-1205231477
This is interesting, might have to add a retry loop in the case where I'm trying to dump the ZK contents. MiniAccumuloClusterImpl started ZK, called Accumulo `Initialize`, but could not find the resulting instanceId. I'm going to make a couple of tweaks.
```
2022-08-03T19:49:19,303 [miniclusterImpl.MiniAccumuloClusterImpl] WARN : Starting ZooKeeper
2022-08-03T19:49:20,628 [miniclusterImpl.MiniAccumuloClusterImpl] WARN : Initializing ZooKeeper
2022-08-03T19:49:23,445 [miniclusterImpl.MiniAccumuloClusterImpl] INFO : Starting MAC against instance miniInstance and zookeeper(s) localhost:45451.
2022-08-03T19:49:29,347 [common.X509Util] INFO : Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2022-08-03T19:49:39,460 [miniclusterImpl.MiniAccumuloClusterImpl] WARN : ******* COULD NOT FIND INSTANCE ID - DUMPING ZK ************
2022-08-03T19:49:39,460 [miniclusterImpl.MiniAccumuloClusterImpl] WARN : Connected to ZooKeeper: localhost:45451
2022-08-03T19:49:39,460 [miniclusterImpl.MiniAccumuloClusterImpl] WARN : Looking for instanceId at /accumulo/instances/miniInstance
2022-08-03T19:49:39,523 [miniclusterImpl.MiniAccumuloClusterImpl] ERROR: Error dumping zk
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /accumulo
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[zookeeper-3.7.1.jar:3.7.1]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.7.1.jar:3.7.1]
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2028) ~[zookeeper-3.7.1.jar:3.7.1]
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2056) ~[zookeeper-3.7.1.jar:3.7.1]
at org.apache.zookeeper.ZKUtil.visitSubTreeDFS(ZKUtil.java:236) ~[zookeeper-3.7.1.jar:3.7.1]
at org.apache.accumulo.miniclusterImpl.MiniAccumuloClusterImpl.verifyUp(MiniAccumuloClusterImpl.java:681) ~[accumulo-minicluster-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.miniclusterImpl.MiniAccumuloClusterImpl.start(MiniAccumuloClusterImpl.java:620) ~[accumulo-minicluster-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.harness.AccumuloClusterHarness.setupCluster(AccumuloClusterHarness.java:167) ~[classes/:?]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on pull request #2846: Added logging to try and find the cause of the intermittent ZK failures
Posted by GitBox <gi...@apache.org>.
dlmarion commented on PR #2846:
URL: https://github.com/apache/accumulo/pull/2846#issuecomment-1204258965
Yeah, if you could do that, that would be great. I tried running it via GH Actions, but it didn't fail with the ZK error and I don't think I can get in to see the workspace.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on pull request #2846: Added logging to try and find the cause of the intermittent ZK failures
Posted by GitBox <gi...@apache.org>.
milleruntime commented on PR #2846:
URL: https://github.com/apache/accumulo/pull/2846#issuecomment-1205420215
> What you are saying is an option though.
I started tinkering in my workspace and it shouldn't be too bad of a change. The refactoring I did with the Initialize classes should help. I think it would help with the flakiness of our ITs if we don't spin off a separate process for initialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on pull request #2846: Added logging to try and find the cause of the intermittent ZK failures
Posted by GitBox <gi...@apache.org>.
milleruntime commented on PR #2846:
URL: https://github.com/apache/accumulo/pull/2846#issuecomment-1204241100
> I want to push a commit into `main`, that can be reverted later, to try and get more information about the intermittent ZK failures in the ITs.
Interesting... Do you want to see it run on Jenkins? I can run your branch on Github with the full set of ITs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org