You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2020/02/27 13:45:27 UTC
[GitHub] [accumulo] milleruntime opened a new issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
milleruntime opened a new issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534
The GarbageCollectorIT is currently failing, specifically gcLotsOfCandidatesIT is timing out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] jzgithub1 commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
jzgithub1 commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594101065
Let me look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] jzgithub1 edited a comment on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
jzgithub1 edited a comment on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-593978837
@ctubbsii between 10-20% of the runs I do fail here:
`org.junit.runners.model.TestTimedOutException: test timed out after 300 seconds
at java.base@11.0.6/sun.nio.cs.UTF_8$Decoder.decodeArrayLoop(UTF_8.java:236)
at java.base@11.0.6/sun.nio.cs.UTF_8$Decoder.decodeLoop(UTF_8.java:414)
at java.base@11.0.6/java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:576)
at java.base@11.0.6/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:318)
at java.base@11.0.6/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base@11.0.6/java.io.InputStreamReader.read(InputStreamReader.java:185)
at java.base@11.0.6/java.io.Reader.read(Reader.java:229)
at app//org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2537)
at app//org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2516)
at app//org.apache.commons.io.IOUtils.copy(IOUtils.java:2493)
at app//org.apache.commons.io.IOUtils.copy(IOUtils.java:2441)
at app//org.apache.commons.io.IOUtils.toString(IOUtils.java:1084)
at app//org.apache.accumulo.miniclusterImpl.MiniAccumuloClusterImpl$ProcessInfo.readStdOut(MiniAccumuloClusterImpl.java:195)
at app//org.apache.accumulo.test.functional.GarbageCollectorIT.gcLotsOfCandidatesIT(GarbageCollectorIT.java:156)
at java.base@11.0.6/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base@11.0.6/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base@11.0.6/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base@11.0.6/java.lang.reflect.Method.invoke(Method.java:566)
at app//org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at app//org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at app//org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at app//org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at app//org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at app//org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at app//org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at app//org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.base@11.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base@11.0.6/java.lang.Thread.run(Thread.java:834)
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] ctubbsii commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594203437
The G1GC (the default GC in newer JVMs) has more overhead, and can very quickly run out of memory on a JVM configured with too little memory. We saw the same issue in Upgrader9to10 and fixed it in #1441. I checked other places in our ITs to see where we set the JVM memory size for processes launched by mini, and this was the only one as low as 16MB. 32MB seems to be a reasonable minimum for G1GC.
Thanks for looking into this @jzgithub1 and @milleruntime . Glad we got it figured it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] jzgithub1 commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
jzgithub1 commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594153515
@milleruntime or @EdColeman , I tailed all of the logs while running the test and when it does fail I see a connection loss to ZooKeeper in the TabletServer log.
`./test/target/mini-tests/org.apache.accumulo.test.functional.GarbageCollectorIT_gcLotsOfCandidatesIT/logs/TabletServer_812067704.out <==
2020-03-03T15:02:01,081 [zookeeper.ZooReader] WARN : Saw (possibly) transient exception communicating with ZooKeeper
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /accumulo/9182a659-cfaf-4a19-b4c7-fb875091fc7b/tservers/CoolJeff:35337`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] jzgithub1 commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
jzgithub1 commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594111519
@milleruntime , I saw:
`Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "org.apache.accumulo.gc.SimpleGarbageCollector"
Exception in thread "Thread-5" java.lang.OutOfMemoryError: Java heap space`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime closed issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
milleruntime closed issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594179085
I am pushing a fix... it had to do with the Java GC not having enough memory to start the process. The test was manually setting the default memory to 16MB, which wasn't enough. The Java GC for mini was changed in 33d8f2039541addc96849bce55e6fff3265957c8
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] jzgithub1 commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
jzgithub1 commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594180052
@milleruntime , thanks a lot. Looking forward to testing it out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] jzgithub1 edited a comment on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
jzgithub1 edited a comment on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594101065
I am not seeing that. But I am running the IT inside of Intellij. I can try running it on the command line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594088076
Looking at the IT, it is waiting for a specific output, which could have changed:
```
while (!output.contains("delete candidates has exceeded")) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] jzgithub1 commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
jzgithub1 commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-593978837
@ctubbsii between 10-20% of the runs I do fail here:
`020-03-02T17:01:02,678 [functional.ConfigurableMacBase] ERROR: Unable to delete ZooLock for mini accumulo-gc
java.lang.IllegalStateException: No lock is held at /accumulo/ebab5cc3-cdad-43fc-980b-c79a4fbda200/gc/lock
at org.apache.accumulo.fate.zookeeper.ZooLock.deleteLock(ZooLock.java:499) ~[classes/:?]
at org.apache.accumulo.test.functional.GarbageCollectorIT.killMacGc(GarbageCollectorIT.java:106) [classes/:?]
at org.apache.accumulo.test.functional.GarbageCollectorIT.gcLotsOfCandidatesIT(GarbageCollectorIT.java:145) [classes/:?]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594091179
> Looking at the IT, it is waiting for a specific output, which could have changed:
>
> ```
> while (!output.contains("delete candidates has exceeded")) {
> ```
The output I think it is waiting for is here:
https://github.com/apache/accumulo/blob/e4b69f976c7223ff7dc3fd930462c323da0ef7ae/server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java#L219-L222
Perhaps with recent improvements, it is not running out of memory so never meets this condition.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] jzgithub1 commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
jzgithub1 commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594097009
I never hit those lines when I set breakpoints in them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594054633
@jzgithub1 thanks for looking at this.
I have seen that ```TestTimedOutException``` but I am not sure how useful that is. The same goes with the similar client exception I have seen:
```
java.lang.IllegalStateException: No lock is held at /accumulo/1d1562ff-4b75-47e1-b7d0-edf08778bd96/gc/lock
at org.apache.accumulo.fate.zookeeper.ZooLock.deleteLock(ZooLock.java:499) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.test.functional.GarbageCollectorIT.killMacGc(GarbageCollectorIT.java:106) [classes/:?]
at org.apache.accumulo.test.functional.GarbageCollectorIT.gcLotsOfCandidatesIT(GarbageCollectorIT.java:145) [classes/:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12]
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12]
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) [junit-4.12.jar:4.12]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) [junit-4.12.jar:4.12]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
```
We need to figure out what is happening on the server side that is causing the timeout. Try doing a ```jstack``` on the pid of one of the IT processes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594114177
OK it looks like the SimpleGarbageCollector is running out of memory before it can call ```almostOutOfMemory``` I posted above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594156282
I think the changes I made in 87ad69931f79c0755ac967d74ad3c9ca3868fa40 changed the timing of the test since it used to run out of memory but it would happen at a different time. I am going to play around with the IT to see if I can get the timing right.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] jzgithub1 commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
jzgithub1 commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-593644739
@milleruntime , I am looking into this. It fails one time out of 10 consecutive runs for me. I am trying to see how I can check if the lock is held before trying to delete it in the killMacGc function in the GarbageCollectorIT. It only times out trying to remove the lock for some reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] jzgithub1 commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
jzgithub1 commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594121328
I wrapped the call gc.runServer() in the main function of SimpleGarbageCollector in a try catch like this and did not hit a breakpoint set in the log.error statement I wrote:
```java
public static void main(String[] args) throws Exception {
try (SimpleGarbageCollector gc = new SimpleGarbageCollector(new ServerOpts(), args)) {
try {
gc.runServer();
} catch (OutOfMemoryError outOfMemoryError) {
log.error("Out of memory error in SimpleGarbageCollector");
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594100874
Are you seeing this in the SimpleGarbageCollector error log?
```bash
cat test/target/mini-tests/org.apache.accumulo.test.functional.GarbageCollectorIT_gcLotsOfCandidatesIT/logs/SimpleGarbageCollector_*.err
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-2"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "org.apache.accumulo.gc.SimpleGarbageCollector"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] EdColeman commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594058617
Could there be a race condition with the shutdown? From one of the stack traces it seemed to hint that the gc was shutdown (which should remove the lock) and then the clean-up is trying to delete the now non-existant lock.
If that is the case, then if gc shutdown is called, the delete could accept that the lock has already been removed by catching / ignoring the node does not exist exception.
Or, is this test also trying to validate the gc behavior w.r.t. losing the lock?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] ctubbsii commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-593684388
@jzgithub1 Do you know where in the code it is getting stuck? Can you run jstack on the process to see what it's doing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] EdColeman commented on issue #1534:
GarbageCollectorIT.gcLotsOfCandidatesIT failure
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #1534: GarbageCollectorIT.gcLotsOfCandidatesIT failure
URL: https://github.com/apache/accumulo/issues/1534#issuecomment-594176742
If things are running out of physical memory I suppose that all sorts of errors could pop up. You might want to check if things are swapping to disk while the test is running. (linux free, top... command) If not, then it could be hitting the JVM limits and doing expensive stop-the-world gcs - might be info in the logs if gc details are enabled, if not you may be able to enable them. On a physical system, just add the jvm options in accumulo-env.sh - but I'm not certain what mini uses, but if not the same, there might be an equivalent
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services