You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2020/03/08 06:49:08 UTC
[GitHub] [accumulo] ctubbsii opened a new issue #1554: Saw periodic failure
of SummaryIT due to missing RFile
ctubbsii opened a new issue #1554: Saw periodic failure of SummaryIT due to missing RFile
URL: https://github.com/apache/accumulo/issues/1554
The following is the error.
```
2020-03-08T02:55:14,387 [thrift.ProcessFunction] ERROR: Internal error processing startGetSummariesFromFiles
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.io.FileNotFoundException: File file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001x.rf does not exist
at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.getSummaries(TabletServer.java:2032) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.startSummaryOperation(TabletServer.java:2046) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.startGetSummariesFromFiles(TabletServer.java:2123) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$8(TraceUtil.java:237) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at com.sun.proxy.$Proxy33.startGetSummariesFromFiles(Unknown Source) ~[?:?]
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startGetSummariesFromFiles.getResult(TabletClientService.java:3607) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$startGetSummariesFromFiles.getResult(TabletClientService.java:3586) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) [libthrift-0.12.0.jar:0.12.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [libthrift-0.12.0.jar:0.12.0]
at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63) [accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) [libthrift-0.12.0.jar:0.12.0]
at org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:114) [accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.server.Invocation.run(Invocation.java:18) [libthrift-0.12.0.jar:0.12.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:37) [accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.io.FileNotFoundException: File file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001x.rf does not exist
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) ~[?:?]
at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.getSummaries(TabletServer.java:2026) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
... 20 more
Caused by: java.io.UncheckedIOException: java.io.FileNotFoundException: File file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001x.rf does not exist
```
Evidence from the logs show the file was previously compacted.
```
2020-03-08T02:55:14,228 [tablet.files] DEBUG: Compacted 1;700000;600000 created file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/C000001z.rf from [file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001x.rf, file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001n.rf, file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001s.rf, file:/var/lib/jenkins/workspace/Accumulo-PR/test/target/mini-tests/org.apache.accumulo.test.functional.SummaryIT_testManyFiles/accumulo/tables/1/t-000003r/F000001i.rf]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] ctubbsii commented on issue #1554: Saw periodic failure
of SummaryIT due to missing RFile
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #1554: Saw periodic failure of SummaryIT due to missing RFile
URL: https://github.com/apache/accumulo/issues/1554#issuecomment-598419696
@jzgithub1 The outdated list of RFiles is my working hypothesis also. I do not know if the test is easily reproducible. Since @keith-turner self-assigned it, I assume he's interested in working on it, but you'd have to check with him. Also, it is likely he knows more about this code anyway, since he implemented much of it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] jzgithub1 commented on issue #1554: Saw periodic failure
of SummaryIT due to missing RFile
Posted by GitBox <gi...@apache.org>.
jzgithub1 commented on issue #1554: Saw periodic failure of SummaryIT due to missing RFile
URL: https://github.com/apache/accumulo/issues/1554#issuecomment-598367826
@ctubbsii , maybe we have another race condition here. The file F000001x.rf is compacted into C000001z.rf. Then F000001x.rf gets deleted (since it had be compacted and wasn't needed anymore - I need to check if that is the way it works) but then the startGetSummariesFromFiles function looks for F000001x.rf after it had been deleted because it is in an outdated list of RFiles. We are seeing a pattern of race conditions in Jenkins being an issue like #1488 that we worked on at the Hackday. Needless to say that I can't replicate this error in the SummaryIT.testManyFiles test. I will work with a race condition hypothesis and try to come up with some solution to this unless you want me to back off of this ticket since I see that @keith-turner self assigned it to himself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services