You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/10/19 15:25:29 UTC
[GitHub] [accumulo] keith-turner opened a new issue, #3031: Killing tserver caused scan using combiner to error out
keith-turner opened a new issue, #3031:
URL: https://github.com/apache/accumulo/issues/3031
**Describe the bug**
While running the random walk bulk import test I killed tablet servers inorder to restart them with more memory. This should not have impacted the test that was running, however I saw the test fail with the following error.
```
2022-10-19T13:16:20,224 [testing.randomwalk.Framework] ERROR: Error during random walk
java.lang.Exception: Error running node bulk.Verify
at org.apache.accumulo.testing.randomwalk.Module.visit(Module.java:360) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.testing.randomwalk.Framework.run(Framework.java:54) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.testing.randomwalk.Framework.main(Framework.java:98) ~[accumulo-testing-shaded.jar:?]
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server localhost:10003
at org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:177) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.clientImpl.ScannerIterator.hasNext(ScannerIterator.java:106) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.util.PeekingIterator.next(PeekingIterator.java:70) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.client.RowIterator$SingleRowIter._next(RowIterator.java:76) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.client.RowIterator$SingleRowIter.next(RowIterator.java:69) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.client.RowIterator$SingleRowIter.next(RowIterator.java:42) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.testing.randomwalk.bulk.Verify.visit(Verify.java:86) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.testing.randomwalk.Module.lambda$visit$0(Module.java:295) ~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.util.concurrent.ExecutionException: org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server localhost:10003
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
at org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:172) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.clientImpl.ScannerIterator.hasNext(ScannerIterator.java:106) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.util.PeekingIterator.next(PeekingIterator.java:70) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.client.RowIterator$SingleRowIter._next(RowIterator.java:76) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.client.RowIterator$SingleRowIter.next(RowIterator.java:69) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.client.RowIterator$SingleRowIter.next(RowIterator.java:42) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.testing.randomwalk.bulk.Verify.visit(Verify.java:86) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.testing.randomwalk.Module.lambda$visit$0(Module.java:295) ~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.apache.accumulo.core.clientImpl.AccumuloServerException: Error on server localhost:10003
at org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:363) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.clientImpl.ScannerIterator.readBatch(ScannerIterator.java:152) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.trace.TraceWrappedCallable.call(TraceWrappedCallable.java:53) ~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.apache.thrift.TApplicationException: Internal error processing continueScan
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:81) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.tabletserver.thrift.TabletScanClientService$Client.recv_continueScan(TabletScanClientService.java:162) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.tabletserver.thrift.TabletScanClientService$Client.continueScan(TabletScanClientService.java:147) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.clientImpl.ThriftScanner.scanRpc(ThriftScanner.java:646) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:575) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:354) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.clientImpl.ScannerIterator.readBatch(ScannerIterator.java:152) ~[accumulo-testing-shaded.jar:?]
at org.apache.accumulo.core.trace.TraceWrappedCallable.call(TraceWrappedCallable.java:53) ~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2022-10-19T13:16:20,233 [testing.randomwalk.Framework] INFO : Test finished
```
On one of the tservers, saw the following error.
```
2022-10-19T13:16:20,210 [thrift.ProcessFunction] ERROR: Internal error processing continueScan
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Filesystem closed
at org.apache.accumulo.tserver.ThriftScanClientHandler.continueScan(ThriftScanClientHandler.java:281) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.ThriftScanClientHandler.continueScan(ThriftScanClientHandler.java:242) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$0(TraceUtil.java:206) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at com.sun.proxy.$Proxy37.continueScan(Unknown Source) ~[?:?]
at org.apache.accumulo.core.tabletserver.thrift.TabletScanClientService$Processor$continueScan.getResult(TabletScanClientService.java:788) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.tabletserver.thrift.TabletScanClientService$Processor$continueScan.getResult(TabletScanClientService.java:764) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40) ~[libthrift-0.17.0.jar:0.17.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) ~[libthrift-0.17.0.jar:0.17.0]
at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:147) ~[libthrift-0.17.0.jar:0.17.0]
at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:54) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:492) ~[libthrift-0.17.0.jar:0.17.0]
at org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:129) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.server.Invocation.run(Invocation.java:18) ~[libthrift-0.17.0.jar:0.17.0]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Filesystem closed
at org.apache.accumulo.tserver.scan.ScanTask.get(ScanTask.java:158) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.ThriftScanClientHandler.continueScan(ThriftScanClientHandler.java:263) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
... 20 more
Caused by: java.lang.RuntimeException: java.io.IOException: Filesystem closed
at org.apache.accumulo.core.iterators.Combiner$ValueIterator.next(Combiner.java:133) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.Combiner$ValueIterator.next(Combiner.java:96) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.TypedValueCombiner$VIterator.hasNext(TypedValueCombiner.java:90) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.user.SummingCombiner.typedReduce(SummingCombiner.java:33) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.user.SummingCombiner.typedReduce(SummingCombiner.java:29) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.TypedValueCombiner.reduce(TypedValueCombiner.java:180) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.Combiner.findTop(Combiner.java:225) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.Combiner.next(Combiner.java:181) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:95) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:100) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:60) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:178) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:149) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.TabletBase.nextBatch(TabletBase.java:288) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.Scanner.read(Scanner.java:108) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.scan.NextBatchTask.run(NextBatchTask.java:75) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.session.ScanSession$ScanMeasurer.run(ScanSession.java:62) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
... 5 more
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:494) ~[hadoop-client-api-3.3.3.jar:?]
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:866) ~[hadoop-client-api-3.3.3.jar:?]
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:957) ~[hadoop-client-api-3.3.3.jar:?]
at java.io.DataInputStream.read(DataInputStream.java:149) ~[?:?]
at org.apache.accumulo.core.file.streams.RateLimitedInputStream.read(RateLimitedInputStream.java:52) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.io.DataInputStream.read(DataInputStream.java:149) ~[?:?]
at org.apache.accumulo.core.file.streams.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:179) ~[hadoop-client-api-3.3.3.jar:?]
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:163) ~[hadoop-client-api-3.3.3.jar:?]
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105) ~[hadoop-client-api-3.3.3.jar:?]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:252) ~[?:?]
at java.io.BufferedInputStream.read(BufferedInputStream.java:271) ~[?:?]
at java.io.FilterInputStream.read(FilterInputStream.java:83) ~[?:?]
at java.io.DataInputStream.readByte(DataInputStream.java:270) ~[?:?]
at org.apache.accumulo.core.file.rfile.RelativeKey.readFields(RelativeKey.java:164) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._next(RFile.java:859) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.next(RFile.java:824) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iteratorsImpl.system.HeapIterator.next(HeapIterator.java:74) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.server.problems.ProblemReportingIterator.next(ProblemReportingIterator.java:86) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iteratorsImpl.system.HeapIterator.next(HeapIterator.java:74) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iteratorsImpl.system.StatsIterator.next(StatsIterator.java:51) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iteratorsImpl.system.DeletingIterator.next(DeletingIterator.java:65) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.ServerSkippingIterator.next(ServerSkippingIterator.java:45) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.SynchronizedServerFilter.next(SynchronizedServerFilter.java:51) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.Combiner$ValueIterator.next(Combiner.java:130) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.Combiner$ValueIterator.next(Combiner.java:96) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.TypedValueCombiner$VIterator.hasNext(TypedValueCombiner.java:90) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.user.SummingCombiner.typedReduce(SummingCombiner.java:33) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.user.SummingCombiner.typedReduce(SummingCombiner.java:29) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.TypedValueCombiner.reduce(TypedValueCombiner.java:180) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.Combiner.findTop(Combiner.java:225) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.Combiner.next(Combiner.java:181) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.WrappingIterator.next(WrappingIterator.java:95) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.user.VersioningIterator.skipRowColumn(VersioningIterator.java:100) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iterators.user.VersioningIterator.next(VersioningIterator.java:60) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:178) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:149) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.TabletBase.nextBatch(TabletBase.java:288) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.Scanner.read(Scanner.java:108) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.scan.NextBatchTask.run(NextBatchTask.java:75) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.session.ScanSession$ScanMeasurer.run(ScanSession.java:62) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
... 5 more
```
When the process gets a SIGTERM this causes the hadoop filesystem to run its shutdown hook which causes this exception. In the [tablet server scan code](https://github.com/apache/accumulo/blob/78142d74f3ed1a23a374f5bd5fdbd7f245a16721/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Scanner.java#L127-L132) Accumulo tries to work around this by detecting it and wrapping the IOException with something that will cause the client to retry. That is done only done when IOException is seen. The bulk test had configured a combiner, and [the combiner was wrapping the IOException with a RuntimeException](https://github.com/apache/accumulo/blob/78142d74f3ed1a23a374f5bd5fdbd7f245a16721/core/src/main/java/org/apache/accumulo/core/iterators/Combiner.java#L132-L134) evading the check.
**Expected behavior**
An Accumulo scanner should not fail when a tablet server is killed, it should retry. Thinking one possible way to achive this is to do the following.
* Make the code in Combiner throw an UncheckedIOException
* Make the code tablet server scan code catch UncheckedIOException and do the shutdown check.
* Look for other places in the code that catch IOException and throw runtime exception and replace those with UncheckedIOException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #3031: Killing tserver caused scan using combiner to error out
Posted by GitBox <gi...@apache.org>.
keith-turner commented on issue #3031:
URL: https://github.com/apache/accumulo/issues/3031#issuecomment-1284206163
Another possible solution to this problem is to do more comprehensive analysis of exception causes [like this code does](https://github.com/apache/accumulo/blob/78142d74f3ed1a23a374f5bd5fdbd7f245a16721/core/src/main/java/org/apache/accumulo/fate/Fate.java#L138-L178). This is a more robust solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner closed issue #3031: Killing tserver caused scan using combiner to error out
Posted by GitBox <gi...@apache.org>.
keith-turner closed issue #3031: Killing tserver caused scan using combiner to error out
URL: https://github.com/apache/accumulo/issues/3031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #3031: Killing tserver caused scan using combiner to error out
Posted by GitBox <gi...@apache.org>.
keith-turner commented on issue #3031:
URL: https://github.com/apache/accumulo/issues/3031#issuecomment-1285344031
Thanks @ctubbsii for setting the project metadata on this issue. I'll try to remember to do that in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3031: Killing tserver caused scan using combiner to error out
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #3031:
URL: https://github.com/apache/accumulo/issues/3031#issuecomment-1284549260
Fixed by #3032
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org