[jira] [Commented] (HBASE-29131) Introduce the option for post-compaction validation of HFiles

["Nick Dimiduk (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-29131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927757#comment-17927757 ] 

Nick Dimiduk commented on HBASE-29131:
--------------------------------------

While we encountered this problem in a storefile that was produced by compaction, it's possible that the same error could happen from a storefile produced by memstore flush. As these are two different write paths, perhaps we want to resolve them separately. However, I wouldn't consider the matter "resolved" unless both code paths are covered.

> Introduce the option for post-compaction validation of HFiles
> -------------------------------------------------------------
>
>                 Key: HBASE-29131
>                 URL: https://issues.apache.org/jira/browse/HBASE-29131
>             Project: HBase
>          Issue Type: New Feature
>          Components: Compaction
>            Reporter: Nick Dimiduk
>            Priority: Major
>
> After enabling zstd compression on a table, we experienced an incident where data files produced during compaction were subsequently not readable. The client experienced an exception, the call stack for which is is below. Restoring after this incident was quite annoying, we'd like to avoid repeating the experience.
> The idea is that the region server should read back the files it writes out at the end of the compaction. This extra precaution would function as a last-ditch verification that the output file is wholesome before committing the compaction. If the file is not readable, the compaction would be aborted and we'd try again the next time a compaction is requested.
> {noformat}
> ... INFO  o.a.h.h.c.AsyncRequestFutureImpl - id=1, table=XXX, attempt=4/13, failureCount=250ops, last exception=java.io.IOException: java.io.IOException: Src size is incorrect
> 	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:504)
> 	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> 	at org.apache.hadoop.hbase.ipc.CallRunnerWithContext.run(CallRunnerWithContext.java:103)
> 	at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> 	at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> Caused by: com.github.luben.zstd.ZstdException: Src size is incorrect
> 	at com.github.luben.zstd.ZstdDecompressCtx.decompressDirectByteBuffer(ZstdDecompressCtx.java:172)
> 	at com.github.luben.zstd.ZstdDecompressCtx.decompress(ZstdDecompressCtx.java:241)
> 	at org.apache.hadoop.hbase.io.compress.zstd.ZstdDecompressor.decompress(ZstdDecompressor.java:73)
> 	at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:88)
> 	at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
> 	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:345)
> 	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
> 	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)
> 	at org.apache.hadoop.hbase.io.util.BlockIOUtils.readFullyWithHeapBuffer(BlockIOUtils.java:151)
> 	at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:104)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:644)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1353)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1252)
> 	at org.apache.hadoop.hbase.io.hfile.NoOpIndexBlockEncoder$NoOpEncodedSeeker.loadDataBlockWithScanInfo(NoOpIndexBlockEncoder.java:346)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReaderV2.loadDataBlockWithScanInfo(HFileBlockIndex.java:514)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:674)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:659)
> 	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:338)
> 	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:256)
> 	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:457)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:357)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:302)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:266)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:1102)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:1093)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.seekOrSkipToNextColumn(StoreScanner.java:833)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:734)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:151)
> 	at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.populateResult(RegionScannerImpl.java:339)
> 	at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextInternal(RegionScannerImpl.java:511)
> 	at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextRaw(RegionScannerImpl.java:275)
> 	at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.next(RegionScannerImpl.java:253)
> 	at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.next(RegionScannerImpl.java:240)
> 	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2652)
> 	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:859)
> 	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2892)
> 	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
> 	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:437)
> 	... 4 more
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)