[jira] [Updated] (HBASE-29131) Introduce the option for post-compaction validation of HFiles

["Nick Dimiduk (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-29131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nick Dimiduk updated HBASE-29131:
---------------------------------
    Release Note: 
Introduces the option for an HStore to fully read the file it just wrote after a flush or compaction. It does this by walking the block index and reading the first cell off of each data block. During testing, this was observed to have a negative impact on compaction queue depth of about 20%.

To enable this feature, set hbase.hstore.validate.read_fully=true. This is an HStore configuration feature, so it can be enabled in hbase-site.xml, in the TableDescriptor, or in the ColumnFamilyDescriptor.

> Introduce the option for post-compaction validation of HFiles
> -------------------------------------------------------------
>
>                 Key: HBASE-29131
>                 URL: https://issues.apache.org/jira/browse/HBASE-29131
>             Project: HBase
>          Issue Type: New Feature
>          Components: Compaction
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>            Priority: Major
>              Labels: pull-request-available
>
> After enabling zstd compression on a table, we experienced an incident where data files produced during compaction were subsequently not readable. The client experienced an exception, the call stack for which is is below. Restoring after this incident was quite annoying, we'd like to avoid repeating the experience.
> The idea is that the region server should read back the files it writes out at the end of the compaction. This extra precaution would function as a last-ditch verification that the output file is wholesome before committing the compaction. If the file is not readable, the compaction would be aborted and we'd try again the next time a compaction is requested.
> {noformat}
> ... INFO  o.a.h.h.c.AsyncRequestFutureImpl - id=1, table=XXX, attempt=4/13, failureCount=250ops, last exception=java.io.IOException: java.io.IOException: Src size is incorrect
> 	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:504)
> 	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> 	at org.apache.hadoop.hbase.ipc.CallRunnerWithContext.run(CallRunnerWithContext.java:103)
> 	at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> 	at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> Caused by: com.github.luben.zstd.ZstdException: Src size is incorrect
> 	at com.github.luben.zstd.ZstdDecompressCtx.decompressDirectByteBuffer(ZstdDecompressCtx.java:172)
> 	at com.github.luben.zstd.ZstdDecompressCtx.decompress(ZstdDecompressCtx.java:241)
> 	at org.apache.hadoop.hbase.io.compress.zstd.ZstdDecompressor.decompress(ZstdDecompressor.java:73)
> 	at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:88)
> 	at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
> 	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:345)
> 	at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)
> 	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399)
> 	at org.apache.hadoop.hbase.io.util.BlockIOUtils.readFullyWithHeapBuffer(BlockIOUtils.java:151)
> 	at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:104)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:644)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1353)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1252)
> 	at org.apache.hadoop.hbase.io.hfile.NoOpIndexBlockEncoder$NoOpEncodedSeeker.loadDataBlockWithScanInfo(NoOpIndexBlockEncoder.java:346)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReaderV2.loadDataBlockWithScanInfo(HFileBlockIndex.java:514)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:674)
> 	at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:659)
> 	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:338)
> 	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:256)
> 	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:457)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:357)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:302)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:266)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:1102)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:1093)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.seekOrSkipToNextColumn(StoreScanner.java:833)
> 	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:734)
> 	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:151)
> 	at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.populateResult(RegionScannerImpl.java:339)
> 	at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextInternal(RegionScannerImpl.java:511)
> 	at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextRaw(RegionScannerImpl.java:275)
> 	at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.next(RegionScannerImpl.java:253)
> 	at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.next(RegionScannerImpl.java:240)
> 	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2652)
> 	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:859)
> 	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2892)
> 	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961)
> 	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:437)
> 	... 4 more
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)