[jira] [Resolved] (HBASE-29627) Handle any block cache fetching errors when reading a block in HFileReaderImpl

["Wellington Chevreuil (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-29627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wellington Chevreuil resolved HBASE-29627.
------------------------------------------
    Resolution: Fixed

Merged into master, branch-3, branch-2 and branch-2.6. Thanks for reviewing it, [~psomogyi] !

> Handle any block cache fetching errors when reading a block in HFileReaderImpl
> ------------------------------------------------------------------------------
>
>                 Key: HBASE-29627
>                 URL: https://issues.apache.org/jira/browse/HBASE-29627
>             Project: HBase
>          Issue Type: Improvement
>          Components: BlockCache
>    Affects Versions: 3.0.0-beta-1, 2.7.0
>            Reporter: Wellington Chevreuil
>            Assignee: Wellington Chevreuil
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 2.7.0, 3.0.0-beta-2, 2.6.4
>
>
> One of our customers had faced a situation where blocks are getting cached uncompressed into bucket cache (due to this other issue reported in HBASE-29623) following flushes or compactions, then at read time, during cache retrieval in HFileReaderImpl, it fails to decode the block accordingly, throwing below uncaught exception, failing the read indefinitely: 
> {noformat}
> 2025-09-17 06:38:25,607 ERROR org.apache.hadoop.hbase.regionserver.CompactSplit: Compaction failed region=XXXXXXXXXXXXXXXXXXXXXX,1721528038124.a3012627f502c78738430343b0b54966., storeName=a3012627f502c78738430343b0b54966/0, priority=45, startTime=1758091104691
> java.lang.IllegalArgumentException: There is no data block encoder for given id '5'
>         at org.apache.hadoop.hbase.io.encoding.DataBlockEncoding.getEncodingById(DataBlockEncoding.java:157)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock.getDataBlockEncoding(HFileBlock.java:2003)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.getCachedBlock(HFileReaderImpl.java:1122)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1288)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1249)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readNextDataBlock(HFileReaderImpl.java:750)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$EncodedScanner.next(HFileReaderImpl.java:1528)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:194)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:112)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:681)
>         at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:437)
>         at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:354) {noformat}
> I suspect this mostly related to an error in calculating the meta_space initial offset, due to some extra byte in the byte buffer (something in the line of this [comment from HBASE-27053|https://issues.apache.org/jira/browse/HBASE-27053?focusedCommentId=17564026&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17564026]). If the buffer has extra bytes, we could miss calculate the meta space offset, reading a wrong byte value for the "usesChecksum" flag, which then would lead to wrong header size calculation (24 for no checksum, 33 for checksum), then leading to a wrong positioning for reading the encoding type short. 
> Unfortunately, I could not reproduce this issue on a controlled test environment. However, I think we should still change HFileReaderImpl to handle any possible exception happening when retrieving blocks from the cache, so that instead of failing the whole read operation, it should evict the given corrupt block from the cache and resort to read it from the file system.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)