[jira] [Commented] (HBASE-29907) ROWCOL bloom filter + StoreScanner.trySkipToNextColumn can surface out-of-order cells, causing read failure “isDelete failed”

["Daniel Roudnitsky (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-29907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18059258#comment-18059258 ] 

Daniel Roudnitsky commented on HBASE-29907:
-------------------------------------------

This test creates multiple StoreFiles with ROWCOL bloom filters and small block sizes, then runs concurrent scans while a background thread continuously writes, flushes, and triggers minor compactions. The combination of bloom filter fake cell optimization, multiple StoreFiles, and concurrent flush/scan interleaving triggers the backward jump that the buggy check fails to catch.

Hoping to get some opinions on whether its appropriate to commit this style of longer running stress test which only validates that an exception does not get thrown in the course of the stress test

> ROWCOL bloom filter + StoreScanner.trySkipToNextColumn can surface out-of-order cells, causing read failure “isDelete failed”
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-29907
>                 URL: https://issues.apache.org/jira/browse/HBASE-29907
>             Project: HBase
>          Issue Type: Bug
>          Components: Filters, Scanners
>    Affects Versions: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1, 2.5.7, 2.5.8, 2.6.1, 2.5.9, 2.5.10, 2.5.11, 2.6.2, 2.6.3, 2.5.12, 2.6.4, 2.5.13
>            Reporter: Jize Ning
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Summary
> We see intermittent read failures (Multi-column GET) when a column family uses ROWCOL bloom filters. Clients fail with an exception chain that includes:
> {code:java}
> 2026-02-17T07:49:24.041Z, RpcRetryingCaller{globalStartTime=2026-02-17T07:49:23.547Z, pause=250, maxAttempts=3}, java.io.IOException: java.io.IOException: isDelete failed: deleteBuffer=q15, qualifier=q09, timestamp=2010, comparison result: 1
>  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:479)
>  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
>  at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
>  at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
> Caused by: java.lang.IllegalStateException: isDelete failed: deleteBuffer=q15, qualifier=q09, timestamp=2010, comparison result: 1
>  at org.apache.hadoop.hbase.regionserver.querymatcher.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:132)
>  at org.apache.hadoop.hbase.regionserver.querymatcher.ScanQueryMatcher.checkDeleted(ScanQueryMatcher.java:204)
>  at org.apache.hadoop.hbase.regionserver.querymatcher.NormalUserScanQueryMatcher.match(NormalUserScanQueryMatcher.java:76)
>  at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:624)
>  at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145)
>  at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.populateResult(RegionScannerImpl.java:342)
>  at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextInternal(RegionScannerImpl.java:513)
>  at org.apache.hadoop.hbase.regionserver.RegionScannerImpl.nextRaw(RegionScannerImpl.java:278)
>  at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3402)
>  at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3668)
>  at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45006)
>  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415) {code}
>  
> h3. Suspected root cause
> This appears to be the same class of issue described in HBASE-19863, and seems to regress with the narrower guard introduced by HBASE-28055. When ROWCOL bloom indicates a row+qualifier is absent from a StoreFile, the scanner may use a bloom-optimized “fake key”. If such a fake key is consumed during the trySkipToNextColumn skip loop, a subsequent next() can advance from a stale physical HFile position and return a cell that sorts before the column being skipped. When that reaches delete-tracking (isDelete), the read can fail and surface as isDelete failed.
>  
> the HBASE-28055 change *narrowed the safety check* in a way that is {*}semantically wrong{*}:
>  * {*}HBASE-19863's check ({{{}compareKeyForNextColumn < 0{}}}){*}: Catches _any_ backward ordering of the next cell relative to the current cell's expected next column. This is a *general* guard — it handles any case where consuming cells in the loop caused the heap to surface a cell that violates ordering, regardless of whether it was a bloom fake key or something else.
>  * {*}HBASE-28055's check ({{{}timestamp == OLDEST_TIMESTAMP{}}}){*}: Only catches the case where {{cell}} itself is a bloom filter fake key (since fake keys are created with {{{}OLDEST_TIMESTAMP{}}}). But the problem scenario described in HBASE-19863 is that the bloom fake key gets *consumed inside the loop* by {{{}heap.next(){}}}, and the _next_ real cell that surfaces is now out of order. In that case, {{cell}} (the trigger cell passed into {{{}trySkipToNextColumn{}}}) is a *real cell* with a real timestamp — _not_ {{{}OLDEST_TIMESTAMP{}}}. The check misses it entirely
> h3. Impact / correctness concerns
> Beyond the immediate isDelete failed read failures, this indicates a deeper correctness issue: the scan pipeline relies on the invariant that KeyValueHeap delivers Cells in non-decreasing key order across all participating scanners. When ROWCOL bloom + trySkipToNextColumn can result in a “smaller key” being surfaced after a larger key (i.e., a backward jump), the heap’s ordering guarantee is effectively violated from the perspective of consumers (e.g., delete tracking and matchers). 
>  
> We were able to reproduce the bug with both a mini-cluster stress test and a small deterministic unit test. It would hit exception with the HBASE-28055 fix but would not hit it with HBASE-19863 fix. 
> *Proposed fix* 
> We should revert the change in HBASE-28055. The claimed "performance improvement" comes from skipping the reseek that should be used to fix the heap ordering.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)