[jira] [Updated] (HBASE-28589) DoNotRetryException not propagated to client

["ZhenyuLi (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-28589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ZhenyuLi updated HBASE-28589:
-----------------------------
    Summary: DoNotRetryException not propagated to client  (was: HBASE-14598 fix incomplete - DoNotRetryException not propagated to client)

> DoNotRetryException not propagated to client
> --------------------------------------------
>
>                 Key: HBASE-28589
>                 URL: https://issues.apache.org/jira/browse/HBASE-28589
>             Project: HBase
>          Issue Type: Bug
>          Components: IPC/RPC
>    Affects Versions: 2.0.0, 2.4.0, 2.5.0, 2.6.0, 3.0.0
>            Reporter: ZhenyuLi
>            Priority: Major
>
> I have discovered that the fix for HBASE-14598 does not completely resolve the issue, and the problem persists in the latest branches (3.0 and 2.6).
> The original fix for HBASE-14598 addressed two aspects:
>  # When a Scan/Get RPC attempts to allocate an excessively large array that could trigger an OutOfMemoryError (OOM), it checks the array size before allocation and throws a {{BufferOverflowException}} to prevent OOM.
>  # The fix intended to stop client retries for such failures by throwing a {{DoNotRetryException}} when a {{BufferOverflowException}} occurs, as retrying cannot resolve the underlying issue.
> *The Problem:* The {{DoNotRetryException}} is never propagated to the client side. Here's the issue flow:
>  # {{ByteBufferOutputStream.checkSizeAndGrow()}} throws {{BufferOverflowException}}
>  # The exception propagates through the call stack:
>  ** {{ByteBufferOutputStream.checkSizeAndGrow()}}
>  ** {{encoder.write()}}
>  ** {{encodeCellsTo() (Catch BufferOverflowException and turn it into DoNotRetryIOException)}}
>  ** {{this.cellBlockBuilder.buildCellBlockStream()}}
>  ** {{call.setResponse()}}
>  # The {{DoNotRetryException}} is ultimately caught in call.setResponse, where it is merely logged but not sent back to the client
>  # As a result, the client continues retrying indefinitely since the response is null and  netty connection will be closed.
> *Current Status:* In the latest branches (3.0 and 2.6), this issue still exists. In {{{}ServerCall.java{}}}, when {{ALLOCATOR_POOL_ENABLED_KEY}} ({{{}hbase.ipc.server.reservoir.enabled{}}}) is set to {{{}false{}}}, the {{setResponse()}} method follows the same problematic path. If a {{DoNotRetryException}} is thrown in the
> ByteBuffer b = this.cellBlockBuilder.buildCellBlock(this.connection.codec,
> this.connection.compressionCodec, cells); it gets swallowed in the {{setResponse()}} catch block and never reaches the client.
> *Steps to Reproduce:*
>  # Set up a 3-node HBase cluster with 3 RegionServers
>  # Set {{hbase.ipc.server.reservoir.enabled}} to {{false to use ByteBufferOutputStream}}
>  # Inject a {{BufferOverflowException}} at {{ByteBufferOutputStream.checkSizeAndGrow()}} to simulate an OOM condition
>  # Send a scan request
>  # Observe endless client retries
> *Expected Behavior:* The {{DoNotRetryException}} should be properly propagated to the client to prevent retry attempts.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)