You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Csaba Ringhofer (Jira)" <ji...@apache.org> on 2024/05/15 15:24:00 UTC

[jira] [Comment Edited] (HBASE-28595) Losing exception from scan RPC can lead to partial results

    [ https://issues.apache.org/jira/browse/HBASE-28595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846668#comment-17846668 ] 

Csaba Ringhofer edited comment on HBASE-28595 at 5/15/24 3:23 PM:
------------------------------------------------------------------

Thanks [~zhangduo] for the quick response!

Uploaded a patch based on branch-2.2 that fixes the issue on the client side:
https://github.com/csringhofer/hbase/commit/1090407bb2c78aaa839b56a5bbcd8c7ddbbd489f

Generally I think that a server side fix would be cleaner (ensuring that after exceptions all later scan calls on the same scanner also return an exception). But I was worried about compatibility issues and the client side fix seemed the safest.


was (Author: csringhofer):
Thanks [~zhangduo] for the quick response!

Uploaded a patch based on branch-2.2 that fixes the issue on the client side:
https://github.com/csringhofer/hbase/commit/c24c2eeb138cbcd15164a2890aaf408057584c66

Generally I think that a server side fix would be cleaner (ensuring that after exceptions all later scan calls on the same scanner also return an exception). But I was worried about compatibility issues and the client side fix seemed the safest.

> Losing exception from scan RPC can lead to partial results
> ----------------------------------------------------------
>
>                 Key: HBASE-28595
>                 URL: https://issues.apache.org/jira/browse/HBASE-28595
>             Project: HBase
>          Issue Type: Bug
>          Components: Client, regionserver, Scanners
>            Reporter: Csaba Ringhofer
>            Assignee: Csaba Ringhofer
>            Priority: Critical
>
> This was discovered in Apache Impala using HBase 2.2 based branch hbase client and server. It is not clear yet whether other branches are also affected.
> The issue happens if the server side of the scan throws an exception and closes the scanner, but at the same time, the client gets an rpc connection closed error and doesn't process the exception sent by the server. Client then thinks it got a network error, which leads to retrying the RPC instead of opening a new scanner. But then when the client retry reaches the server, the server returns an empty ScanResponse instead of an error, leading to closing the scanner on client side without returning any error.
> A few pointers to critical parts:
> region server:
> 1st call throws exception leading to closing (but not deleting) scanner:
> [https://github.com/apache/hbase/blob/0c8607a35008b7dca15e9daaec41ec362d159d67/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3539]
> 2nd call (retry of 1st) returns empty results:
> [https://github.com/apache/hbase/blob/0c8607a35008b7dca15e9daaec41ec362d159d67/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L3403]
> client:
> some exceptions are handled as non-retriable at RPC level and are only handled through opening a new scanner:
> [https://github.com/apache/hbase/blob/0c8607a35008b7dca15e9daaec41ec362d159d67/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java#L214]
> [https://github.com/apache/hbase/blob/0c8607a35008b7dca15e9daaec41ec362d159d67/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java#L367]
> This mechanism in the client only works if it gets the exception from the server. If there are connection issues during the RPC then the client won't really know the state of the server.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)