You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Toshihiro Suzuki (Jira)" <ji...@apache.org> on 2020/06/07 06:55:00 UTC
[jira] [Updated] (HBASE-24515) batch Increment/Append fails when
retrying the RPC
[ https://issues.apache.org/jira/browse/HBASE-24515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Toshihiro Suzuki updated HBASE-24515:
-------------------------------------
Description:
When a client hits RPC timeout and sends a second RPC request for batch Increment/Append but the first RPC is already processed actually, the nonce of the RPC is saved in the RS.
In this case, for the second RPC, the RS just reads the previous result and returns it to the client to avoid the Increment/Append is processed twice.
At that time, for batch Increment/Append, we try to create a Get object from a CellScanner object in the following code:
[https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L773-L776]
However, the CellScanner object is already consumed to create the Increment/Append object in the following code, and it fails with the following exception:
[https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L757]
{code:java}
2020-06-06 14:09:06,153 WARN [hconnection-0x79c3903e-shared-pool3-t209] client.AsyncRequestFutureImpl: id=1, table=REF_Test, attempt=3/36, failureCount=1ops, last exception=org.apache.hadoop.hbase.DoNotRetryIOException:
org.apache.hadoop.hbase.DoNotRetryIOException: Cell count of 1 but at index 0 no cell returned: row: "xxxxxxxxxxx" mutate_type: INCREMENT timestamp: 9223372036854775807 durability: USE_DEFAULT time_range { from: 0 to: 9223372036854775807 } associated_cell_count: 1 nonce: 5281583076322914765
at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:934)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:737)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2702)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42202)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
was:
When a client hits RPC timeout and sends a second RPC request for batch Increment/Append but the first RPC is already processed actually, the nonce of the RPC is saved in the RS.
In this case, for the second RPC, the RS just reads the previous result and returns it to the client to avoid the Increment/Append is processed twice.
At that time, for batch Increment/Append, we try to create a Get object from a CellScanner object in the following code:
[https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L773-L776]
However, the CellScanner object is already consumed to create the Increment/Append object as follows, and it fails:
[https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L757]
> batch Increment/Append fails when retrying the RPC
> --------------------------------------------------
>
> Key: HBASE-24515
> URL: https://issues.apache.org/jira/browse/HBASE-24515
> Project: HBase
> Issue Type: Bug
> Reporter: Toshihiro Suzuki
> Assignee: Toshihiro Suzuki
> Priority: Major
>
> When a client hits RPC timeout and sends a second RPC request for batch Increment/Append but the first RPC is already processed actually, the nonce of the RPC is saved in the RS.
> In this case, for the second RPC, the RS just reads the previous result and returns it to the client to avoid the Increment/Append is processed twice.
> At that time, for batch Increment/Append, we try to create a Get object from a CellScanner object in the following code:
> [https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L773-L776]
> However, the CellScanner object is already consumed to create the Increment/Append object in the following code, and it fails with the following exception:
> [https://github.com/apache/hbase/blob/66452afc09d8b82927e5e58565f97939faa22c7b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L757]
> {code:java}
> 2020-06-06 14:09:06,153 WARN [hconnection-0x79c3903e-shared-pool3-t209] client.AsyncRequestFutureImpl: id=1, table=REF_Test, attempt=3/36, failureCount=1ops, last exception=org.apache.hadoop.hbase.DoNotRetryIOException:
> org.apache.hadoop.hbase.DoNotRetryIOException: Cell count of 1 but at index 0 no cell returned: row: "xxxxxxxxxxx" mutate_type: INCREMENT timestamp: 9223372036854775807 durability: USE_DEFAULT time_range { from: 0 to: 9223372036854775807 } associated_cell_count: 1 nonce: 5281583076322914765
> at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:934)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:737)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:877)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2702)
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42202)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)