[jira] [Created] (PHOENIX-6331) Increase index retry from 1 to 2 incase of NotServingRegionException

["Ankit Singhal (Jira)" <ji...@apache.org>]

Ankit Singhal created PHOENIX-6331:
--------------------------------------

             Summary: Increase index retry from 1 to 2 incase of NotServingRegionException
                 Key: PHOENIX-6331
                 URL: https://issues.apache.org/jira/browse/PHOENIX-6331
             Project: Phoenix
          Issue Type: Bug
            Reporter: Ankit Singhal


Currently, we move the index to PENDING_DISABLE whenever the single write to index failed and carry out a retry at the client but this can be optimized for NotServingRegionException as Index regions can move very frequently depending on the balancer and one more retry at the server could avoid unnecessary handling of index states and retries at the client.

 
{code:java}
2021-01-20 06:54:58,682 WARN org.apache.hadoop.hbase.client.AsyncProcess: #277, table=<index_table>, attempt=1/1 failed=1ops, last exception: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region <region> is not online on <region_server>
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2997)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1069)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2100)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)
 on <region_server>, tracking started Wed Jan 20 06:54:58 CET 2021; not retrying 1 - final failure
2021-01-20 06:54:58,690 INFO org.apache.phoenix.index.PhoenixIndexFailurePolicy: Successfully update INDEX_DISABLE_TIMESTAMP for <index_table> due to an exception while writing updates. indexState=PENDING_DISABLE
org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException:  disableIndexOnFailure=true, Failed to write to multiple index tables: [<index_table>]
        at org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:236)
        at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:195)
        at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:156)
        at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:145)
        at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:617)
        at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:577)
        at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:560)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1034)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1030)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3421)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2944)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2886)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:765)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:716)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2146)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)
2021-01-20 06:54:58,691 INFO org.apache.phoenix.hbase.index.util.IndexManagementUtil: Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 1121 (XCL21): Write to the index failed.  disableIndexOnFailure=true, Failed to write to multiple index tables: [<index_table>] ,serverTimestamp=1611122098649,
2021-01-20 06:55:01,296 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for hbase (auth:SIMPLE) {code}
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)