[jira] [Created] (PHOENIX-8023) SYSTEM.CATALOG region unavailable for entire scan duration when closing

["Tanuj Khurana (Jira)" <ji...@apache.org>]

Tanuj Khurana created PHOENIX-8023:
--------------------------------------

             Summary: SYSTEM.CATALOG region unavailable for entire scan duration when closing
                 Key: PHOENIX-8023
                 URL: https://issues.apache.org/jira/browse/PHOENIX-8023
             Project: Phoenix
          Issue Type: Bug
            Reporter: Tanuj Khurana


_MetaDataEndpointImpl.buildTable_ holds the SYSTEM.CATALOG region operation (read lock)
while building a PTable, and on that path the TTL resolution helpers
(getTTLFromHierarchy / getTTLForParentTable) introduced as part of PHOENIX-6978 open a HBase client scan of SYSTEM.CATALOG itself via {_}ServerUtil.getHTableForCoprocessorScan(env, SYSTEM.CATALOG){_}. Once a close sets the
closing flag, those nested scans get '{*}NotServingRegionException ... is closing{*}' and the HBase client *retries for the full 60 s scanner timeout* while the outer handler still holds the region read lock. The close cannot take the write lock until they give up and the scan operation times out. 

 
{code:java}

java.net.SocketTimeoutException: callTimeout=60000, callDuration=68468:
  SYSTEM.CATALOG,,1787178855993.695f91ca... is closing ... hostname=regionserver-sys-2...
  at o.a.h.h.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(...:294)
  at o.a.h.h.client.ScannerCallableWithReplicas.call(...:256)
  at o.a.h.h.client.ClientScanner.next(ClientScanner.java:623)
  at o.a.p.coprocessor.MetaDataEndpointImpl.getTTLFromHierarchy(MetaDataEndpointImpl.java:1837)
  at o.a.p.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1571)
  at o.a.p.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1154)
  at o.a.p.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1149)
  at o.a.p.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:809)
  at o.a.p.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:4085)
  at o.a.p.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:728)
  at ...MetaDataProtos$MetaDataService.callMethod(...)
  at o.a.h.h.regionserver.HRegion.execService(HRegion.java:8011)
  ...
  at o.a.h.h.ipc.RpcHandler.run(RpcHandler.java:82)
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68468: ... is closing
  at o.a.h.h.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:157)
  at o.a.h.h.client.ResultBoundedCompletionService$QueueingFuture.run(...:74)
Caused by: org.apache.hadoop.hbase.NotServingRegionException: SYSTEM.CATALOG,... is closing
  at o.a.h.h.regionserver.HRegion.startRegionOperation(HRegion.java:8130)
  at o.a.h.h.regionserver.HRegion.getScanner(HRegion.java:3059)
  at o.a.h.h.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3205)
  at o.a.h.h.regionserver.RSRpcServices.scan(RSRpcServices.java:3578)
  at o.a.h.h.client.ServerConnectionUtils$ShortCircuitingClusterConnection$ClientServiceBlockingInterfaceWrapper.scan(...:149)
  at o.a.h.h.client.ScannerCallable.openScanner(ScannerCallable.java:363)
  ...
  at o.a.h.h.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:103){code}
 

This happens twice because on a graceful shutdown syscat region first moves from the original regionserver to another regionserver and then again moves back to the original regionserver.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)