[jira] [Created] (HBASE-18141) Regionserver fails to shutdown when abort triggered in RegionScannerImpl during RPC call

["Gary Helmling (JIRA)" <ji...@apache.org>]

Gary Helmling created HBASE-18141:
-------------------------------------

             Summary: Regionserver fails to shutdown when abort triggered in RegionScannerImpl during RPC call
                 Key: HBASE-18141
                 URL: https://issues.apache.org/jira/browse/HBASE-18141
             Project: HBase
          Issue Type: Bug
          Components: regionserver, security
    Affects Versions: 1.3.1
            Reporter: Gary Helmling
            Assignee: Gary Helmling
            Priority: Critical
             Fix For: 1.3.2


When an abort is triggered within the RPC call path by HRegion.RegionScannerImpl, AccessController is incorrectly apply the RPC caller identity in the RegionServerObserver.preStopRegionServer() hook.  This leaves the regionserver in a non-responsive state, where its regions are not reassigned and it returns exceptions for all requests.

When an abort is triggered on the server side, we should not allow a coprocessor to reject the abort at all.

Here is a sample stack trace:
{noformat}
17/05/25 06:10:29 FATAL regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, org.apache.hadoop.hbase.security.token.TokenProvider]
17/05/25 06:10:29 WARN regionserver.HRegionServer: The region server did not stop
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'rpcuser' (global, action=ADMIN)
        at org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:548)
        at org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
        at org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
        at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:86)
        at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execShutdown(RegionServerCoprocessorHost.java:300)
        at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:82)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2118)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2125)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.abortRegionServer(HRegion.java:6326)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleFileNotFound(HRegion.java:6319)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5941)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6084)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5858)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2649)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2320)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
{noformat}

I haven't yet evaluated which other release branches this might apply to.

I have a patch currently in progress, which I will post as soon as I complete a test case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)