You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Kyle Purtell (Jira)" <ji...@apache.org> on 2023/06/23 22:56:00 UTC
[jira] [Updated] (HBASE-27951) Use ADMIN_QOS in MasterRpcServices for regionServerStartup and regionServerReport
[ https://issues.apache.org/jira/browse/HBASE-27951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Kyle Purtell updated HBASE-27951:
----------------------------------------
Description:
Analysis of a recent production incident is not yet complete but an item of note is an apparent deadlock. Imagine you are gracefully draining a regionserver by way of a flurry of moveRegion requests. The handler for moveRegion submits a TRSP and then waits on its future without timeout. Imagine that there are sufficient number of moveRegion requests to tie up the normal priority master RPC pool. Now imagine that all of those requests are waiting on TRSPs pending on a regionserver that is concurrently bounced or maybe it fails. The TRSPs are blocked in REGION_STATE_TRANSITION_CLOSE because the target regionserver terminated before responding to the close requests, blocking the moveRegion requests, blocking the RPC handlers. The regionserver restarts and tries to check in, but cannot report to the master because regionServerReport is a normal priority admin RPC and there are no free normal priority handlers to handle it. It seems not correct to have regionServerStartup and regionServerReport, which is important, contending with normal priority requests.
They should be made ADMIN_QOS priority to avoid this case.
was:
Analysis of a recent production incident is not yet complete but an item of note is an apparent deadlock. Imagine you are gracefully draining a regionserver by way of a flurry of moveRegion requests. The handler for moveRegion submits a TRSP and then waits on its future without timeout. Imagine that there are sufficient number of moveRegion requests to tie up the normal priority master RPC pool. Now imagine that all of those requests are waiting on TRSPs pending on a regionserver that is concurrently bounced or maybe it fails. The TRSPs are blocked in REGION_STATE_TRANSITION_CLOSE because the target regionserver terminated before responding to the close requests, blocking the moveRegion requests, blocking the RPC handlers. The regionserver restarts and tries to check in, but cannot report to the master because regionServerReport is a normal priority admin RPC and there are no free normal priority handlers to handle it. It seems not correct to have regionServerReport, which is important, contending with normal priority requests.
regionServerReport should be made ADMIN_QOS to avoid this case.
> Use ADMIN_QOS in MasterRpcServices for regionServerStartup and regionServerReport
> ---------------------------------------------------------------------------------
>
> Key: HBASE-27951
> URL: https://issues.apache.org/jira/browse/HBASE-27951
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.4.10
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Major
> Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1
>
>
> Analysis of a recent production incident is not yet complete but an item of note is an apparent deadlock. Imagine you are gracefully draining a regionserver by way of a flurry of moveRegion requests. The handler for moveRegion submits a TRSP and then waits on its future without timeout. Imagine that there are sufficient number of moveRegion requests to tie up the normal priority master RPC pool. Now imagine that all of those requests are waiting on TRSPs pending on a regionserver that is concurrently bounced or maybe it fails. The TRSPs are blocked in REGION_STATE_TRANSITION_CLOSE because the target regionserver terminated before responding to the close requests, blocking the moveRegion requests, blocking the RPC handlers. The regionserver restarts and tries to check in, but cannot report to the master because regionServerReport is a normal priority admin RPC and there are no free normal priority handlers to handle it. It seems not correct to have regionServerStartup and regionServerReport, which is important, contending with normal priority requests.
> They should be made ADMIN_QOS priority to avoid this case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)