You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Haiping lv (Jira)" <ji...@apache.org> on 2023/09/27 00:30:00 UTC

[jira] [Updated] (HBASE-28113) All threads of HMaster PEWorker are blocked at updateRegionLocation and preventing the processing of TRSP for the Meta region.

     [ https://issues.apache.org/jira/browse/HBASE-28113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Haiping lv updated HBASE-28113:
-------------------------------
    Description: 
HBase ** Cluster description: *1 master and 5 region servers*

During the execution of itbll process, when ChaosMonkey performs RestartRandomRsAction, it triggers this issue.

 ** The steps for the RestartRandomRsAction operation are as follows.{*}:{*}
 # stop node-3, node-2, node-4。
 # then stop the node-5 that holds the meta node.
 # start node-3
 # then stop node-1。
 # start node-2, node-4, node-5, node-1。

*Fault description:*
 # The RegionServer nodes, including node-2, node-4, node-5, and node-1, are unable to come online.

Observing the RegionServer logs, the reportForDuty operation consistently times out. The log is as follows:

 
{code:java}
2023-09-21T08:05:30,251 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
2023-09-21T08:05:43,581 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
2023-09-21T08:05:59,591 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
2023-09-21T08:06:21,601 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
2023-09-21T08:06:55,611 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
2023-09-21T08:07:53,620 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
2023-09-21T08:09:39,631 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
2023-09-21T08:13:01,642 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874 {code}
2. The master thread is blocked.
 * All two RpcServer.priority.RWQ.Fifo.write.handler threads are blocked on RegionStateNode.lock

{code:java}
"RpcServer.priority.RWQ.Fifo.write.handler=1,queue=0,port=16000" #67 daemon prio=5 os_prio=0 tid=0x00007f6ae3caf800 nid=0xea405 waiting on condition [0x00007f6aa1dcd000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000004e3c8e6f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at org.apache.hadoop.hbase.master.assignment.RegionStateNode.lock(RegionStateNode.java:323)
    at org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1401)
    at org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:1363)
    at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:639)
    at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:17395)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:437)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
    at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
    at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
 

> All threads of HMaster PEWorker are blocked at updateRegionLocation and preventing the processing of TRSP for the Meta region.
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-28113
>                 URL: https://issues.apache.org/jira/browse/HBASE-28113
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 3.0.0-beta-1
>            Reporter: Haiping lv
>            Assignee: Haiping lv
>            Priority: Major
>
> HBase ** Cluster description: *1 master and 5 region servers*
> During the execution of itbll process, when ChaosMonkey performs RestartRandomRsAction, it triggers this issue.
>  ** The steps for the RestartRandomRsAction operation are as follows.{*}:{*}
>  # stop node-3, node-2, node-4。
>  # then stop the node-5 that holds the meta node.
>  # start node-3
>  # then stop node-1。
>  # start node-2, node-4, node-5, node-1。
> *Fault description:*
>  # The RegionServer nodes, including node-2, node-4, node-5, and node-1, are unable to come online.
> Observing the RegionServer logs, the reportForDuty operation consistently times out. The log is as follows:
>  
> {code:java}
> 2023-09-21T08:05:30,251 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
> 2023-09-21T08:05:43,581 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
> 2023-09-21T08:05:59,591 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
> 2023-09-21T08:06:21,601 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
> 2023-09-21T08:06:55,611 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
> 2023-09-21T08:07:53,620 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
> 2023-09-21T08:09:39,631 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874
> 2023-09-21T08:13:01,642 INFO  [regionserver/core-1-2:16020] regionserver.HRegionServer: reportForDuty to master=master-1-1,16000,1695254395517 with port=16020, startcode=1695254725874 {code}
> 2. The master thread is blocked.
>  * All two RpcServer.priority.RWQ.Fifo.write.handler threads are blocked on RegionStateNode.lock
> {code:java}
> "RpcServer.priority.RWQ.Fifo.write.handler=1,queue=0,port=16000" #67 daemon prio=5 os_prio=0 tid=0x00007f6ae3caf800 nid=0xea405 waiting on condition [0x00007f6aa1dcd000]
>    java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00000004e3c8e6f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>     at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>     at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>     at org.apache.hadoop.hbase.master.assignment.RegionStateNode.lock(RegionStateNode.java:323)
>     at org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1401)
>     at org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:1363)
>     at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:639)
>     at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:17395)
>     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:437)
>     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
>     at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
>     at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)