You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Haiping lv (Jira)" <ji...@apache.org> on 2023/09/01 01:37:00 UTC

[jira] [Updated] (HBASE-28058) HMaster snapshot file clean thread and the snapshot request handler thread encountered a deadlock

     [ https://issues.apache.org/jira/browse/HBASE-28058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Haiping lv updated HBASE-28058:
-------------------------------
    Description: 
HMaster encountered a deadlock when runing ITBLL test.

The handler thread of HMaster holds the synchronized lock of SnapshotManager and competes for the readLock of takingSnapshotLock.
{code:java}
Thread 57 (RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16000):
  State: WAITING
  Blocked count: 14
  Waited count: 206
  Waiting on java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@a7a58d4
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
    java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
    org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:675)
    org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.java:1765)
    org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
    org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:437)
    org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
    org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
    org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}

The Master Snapshot cleaning thread holds the writeLock of takingSnapshotLock to compete for the synchronized lock of SnapshotManager.
{code:java}
Thread 5405 (hfile_cleaner-dir-scan-pool-2):
State: BLOCKED
Blocked count: 50535
Waited count: 92200
Blocked on org.apache.hadoop.hbase.master.snapshot.SnapshotManager@21a16019
Blocked by 57 (RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16000)
Stack:
  org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isTakingAnySnapshot(SnapshotManager.java:655)
  org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.getUnreferencedFiles(SnapshotFileCache.java:197)
  org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner.getDeletableFiles(SnapshotHFileCleaner.java:78)
  org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:375)
  org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$1(CleanerChore.java:483)
  org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$788/1121007598.act(Unknown Source)
  org.apache.hadoop.hbase.master.cleaner.CleanerChore.deleteAction(CleanerChore.java:565)
  org.apache.hadoop.hbase.master.cleaner.CleanerChore.traverseAndDelete(CleanerChore.java:483)
  org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$2(CleanerChore.java:495)
  org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$794/1250580839.run(Unknown Source)
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  java.lang.Thread.run(Thread.java:750){{}} {code}

  was:
HMaster encountered a deadlock when runing ITBLL test.

The handler thread of HMaster holds the synchronized lock of SnapshotManager and competes for the readLock of takingSnapshotLock.
Thread 57 (RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16000):
  State: WAITING
  Blocked count: 14
  Waited count: 206
  Waiting on java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@a7a58d4
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
    java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
    org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:675)
    org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.java:1765)
    org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
    org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:437)
    org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
    org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
    org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
The Master Snapshot cleaning thread holds the writeLock of takingSnapshotLock to compete for the synchronized lock of SnapshotManager.
Thread 5405 (hfile_cleaner-dir-scan-pool-2):
  State: BLOCKED
  Blocked count: 50535
  Waited count: 92200
  Blocked on org.apache.hadoop.hbase.master.snapshot.SnapshotManager@21a16019
  Blocked by 57 (RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16000)
  Stack:
    org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isTakingAnySnapshot(SnapshotManager.java:655)
    org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.getUnreferencedFiles(SnapshotFileCache.java:197)
    org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner.getDeletableFiles(SnapshotHFileCleaner.java:78)
    org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:375)
    org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$1(CleanerChore.java:483)
    org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$788/1121007598.act(Unknown Source)
    org.apache.hadoop.hbase.master.cleaner.CleanerChore.deleteAction(CleanerChore.java:565)
    org.apache.hadoop.hbase.master.cleaner.CleanerChore.traverseAndDelete(CleanerChore.java:483)
    org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$2(CleanerChore.java:495)
    org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$794/1250580839.run(Unknown Source)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    java.lang.Thread.run(Thread.java:750){{}}


> HMaster snapshot file clean thread and the snapshot request handler thread encountered a deadlock
> -------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-28058
>                 URL: https://issues.apache.org/jira/browse/HBASE-28058
>             Project: HBase
>          Issue Type: Bug
>          Components: snapshots
>    Affects Versions: 3.0.0-beta-1
>            Reporter: Haiping lv
>            Assignee: Haiping lv
>            Priority: Major
>
> HMaster encountered a deadlock when runing ITBLL test.
> The handler thread of HMaster holds the synchronized lock of SnapshotManager and competes for the readLock of takingSnapshotLock.
> {code:java}
> Thread 57 (RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16000):
>   State: WAITING
>   Blocked count: 14
>   Waited count: 206
>   Waiting on java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@a7a58d4
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
>     java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
>     org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:675)
>     org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.java:1765)
>     org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
>     org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:437)
>     org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
>     org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
>     org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
> The Master Snapshot cleaning thread holds the writeLock of takingSnapshotLock to compete for the synchronized lock of SnapshotManager.
> {code:java}
> Thread 5405 (hfile_cleaner-dir-scan-pool-2):
> State: BLOCKED
> Blocked count: 50535
> Waited count: 92200
> Blocked on org.apache.hadoop.hbase.master.snapshot.SnapshotManager@21a16019
> Blocked by 57 (RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16000)
> Stack:
>   org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isTakingAnySnapshot(SnapshotManager.java:655)
>   org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.getUnreferencedFiles(SnapshotFileCache.java:197)
>   org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner.getDeletableFiles(SnapshotHFileCleaner.java:78)
>   org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:375)
>   org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$1(CleanerChore.java:483)
>   org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$788/1121007598.act(Unknown Source)
>   org.apache.hadoop.hbase.master.cleaner.CleanerChore.deleteAction(CleanerChore.java:565)
>   org.apache.hadoop.hbase.master.cleaner.CleanerChore.traverseAndDelete(CleanerChore.java:483)
>   org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$2(CleanerChore.java:495)
>   org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$794/1250580839.run(Unknown Source)
>   java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   java.lang.Thread.run(Thread.java:750){{}} {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)