You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Duo Zhang (Jira)" <ji...@apache.org> on 2024/04/07 00:20:00 UTC
[jira] [Commented] (HBASE-28494) "WAL system stuck?" due to deadlock at org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.complete
[ https://issues.apache.org/jira/browse/HBASE-28494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17834573#comment-17834573 ]
Duo Zhang commented on HBASE-28494:
-----------------------------------
This is not a 'dead' lock, it seems something wrong with the HDFS so the ring buffer is full, so there is no room to get the next slot in the ring buffer for append.
Could you please check the status of your HDFS cluster?
Thanks.
> "WAL system stuck?" due to deadlock at org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.complete
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-28494
> URL: https://issues.apache.org/jira/browse/HBASE-28494
> Project: HBase
> Issue Type: Bug
> Components: regionserver, wal
> Affects Versions: 2.5.5
> Environment: hbase-2.5.5
> hadoop-3.3.6
> kerberos authentication enabled.
> OS: debian 11
> Reporter: Athish Babu
> Priority: Major
> Attachments: RS_thread_dump.txt
>
>
> Currently we come across a issue in write handler threads of a regionserver during AsyncFSWAL append operation. We could see regionserver's write handler threads is going to WAITING State while acquiring lock for WAL append operation at MultiVersionConcurrencyControl.begin
>
> {code:java}
> "RpcServer.default.FPRWQ.Fifo.write.handler=7,queue=3,port=16020" #133 daemon prio=5 os_prio=0 tid=0x00007f9301fe7800 nid=0x329a02 runnable [0x00007f8a6489a000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
> at com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:263)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$stampSequenceIdAndPublishToRingBuffer$10(AbstractFSWAL.java:1202)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL$$Lambda$631/875615795.run(Unknown Source)
> at org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.begin(MultiVersionConcurrencyControl.java:144)
> - locked <0x00007f8afa4d1a80> (a java.util.LinkedList)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.stampSequenceIdAndPublishToRingBuffer(AbstractFSWAL.java:1201)
> at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.append(AsyncFSWAL.java:647)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$appendData$14(AbstractFSWAL.java:1255)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL$$Lambda$699/1762709833.call(Unknown Source)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.appendData(AbstractFSWAL.java:1255)
> at org.apache.hadoop.hbase.regionserver.HRegion.doWALAppend(HRegion.java:7800)
> at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4522)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4446)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4368)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1033)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:951)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:916)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2892)
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45008)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
>
> Other write handler threads are getting BLOCKED state while waiting for above lock to get released.
>
> {code:java}
> "RpcServer.default.FPRWQ.Fifo.write.handler=38,queue=2,port=16020" #164 daemon prio=5 os_prio=0 tid=0x00007f9303147800 nid=0x329a21 waiting for monitor entry [0x00007f8a61586000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.complete(MultiVersionConcurrencyControl.java:179)
> - waiting to lock <0x00007f8afa4d1a80> (a java.util.LinkedList)
> at org.apache.hadoop.hbase.regionserver.HRegion.doWALAppend(HRegion.java:7808)
> at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4522)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4446)
> at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4368)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1033)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:951)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:916)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2892)
> at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45008)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82){code}
> Due to above occurence , WAL system getting stucked and facing callqueueTooBig Exception
> {code:java}
> ERROR [MemStoreFlusher.0] o.a.h.h.r.MemStoreFlusher - Cache flush failed for region analytics:RawData,DIO|DISK_USAGE_STATS,1710357005367.55ec1db85903bbf9d2588b6d60b7b66f.
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=5472571, WAL system stuck?
> at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:171) ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:869) ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.doSync(AsyncFSWAL.java:668) ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$sync$1(AbstractFSWAL.java:590) ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-common-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.sync(AbstractFSWAL.java:590) ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.sync(AbstractFSWAL.java:580) ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2779) ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2721) ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2580) ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2554) ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2424) ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:603) [hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:572) [hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:65) [hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:344) [hbase-server-2.5.5.jar:2.5.5] {code}
> Issue get fixed in above mentioned regionserver after restarting the process. But after sometime again same issue occurs in other regionserver.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)