[jira] [Created] (HBASE-28015) rpc handler can get stuck on LruBlockCache

["ruanhui (Jira)" <ji...@apache.org>]

ruanhui created HBASE-28015:
-------------------------------

             Summary: rpc handler can get stuck on LruBlockCache
                 Key: HBASE-28015
                 URL: https://issues.apache.org/jira/browse/HBASE-28015
             Project: HBase
          Issue Type: Bug
          Components: BlockCache
    Affects Versions: 3.0.0-alpha-4
            Reporter: ruanhui
             Fix For: 4.0.0-alpha-1


We found lots of read handlers got stuck on LruBlockCache#getBlock, this may be caused by a bug in jdk8 ConcurrentHashMap. To make common fast, I think we'd better get and check it before call ConcurrentHashMap#computeIfAbsent.

 

 

"RpcServer.priority.RWQ.Fifo.scan.handler=190,queue=57,port=60020" #1807 daemon prio=5 os_prio=0 cpu=9703.28ms elapsed=88160.93s tid=0x00007f38d338a800 nid=0x8f4 waiting for monitor entry [0x00007f0af4baa000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1760)
        - waiting to lock <0x00007f2fc6495fe0> (a java.util.concurrent.ConcurrentHashMap$Node)
        at org.apache.hadoop.hbase.io.hfile.LruBlockCache.getBlock(LruBlockCache.java:538)
        at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:88)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.getCachedBlock(HFileReaderImpl.java:1124)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1300)
        at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:331)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:679)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:631)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:315)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:216)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.backwardSeek(StoreFileScanner.java:561)
        at org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.backwardSeek(ReversedKeyValueHeap.java:117)
        at org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.backwardSeek(ReversedStoreScanner.java:134)
        at org.apache.hadoop.hbase.regionserver.ReversedStoreScanner.seekAsDirection(ReversedStoreScanner.java:94)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.seekOrSkipToNextColumn(StoreScanner.java:821)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:727)
        at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:155)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7515)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7683)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7447)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3403)
        - locked <0x00007f2ff1fc8f40> (a org.apache.hadoop.hbase.regionserver.ReversedRegionScannerImpl)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3662)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45253)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:447)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:136)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

   Locked ownable synchronizers:
        - None



--
This message was sent by Atlassian Jira
(v8.20.10#820010)