[jira] [Created] (HBASE-25507) Leak of ESTABLISHED sockets when encountered "java.io.IOException: Invalid HFile block magic"

["Xiaolin Ha (Jira)" <ji...@apache.org>]

Xiaolin Ha created HBASE-25507:
----------------------------------

             Summary: Leak of ESTABLISHED sockets when encountered "java.io.IOException: Invalid HFile block magic"
                 Key: HBASE-25507
                 URL: https://issues.apache.org/jira/browse/HBASE-25507
             Project: HBase
          Issue Type: Improvement
            Reporter: Xiaolin Ha
            Assignee: Xiaolin Ha
         Attachments: errorlogs.png, increasing-of-established-sockets-image.png, problem-region-move-logs.png

Recently, we found socket leaks on our production cluster. The leaked sockets are in ESTABLISHED state. We found this happened on RS who owned a particular region from our analysis of metrics monitor and logs. RS without this region works normally. 

On the RS who owns the particular region, we found Exceptions as follows,
{code:java}
java.io.IOException: java.io.IOException: Could not seek StoreFileScanner[org.apache.hadoop.hbase.ioo.HalfStoreFileReader$1@5388be2f, cur=null] to key org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFFamilyCell@25aa56fd        at org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)        at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)        at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:329)        at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:302)        at org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:8806)        at org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacctory.createScanner(StripeCompactor.java:82)        at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)        at org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..java:120)        at org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacctionRequest.execute(StripeCompactionPolicy.java:662)        at org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStooreEngine.java:114)        at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)        at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)        at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CommpactSplitThread.java:519)        at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplittThread.java:555)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)        at java.lang.Thread.run(Thread.java:745)Caused by: java.io.IOException: Could not seek StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreeFileReader$1@5388be2f, cur=null] to key org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@25aa56fd        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)        at org.apache.hadoop.hbase.regionserver.handler.ParallelSeekHandler.process(ParallelSeekHanddler.java:56)        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)        ... 3 moreCaused by: java.io.IOException: Invalid HFile block magic: \x00\x00\x00\x00\x00\x00\x00\x00        at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:159)        at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:171)        at org.apache.hadoop.hbase.io.hfile.HFileBlock.createFromBuff(HFileBlock.java:333)        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlockk.java:1753)        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:15552)        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:539)        at org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.readAndUpdateNewBlock(HFileScannerImpl..java:737)        at org.apache.hadoop.hbase.io.hfile.HFileScannerImpl.seekTo(HFileScannerImpl.java:726)        at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.seekTo(HalfStoreFileReader.java:161)        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.javaa:315)        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:211)        ... 5 more
{code}
The count of established sockets is always increasing, see picture,

!increasing-of-established-sockets-image.png!

 

!problem-region-move-logs.png!

!errorlogs.png!

 

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)