You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Yang Jiandan (JIRA)" <ji...@apache.org> on 2014/09/05 04:28:24 UTC
[jira] [Commented] (HDFS-6999) PacketReceiver#readChannelFully is
in an infinite loop
[ https://issues.apache.org/jira/browse/HDFS-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122300#comment-14122300 ]
Yang Jiandan commented on HDFS-6999:
------------------------------------
the stack is :
"regionserver60020-largeCompactions-1409055324582" daemon prio=10 tid=0x0000000001080800 nid=0x2c7c runnable [0x00000000601cb000]
java.lang.Thread.State: RUNNABLE
at org.apache.hadoop.net.unix.DomainSocket.readByteBufferDirect0(Native Method)
at org.apache.hadoop.net.unix.DomainSocket.access$400(DomainSocket.java:45)
at org.apache.hadoop.net.unix.DomainSocket$DomainChannel.read(DomainSocket.java:628)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.readChannelFully(PacketReceiver.java:258)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:209)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:102)
at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:173)
at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:138)
- locked <0x000000047c41f7e0> (a org.apache.hadoop.hdfs.RemoteBlockReader2)
at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682)
at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738)
- locked <0x00000004aaceca60> (a org.apache.hadoop.hdfs.DFSInputStream)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836)
- locked <0x00000004aaceca60> (a org.apache.hadoop.hdfs.DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:563)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1215)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1430)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1312)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:392)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:643)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:757)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:136)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:507)
at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:217)
at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:76)
at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:109)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1086)
at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1480)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:475)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- <0x000000049e162b60> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x00000005974a84f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x000000065e45cf58> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> PacketReceiver#readChannelFully is in an infinite loop
> ------------------------------------------------------
>
> Key: HDFS-6999
> URL: https://issues.apache.org/jira/browse/HDFS-6999
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, hdfs-client
> Affects Versions: 2.4.1
> Reporter: Yang Jiandan
> Priority: Critical
>
> In our cluster, we found hbase handler may be never return when it reads hdfs file using RemoteBlockReader2, and the hander thread occupys 100% cup. wo found this is because PacketReceiver#readChannelFully is in an infinite loop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)