[jira] [Created] (HDFS-7212) Huge number of BLOCKED threads rendering DataNodes useless

["Istvan Szukacs (JIRA)" <ji...@apache.org>]

Istvan Szukacs created HDFS-7212:
------------------------------------

             Summary: Huge number of BLOCKED threads rendering DataNodes useless
                 Key: HDFS-7212
                 URL: https://issues.apache.org/jira/browse/HDFS-7212
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode
    Affects Versions: 2.4.0
         Environment: PROD
            Reporter: Istvan Szukacs


There are 3000 - 8000 threads in each datanode JVM, blocking the entire VM and rendering the service unusable, missing heartbeats and stopping data access. The threads look like this:

{code}
3415 (state = BLOCKED)

- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=834 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=867 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1197 (Interpreted frame)
- java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21, line=214 (Compiled frame)
- java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290 (Compiled frame)
- org.apache.hadoop.net.unix.DomainSocketWatcher.add(org.apache.hadoop.net.unix.DomainSocket, org.apache.hadoop.net.unix.DomainSocketWatcher$Handler) @bci=4, line=286 (Interpreted frame)
- org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.createNewMemorySegment(java.lang.String, org.apache.hadoop.net.unix.DomainSocket) @bci=169, line=283 (Interpreted frame)
- org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitShm(java.lang.String) @bci=212, line=413 (Interpreted frame)
- org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitShm(java.io.DataInputStream) @bci=13, line=172 (Interpreted frame)
- org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(org.apache.hadoop.hdfs.protocol.datatransfer.Op) @bci=149, line=92 (Compiled frame)
- org.apache.hadoop.hdfs.server.datanode.DataXceiver.run() @bci=510, line=232 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)
{code}

Has anybody seen this before?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)