[jira] [Created] (HDFS-1868) HDFS-client may block shutdown of JVM

["Dennis Cheung (JIRA)" <ji...@apache.org>]

HDFS-client may block shutdown of JVM
-------------------------------------

                 Key: HDFS-1868
                 URL: https://issues.apache.org/jira/browse/HDFS-1868
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: hdfs client
    Affects Versions: 0.20.2
         Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
java version "1.6.0_23"
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
            Reporter: Dennis Cheung
            Priority: Minor


When I send a "kill -15" to shutdown my process, sometime it will freeze after the safe-shutdown process complete.

I ran jstack and found the following thread blocking.


"SIGTERM handler" daemon prio=10 tid=0x00002aaab12c4000 nid=0x6274 waiting for monitor entry [0x000000004e056000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Thread.join(Thread.java:1177)
        - waiting to lock <0x000000078376e8b8> (a org.apache.hadoop.fs.FileSystem$ClientFinalizer)
        at java.lang.Thread.join(Thread.java:1239)
        at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
        at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
        at java.lang.Shutdown.runHooks(Shutdown.java:79)
        at java.lang.Shutdown.sequence(Shutdown.java:123)
        at java.lang.Shutdown.exit(Shutdown.java:168)
        - locked <0x000000077e8127e8> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Terminator$1.handle(Terminator.java:35)
        at sun.misc.Signal$1.run(Signal.java:195)
        at java.lang.Thread.run(Thread.java:662)


Beside the thread "SIGTERM handler", there are some few threads of my application are WAITING or BLOCKED with this pattern
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:3002)
        - locked <0x00000007b24fee10> (a java.util.LinkedList)
        - locked <0x00000007b2512070> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
        at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
        at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
        at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
        - locked <0x00000007b2512070> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        - locked <0x00000007b25126e0> (a org.apache.hadoop.fs.FSDataOutputStream)


However, there are no other HDFS threads (e.g. DataStreamer) running.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira