You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Sedighe Tabatabaei <ta...@gmail.com> on 2012/12/22 11:16:20 UTC
Child processes on datanodes/task trackers
Hello
I wanted to know why I see running Child processes on my datanodes
even though there is no job running at that time.
when i use jstack <pid> command,result is:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00007f2a74002000 nid=0x1f3
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"SIGTERM handler" daemon prio=10 tid=0x00007f2a74001000 nid=0x2fae
waiting for monitor entry [0x00007f2a79891000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Shutdown.exit(Shutdown.java:168)
- waiting to lock <0x00007f2a9ffa68d0> (a java.lang.Class for
java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:35)
at sun.misc.Signal$1.run(Signal.java:195)
at java.lang.Thread.run(Thread.java:662)
"Thread-16" prio=10 tid=0x00007f2ae83d1000 nid=0x2f85 runnable
[0x00007f2a9cb5a000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
- locked <0x00007f2ad36e4d50> (a java.util.ConcurrentModificationException)
at java.lang.Throwable.<init>(Throwable.java:181)
at java.lang.Exception.<init>(Exception.java:29)
at java.lang.RuntimeException.<init>(RuntimeException.java:32)
at java.util.ConcurrentModificationException.<init>(ConcurrentModificationException.java:57)
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$KeyIterator.next(HashMap.java:828)
at com.sun.media.jai.codec.TempFileCleanupThread.run(FileCacheSeekableStream.java:315)
"DestroyJavaVM" prio=10 tid=0x00007f2ae8009800 nid=0x1113 in
Object.wait() [0x00007f2aedb81000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f2aa5248528> (a
com.sun.media.jai.codec.TempFileCleanupThread)
at java.lang.Thread.join(Thread.java:1186)
- locked <0x00007f2aa5248528> (a com.sun.media.jai.codec.TempFileCleanupThread)
at java.lang.Thread.join(Thread.java:1239)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
at java.lang.Shutdown.runHooks(Shutdown.java:79)
at java.lang.Shutdown.sequence(Shutdown.java:123)
at java.lang.Shutdown.shutdown(Shutdown.java:190)
- locked <0x00007f2a9ffa68d0> (a java.lang.Class for java.lang.Shutdown)
"Java2D Disposer" daemon prio=10 tid=0x00007f2ae8415000 nid=0x1775 in
Object.wait() [0x00007f2a7a16d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at sun.java2d.Disposer.run(Disposer.java:127)
at java.lang.Thread.run(Thread.java:662)
"Thread for syncLogs" daemon prio=10 tid=0x00007f2ae83bf000 nid=0x1208
waiting on condition [0x00007f2a9f2fa000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
"Low Memory Detector" daemon prio=10 tid=0x00007f2ae8093000 nid=0x112f
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x00007f2ae8090800 nid=0x112d
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x00007f2ae808d800 nid=0x112a
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f2ae808b800 nid=0x1129
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007f2ae806d000 nid=0x1126 in
Object.wait() [0x00007f2a9fafd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00007f2ae806b000 nid=0x1124
in Object.wait() [0x00007f2a9fbfe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x00007f2ae8066800 nid=0x1122 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f2ae801c800
nid=0x1114 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f2ae801e800
nid=0x1115 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f2ae8020000
nid=0x111c runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f2ae8022000
nid=0x111f runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f2ae809d000 nid=0x1131
waiting on condition
JNI global references: 1774
Thanks, Tabatabaei
Re: Child processes on datanodes/task trackers
Posted by Harsh J <ha...@cloudera.com>.
This is an interesting stacktrace. The JVM had called exit but there
are threads stuck. Unsure if your JVM version (6u22) has anything to
do with it.
Are you using the JVM reuse feature?
On Sat, Dec 22, 2012 at 3:46 PM, Sedighe Tabatabaei
<ta...@gmail.com> wrote:
> Hello
> I wanted to know why I see running Child processes on my datanodes
> even though there is no job running at that time.
> when i use jstack <pid> command,result is:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x00007f2a74002000 nid=0x1f3
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "SIGTERM handler" daemon prio=10 tid=0x00007f2a74001000 nid=0x2fae
> waiting for monitor entry [0x00007f2a79891000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Shutdown.exit(Shutdown.java:168)
> - waiting to lock <0x00007f2a9ffa68d0> (a java.lang.Class for
> java.lang.Shutdown)
> at java.lang.Terminator$1.handle(Terminator.java:35)
> at sun.misc.Signal$1.run(Signal.java:195)
> at java.lang.Thread.run(Thread.java:662)
>
> "Thread-16" prio=10 tid=0x00007f2ae83d1000 nid=0x2f85 runnable
> [0x00007f2a9cb5a000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.Throwable.fillInStackTrace(Native Method)
> - locked <0x00007f2ad36e4d50> (a java.util.ConcurrentModificationException)
> at java.lang.Throwable.<init>(Throwable.java:181)
> at java.lang.Exception.<init>(Exception.java:29)
> at java.lang.RuntimeException.<init>(RuntimeException.java:32)
> at java.util.ConcurrentModificationException.<init>(ConcurrentModificationException.java:57)
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at com.sun.media.jai.codec.TempFileCleanupThread.run(FileCacheSeekableStream.java:315)
>
> "DestroyJavaVM" prio=10 tid=0x00007f2ae8009800 nid=0x1113 in
> Object.wait() [0x00007f2aedb81000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa5248528> (a
> com.sun.media.jai.codec.TempFileCleanupThread)
> at java.lang.Thread.join(Thread.java:1186)
> - locked <0x00007f2aa5248528> (a com.sun.media.jai.codec.TempFileCleanupThread)
> at java.lang.Thread.join(Thread.java:1239)
> at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
> at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
> at java.lang.Shutdown.runHooks(Shutdown.java:79)
> at java.lang.Shutdown.sequence(Shutdown.java:123)
> at java.lang.Shutdown.shutdown(Shutdown.java:190)
> - locked <0x00007f2a9ffa68d0> (a java.lang.Class for java.lang.Shutdown)
>
> "Java2D Disposer" daemon prio=10 tid=0x00007f2ae8415000 nid=0x1775 in
> Object.wait() [0x00007f2a7a16d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at sun.java2d.Disposer.run(Disposer.java:127)
> at java.lang.Thread.run(Thread.java:662)
>
> "Thread for syncLogs" daemon prio=10 tid=0x00007f2ae83bf000 nid=0x1208
> waiting on condition [0x00007f2a9f2fa000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
>
> "Low Memory Detector" daemon prio=10 tid=0x00007f2ae8093000 nid=0x112f
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x00007f2ae8090800 nid=0x112d
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x00007f2ae808d800 nid=0x112a
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007f2ae808b800 nid=0x1129
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007f2ae806d000 nid=0x1126 in
> Object.wait() [0x00007f2a9fafd000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x00007f2ae806b000 nid=0x1124
> in Object.wait() [0x00007f2a9fbfe000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x00007f2ae8066800 nid=0x1122 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f2ae801c800
> nid=0x1114 runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f2ae801e800
> nid=0x1115 runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f2ae8020000
> nid=0x111c runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f2ae8022000
> nid=0x111f runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007f2ae809d000 nid=0x1131
> waiting on condition
>
> JNI global references: 1774
>
>
> Thanks, Tabatabaei
--
Harsh J
Re: Child processes on datanodes/task trackers
Posted by Harsh J <ha...@cloudera.com>.
This is an interesting stacktrace. The JVM had called exit but there
are threads stuck. Unsure if your JVM version (6u22) has anything to
do with it.
Are you using the JVM reuse feature?
On Sat, Dec 22, 2012 at 3:46 PM, Sedighe Tabatabaei
<ta...@gmail.com> wrote:
> Hello
> I wanted to know why I see running Child processes on my datanodes
> even though there is no job running at that time.
> when i use jstack <pid> command,result is:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x00007f2a74002000 nid=0x1f3
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "SIGTERM handler" daemon prio=10 tid=0x00007f2a74001000 nid=0x2fae
> waiting for monitor entry [0x00007f2a79891000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Shutdown.exit(Shutdown.java:168)
> - waiting to lock <0x00007f2a9ffa68d0> (a java.lang.Class for
> java.lang.Shutdown)
> at java.lang.Terminator$1.handle(Terminator.java:35)
> at sun.misc.Signal$1.run(Signal.java:195)
> at java.lang.Thread.run(Thread.java:662)
>
> "Thread-16" prio=10 tid=0x00007f2ae83d1000 nid=0x2f85 runnable
> [0x00007f2a9cb5a000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.Throwable.fillInStackTrace(Native Method)
> - locked <0x00007f2ad36e4d50> (a java.util.ConcurrentModificationException)
> at java.lang.Throwable.<init>(Throwable.java:181)
> at java.lang.Exception.<init>(Exception.java:29)
> at java.lang.RuntimeException.<init>(RuntimeException.java:32)
> at java.util.ConcurrentModificationException.<init>(ConcurrentModificationException.java:57)
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at com.sun.media.jai.codec.TempFileCleanupThread.run(FileCacheSeekableStream.java:315)
>
> "DestroyJavaVM" prio=10 tid=0x00007f2ae8009800 nid=0x1113 in
> Object.wait() [0x00007f2aedb81000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa5248528> (a
> com.sun.media.jai.codec.TempFileCleanupThread)
> at java.lang.Thread.join(Thread.java:1186)
> - locked <0x00007f2aa5248528> (a com.sun.media.jai.codec.TempFileCleanupThread)
> at java.lang.Thread.join(Thread.java:1239)
> at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
> at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
> at java.lang.Shutdown.runHooks(Shutdown.java:79)
> at java.lang.Shutdown.sequence(Shutdown.java:123)
> at java.lang.Shutdown.shutdown(Shutdown.java:190)
> - locked <0x00007f2a9ffa68d0> (a java.lang.Class for java.lang.Shutdown)
>
> "Java2D Disposer" daemon prio=10 tid=0x00007f2ae8415000 nid=0x1775 in
> Object.wait() [0x00007f2a7a16d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at sun.java2d.Disposer.run(Disposer.java:127)
> at java.lang.Thread.run(Thread.java:662)
>
> "Thread for syncLogs" daemon prio=10 tid=0x00007f2ae83bf000 nid=0x1208
> waiting on condition [0x00007f2a9f2fa000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
>
> "Low Memory Detector" daemon prio=10 tid=0x00007f2ae8093000 nid=0x112f
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x00007f2ae8090800 nid=0x112d
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x00007f2ae808d800 nid=0x112a
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007f2ae808b800 nid=0x1129
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007f2ae806d000 nid=0x1126 in
> Object.wait() [0x00007f2a9fafd000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x00007f2ae806b000 nid=0x1124
> in Object.wait() [0x00007f2a9fbfe000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x00007f2ae8066800 nid=0x1122 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f2ae801c800
> nid=0x1114 runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f2ae801e800
> nid=0x1115 runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f2ae8020000
> nid=0x111c runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f2ae8022000
> nid=0x111f runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007f2ae809d000 nid=0x1131
> waiting on condition
>
> JNI global references: 1774
>
>
> Thanks, Tabatabaei
--
Harsh J
Re: Child processes on datanodes/task trackers
Posted by Harsh J <ha...@cloudera.com>.
This is an interesting stacktrace. The JVM had called exit but there
are threads stuck. Unsure if your JVM version (6u22) has anything to
do with it.
Are you using the JVM reuse feature?
On Sat, Dec 22, 2012 at 3:46 PM, Sedighe Tabatabaei
<ta...@gmail.com> wrote:
> Hello
> I wanted to know why I see running Child processes on my datanodes
> even though there is no job running at that time.
> when i use jstack <pid> command,result is:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x00007f2a74002000 nid=0x1f3
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "SIGTERM handler" daemon prio=10 tid=0x00007f2a74001000 nid=0x2fae
> waiting for monitor entry [0x00007f2a79891000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Shutdown.exit(Shutdown.java:168)
> - waiting to lock <0x00007f2a9ffa68d0> (a java.lang.Class for
> java.lang.Shutdown)
> at java.lang.Terminator$1.handle(Terminator.java:35)
> at sun.misc.Signal$1.run(Signal.java:195)
> at java.lang.Thread.run(Thread.java:662)
>
> "Thread-16" prio=10 tid=0x00007f2ae83d1000 nid=0x2f85 runnable
> [0x00007f2a9cb5a000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.Throwable.fillInStackTrace(Native Method)
> - locked <0x00007f2ad36e4d50> (a java.util.ConcurrentModificationException)
> at java.lang.Throwable.<init>(Throwable.java:181)
> at java.lang.Exception.<init>(Exception.java:29)
> at java.lang.RuntimeException.<init>(RuntimeException.java:32)
> at java.util.ConcurrentModificationException.<init>(ConcurrentModificationException.java:57)
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at com.sun.media.jai.codec.TempFileCleanupThread.run(FileCacheSeekableStream.java:315)
>
> "DestroyJavaVM" prio=10 tid=0x00007f2ae8009800 nid=0x1113 in
> Object.wait() [0x00007f2aedb81000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa5248528> (a
> com.sun.media.jai.codec.TempFileCleanupThread)
> at java.lang.Thread.join(Thread.java:1186)
> - locked <0x00007f2aa5248528> (a com.sun.media.jai.codec.TempFileCleanupThread)
> at java.lang.Thread.join(Thread.java:1239)
> at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
> at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
> at java.lang.Shutdown.runHooks(Shutdown.java:79)
> at java.lang.Shutdown.sequence(Shutdown.java:123)
> at java.lang.Shutdown.shutdown(Shutdown.java:190)
> - locked <0x00007f2a9ffa68d0> (a java.lang.Class for java.lang.Shutdown)
>
> "Java2D Disposer" daemon prio=10 tid=0x00007f2ae8415000 nid=0x1775 in
> Object.wait() [0x00007f2a7a16d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at sun.java2d.Disposer.run(Disposer.java:127)
> at java.lang.Thread.run(Thread.java:662)
>
> "Thread for syncLogs" daemon prio=10 tid=0x00007f2ae83bf000 nid=0x1208
> waiting on condition [0x00007f2a9f2fa000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
>
> "Low Memory Detector" daemon prio=10 tid=0x00007f2ae8093000 nid=0x112f
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x00007f2ae8090800 nid=0x112d
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x00007f2ae808d800 nid=0x112a
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007f2ae808b800 nid=0x1129
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007f2ae806d000 nid=0x1126 in
> Object.wait() [0x00007f2a9fafd000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x00007f2ae806b000 nid=0x1124
> in Object.wait() [0x00007f2a9fbfe000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x00007f2ae8066800 nid=0x1122 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f2ae801c800
> nid=0x1114 runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f2ae801e800
> nid=0x1115 runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f2ae8020000
> nid=0x111c runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f2ae8022000
> nid=0x111f runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007f2ae809d000 nid=0x1131
> waiting on condition
>
> JNI global references: 1774
>
>
> Thanks, Tabatabaei
--
Harsh J
Re: Child processes on datanodes/task trackers
Posted by Harsh J <ha...@cloudera.com>.
This is an interesting stacktrace. The JVM had called exit but there
are threads stuck. Unsure if your JVM version (6u22) has anything to
do with it.
Are you using the JVM reuse feature?
On Sat, Dec 22, 2012 at 3:46 PM, Sedighe Tabatabaei
<ta...@gmail.com> wrote:
> Hello
> I wanted to know why I see running Child processes on my datanodes
> even though there is no job running at that time.
> when i use jstack <pid> command,result is:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x00007f2a74002000 nid=0x1f3
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "SIGTERM handler" daemon prio=10 tid=0x00007f2a74001000 nid=0x2fae
> waiting for monitor entry [0x00007f2a79891000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Shutdown.exit(Shutdown.java:168)
> - waiting to lock <0x00007f2a9ffa68d0> (a java.lang.Class for
> java.lang.Shutdown)
> at java.lang.Terminator$1.handle(Terminator.java:35)
> at sun.misc.Signal$1.run(Signal.java:195)
> at java.lang.Thread.run(Thread.java:662)
>
> "Thread-16" prio=10 tid=0x00007f2ae83d1000 nid=0x2f85 runnable
> [0x00007f2a9cb5a000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.Throwable.fillInStackTrace(Native Method)
> - locked <0x00007f2ad36e4d50> (a java.util.ConcurrentModificationException)
> at java.lang.Throwable.<init>(Throwable.java:181)
> at java.lang.Exception.<init>(Exception.java:29)
> at java.lang.RuntimeException.<init>(RuntimeException.java:32)
> at java.util.ConcurrentModificationException.<init>(ConcurrentModificationException.java:57)
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at com.sun.media.jai.codec.TempFileCleanupThread.run(FileCacheSeekableStream.java:315)
>
> "DestroyJavaVM" prio=10 tid=0x00007f2ae8009800 nid=0x1113 in
> Object.wait() [0x00007f2aedb81000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa5248528> (a
> com.sun.media.jai.codec.TempFileCleanupThread)
> at java.lang.Thread.join(Thread.java:1186)
> - locked <0x00007f2aa5248528> (a com.sun.media.jai.codec.TempFileCleanupThread)
> at java.lang.Thread.join(Thread.java:1239)
> at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
> at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
> at java.lang.Shutdown.runHooks(Shutdown.java:79)
> at java.lang.Shutdown.sequence(Shutdown.java:123)
> at java.lang.Shutdown.shutdown(Shutdown.java:190)
> - locked <0x00007f2a9ffa68d0> (a java.lang.Class for java.lang.Shutdown)
>
> "Java2D Disposer" daemon prio=10 tid=0x00007f2ae8415000 nid=0x1775 in
> Object.wait() [0x00007f2a7a16d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at sun.java2d.Disposer.run(Disposer.java:127)
> at java.lang.Thread.run(Thread.java:662)
>
> "Thread for syncLogs" daemon prio=10 tid=0x00007f2ae83bf000 nid=0x1208
> waiting on condition [0x00007f2a9f2fa000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
>
> "Low Memory Detector" daemon prio=10 tid=0x00007f2ae8093000 nid=0x112f
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x00007f2ae8090800 nid=0x112d
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x00007f2ae808d800 nid=0x112a
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007f2ae808b800 nid=0x1129
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007f2ae806d000 nid=0x1126 in
> Object.wait() [0x00007f2a9fafd000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x00007f2aa50c8bf0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x00007f2ae806b000 nid=0x1124
> in Object.wait() [0x00007f2a9fbfe000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x00007f2aa50c8ba8> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x00007f2ae8066800 nid=0x1122 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f2ae801c800
> nid=0x1114 runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f2ae801e800
> nid=0x1115 runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f2ae8020000
> nid=0x111c runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f2ae8022000
> nid=0x111f runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007f2ae809d000 nid=0x1131
> waiting on condition
>
> JNI global references: 1774
>
>
> Thanks, Tabatabaei
--
Harsh J