You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Zheng Lv <lv...@gmail.com> on 2010/02/11 04:59:07 UTC
Many child processes dont exit
Hello Everyone,
We often find many child processes in datanodes, which have already
finished for long time. And following are the jstack log:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode):
"DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826 runnable
[0x000000004100a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1)
- locked <0x00002aaab9b5f710> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaab9b5f680> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
at
org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825 waiting on
condition [0x0000000041604000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab9b61620> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Attach Listener" daemon prio=10 tid=0x0000000043a22800 nid=0x2608 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"pool-3-thread-1" prio=10 tid=0x00000000438ad000 nid=0x243f waiting on
condition [0x0000000042575000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab9ae8770> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread for syncLogs" daemon prio=10 tid=0x00002aaac4446800 nid=0x2437
waiting on condition [0x0000000040f09000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
"Low Memory Detector" daemon prio=10 tid=0x0000000043771800 nid=0x242d
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x000000004376e800 nid=0x242c waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x000000004376a800 nid=0x242b waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x0000000043768800 nid=0x242a
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x0000000043746000 nid=0x2429 in
Object.wait() [0x0000000042171000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab9a8afe0> (a
java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00002aaab9a8afe0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x0000000043744000 nid=0x2428 in
Object.wait() [0x0000000042070000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab96b13a0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaab96b13a0> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x000000004373d800 nid=0x2427 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000436eb800 nid=0x2423
runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000436ed800 nid=0x2424
runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00000000436ef800 nid=0x2425
runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00000000436f1000 nid=0x2426
runnable
"VM Periodic Task Thread" prio=10 tid=0x0000000043774000 nid=0x242e waiting
on condition
JNI global references: 871
any suggestion?
best regards,
LvZheng.
Re: Many child processes dont exit
Posted by Zheng Lv <lv...@gmail.com>.
Thank you Jason, your reply is helpful.
2010/2/23 Jason Venner <ja...@gmail.com>
> Someone is using a threadpool that does not have daemon priority
> threads, and that is not shutdown before the main method returns.
> The non daemon threads prevent the jvm from exiting.
> We had this problem for a while and modified the Child.main to exit,
> rather than trying to work out and fix the third party library that
> ran the thread pool. This thecnique does of prevent jvm reuse.
>
> On Sat, Feb 20, 2010 at 6:49 AM, Ted Yu <yu...@gmail.com> wrote:
> > Do you have System.exit() as the last line in your main() ?
> > Job job = createSubmittableJob(conf, otherArgs);
> > System.exit(job.waitForCompletion(true)? 0 : 1);
> >
> >
> > On Sat, Feb 20, 2010 at 12:32 AM, Zheng Lv <lvzheng19800619@gmail.com
> >wrote:
> >
> >> Hello Ted,
> >> Yes. Every hour a job will be created and started, and when it
> finished,
> >> it will maintain. The logs looks like normal, do you know what can lead
> to
> >> this happen?Thank you.
> >> LvZheng
> >> 2010/2/20 Ted Yu <yu...@gmail.com>
> >>
> >> > Did the number of child processes increase over time ?
> >> >
> >> > On Friday, February 19, 2010, Zheng Lv <lv...@gmail.com>
> >> wrote:
> >> > > Hello Edson,
> >> > > Thank you for your reply. I don't want to kill them, I want to
> know
> >> why
> >> > > these child processes don't exit, and to know how to make them exit
> >> > > successfully when they finish. Any ideas? Thank you.
> >> > > LvZheng
> >> > >
> >> > > 2010/2/18 Edson Ramiro <er...@gmail.com>
> >> > >
> >> > >> Do you want to kill them ?
> >> > >>
> >> > >> if yes, you can use
> >> > >>
> >> > >> ./bin/slaves.sh pkill java
> >> > >>
> >> > >> but it will kill the datanode and tasktracker processes
> >> > >> in all slaves and you'll need to start these processes again.
> >> > >>
> >> > >> Edson Ramiro
> >> > >>
> >> > >>
> >> > >> On 14 February 2010 22:09, Zheng Lv <lv...@gmail.com>
> >> wrote:
> >> > >>
> >> > >> > any idea?
> >> > >> >
> >> > >> > 2010/2/11 Zheng Lv <lv...@gmail.com>
> >> > >> >
> >> > >> > > Hello Everyone,
> >> > >> > > We often find many child processes in datanodes, which have
> >> > already
> >> > >> > > finished for long time. And following are the jstack log:
> >> > >> > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01
> >> mixed
> >> > >> > mode):
> >> > >> > > "DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422
> waiting
> >> on
> >> > >> > > condition [0x0000000000000000]
> >> > >> > > java.lang.Thread.State: RUNNABLE
> >> > >> > > "NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826
> >> runnable
> >> > >> > > [0x000000004100a000]
> >> > >> > > java.lang.Thread.State: RUNNABLE
> >> > >> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native
> Method)
> >> > >> > > at
> >> > >> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> >> > >> > > at
> >> > >> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> >> > >> > > at
> >> > >> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> >> > >> > > - locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1)
> >> > >> > > - locked <0x00002aaab9b5f710> (a
> >> > >> > > java.util.Collections$UnmodifiableSet)
> >> > >> > > - locked <0x00002aaab9b5f680> (a
> >> > sun.nio.ch.EPollSelectorImpl)
> >> > >> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> >> > >> > > at
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
> >> > >> > > at
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672)
> >> > >> > > at
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
> >> > >> > > at
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> > >> > > at
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> > >> > > at java.lang.Thread.run(Thread.java:619)
> >> > >> > > "pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825
> >> waiting
> >> > on
> >> > >> > > condition [0x0000000041604000]
> >> > >> > > java.lang.Thread.State: WAITING (parking)
> >> > >> > > at sun.misc.Unsafe.park(Native Method)
> >> > >> > > - parking to wait for <0x00002aaab9b61620> (a
> >> > >> > >
> >> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >> > >> > > at
> >> > >> > >
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >> > >> > > at
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> >> > >> > > at
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> >> > >> > > at
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> >> > >> > > at
> >> > >> > >
> >> > >> >
> >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa
> >> >
> >>
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>
Re: Many child processes dont exit
Posted by Jason Venner <ja...@gmail.com>.
Someone is using a threadpool that does not have daemon priority
threads, and that is not shutdown before the main method returns.
The non daemon threads prevent the jvm from exiting.
We had this problem for a while and modified the Child.main to exit,
rather than trying to work out and fix the third party library that
ran the thread pool. This thecnique does of prevent jvm reuse.
On Sat, Feb 20, 2010 at 6:49 AM, Ted Yu <yu...@gmail.com> wrote:
> Do you have System.exit() as the last line in your main() ?
> Job job = createSubmittableJob(conf, otherArgs);
> System.exit(job.waitForCompletion(true)? 0 : 1);
>
>
> On Sat, Feb 20, 2010 at 12:32 AM, Zheng Lv <lv...@gmail.com>wrote:
>
>> Hello Ted,
>> Yes. Every hour a job will be created and started, and when it finished,
>> it will maintain. The logs looks like normal, do you know what can lead to
>> this happen?Thank you.
>> LvZheng
>> 2010/2/20 Ted Yu <yu...@gmail.com>
>>
>> > Did the number of child processes increase over time ?
>> >
>> > On Friday, February 19, 2010, Zheng Lv <lv...@gmail.com>
>> wrote:
>> > > Hello Edson,
>> > > Thank you for your reply. I don't want to kill them, I want to know
>> why
>> > > these child processes don't exit, and to know how to make them exit
>> > > successfully when they finish. Any ideas? Thank you.
>> > > LvZheng
>> > >
>> > > 2010/2/18 Edson Ramiro <er...@gmail.com>
>> > >
>> > >> Do you want to kill them ?
>> > >>
>> > >> if yes, you can use
>> > >>
>> > >> ./bin/slaves.sh pkill java
>> > >>
>> > >> but it will kill the datanode and tasktracker processes
>> > >> in all slaves and you'll need to start these processes again.
>> > >>
>> > >> Edson Ramiro
>> > >>
>> > >>
>> > >> On 14 February 2010 22:09, Zheng Lv <lv...@gmail.com>
>> wrote:
>> > >>
>> > >> > any idea?
>> > >> >
>> > >> > 2010/2/11 Zheng Lv <lv...@gmail.com>
>> > >> >
>> > >> > > Hello Everyone,
>> > >> > > We often find many child processes in datanodes, which have
>> > already
>> > >> > > finished for long time. And following are the jstack log:
>> > >> > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01
>> mixed
>> > >> > mode):
>> > >> > > "DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422 waiting
>> on
>> > >> > > condition [0x0000000000000000]
>> > >> > > java.lang.Thread.State: RUNNABLE
>> > >> > > "NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826
>> runnable
>> > >> > > [0x000000004100a000]
>> > >> > > java.lang.Thread.State: RUNNABLE
>> > >> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > >> > > at
>> > >> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>> > >> > > at
>> > >> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> > >> > > at
>> > >> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> > >> > > - locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1)
>> > >> > > - locked <0x00002aaab9b5f710> (a
>> > >> > > java.util.Collections$UnmodifiableSet)
>> > >> > > - locked <0x00002aaab9b5f680> (a
>> > sun.nio.ch.EPollSelectorImpl)
>> > >> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> > >> > > at
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
>> > >> > > at
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672)
>> > >> > > at
>> > >> > >
>> > >> >
>> > >>
>> >
>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
>> > >> > > at
>> > >> > >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > >> > > at
>> > >> > >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > >> > > at java.lang.Thread.run(Thread.java:619)
>> > >> > > "pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825
>> waiting
>> > on
>> > >> > > condition [0x0000000041604000]
>> > >> > > java.lang.Thread.State: WAITING (parking)
>> > >> > > at sun.misc.Unsafe.park(Native Method)
>> > >> > > - parking to wait for <0x00002aaab9b61620> (a
>> > >> > >
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > >> > > at
>> > >> > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> > >> > > at
>> > >> > >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>> > >> > > at
>> > >> > >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> > >> > > at
>> > >> > >
>> > >> >
>> > >>
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>> > >> > > at
>> > >> > >
>> > >> >
>> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa
>> >
>>
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals
Re: Many child processes dont exit
Posted by Ted Yu <yu...@gmail.com>.
Do you have System.exit() as the last line in your main() ?
Job job = createSubmittableJob(conf, otherArgs);
System.exit(job.waitForCompletion(true)? 0 : 1);
On Sat, Feb 20, 2010 at 12:32 AM, Zheng Lv <lv...@gmail.com>wrote:
> Hello Ted,
> Yes. Every hour a job will be created and started, and when it finished,
> it will maintain. The logs looks like normal, do you know what can lead to
> this happen?Thank you.
> LvZheng
> 2010/2/20 Ted Yu <yu...@gmail.com>
>
> > Did the number of child processes increase over time ?
> >
> > On Friday, February 19, 2010, Zheng Lv <lv...@gmail.com>
> wrote:
> > > Hello Edson,
> > > Thank you for your reply. I don't want to kill them, I want to know
> why
> > > these child processes don't exit, and to know how to make them exit
> > > successfully when they finish. Any ideas? Thank you.
> > > LvZheng
> > >
> > > 2010/2/18 Edson Ramiro <er...@gmail.com>
> > >
> > >> Do you want to kill them ?
> > >>
> > >> if yes, you can use
> > >>
> > >> ./bin/slaves.sh pkill java
> > >>
> > >> but it will kill the datanode and tasktracker processes
> > >> in all slaves and you'll need to start these processes again.
> > >>
> > >> Edson Ramiro
> > >>
> > >>
> > >> On 14 February 2010 22:09, Zheng Lv <lv...@gmail.com>
> wrote:
> > >>
> > >> > any idea?
> > >> >
> > >> > 2010/2/11 Zheng Lv <lv...@gmail.com>
> > >> >
> > >> > > Hello Everyone,
> > >> > > We often find many child processes in datanodes, which have
> > already
> > >> > > finished for long time. And following are the jstack log:
> > >> > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01
> mixed
> > >> > mode):
> > >> > > "DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422 waiting
> on
> > >> > > condition [0x0000000000000000]
> > >> > > java.lang.Thread.State: RUNNABLE
> > >> > > "NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826
> runnable
> > >> > > [0x000000004100a000]
> > >> > > java.lang.Thread.State: RUNNABLE
> > >> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > >> > > at
> > >> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> > >> > > at
> > >> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > >> > > at
> > >> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > >> > > - locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1)
> > >> > > - locked <0x00002aaab9b5f710> (a
> > >> > > java.util.Collections$UnmodifiableSet)
> > >> > > - locked <0x00002aaab9b5f680> (a
> > sun.nio.ch.EPollSelectorImpl)
> > >> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > >> > > at
> > >> > >
> > >> >
> > >>
> >
> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
> > >> > > at
> > >> > >
> > >> >
> > >>
> >
> org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672)
> > >> > > at
> > >> > >
> > >> >
> > >>
> >
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
> > >> > > at
> > >> > >
> > >> >
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > >> > > at
> > >> > >
> > >> >
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > >> > > at java.lang.Thread.run(Thread.java:619)
> > >> > > "pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825
> waiting
> > on
> > >> > > condition [0x0000000041604000]
> > >> > > java.lang.Thread.State: WAITING (parking)
> > >> > > at sun.misc.Unsafe.park(Native Method)
> > >> > > - parking to wait for <0x00002aaab9b61620> (a
> > >> > >
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > >> > > at
> > >> > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> > >> > > at
> > >> > >
> > >> >
> > >>
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> > >> > > at
> > >> > >
> > >> >
> > >>
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> > >> > > at
> > >> > >
> > >> >
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > >> > > at
> > >> > >
> > >> >
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa
> >
>
Re: Many child processes dont exit
Posted by Zheng Lv <lv...@gmail.com>.
Hello Ted,
Yes. Every hour a job will be created and started, and when it finished,
it will maintain. The logs looks like normal, do you know what can lead to
this happen?Thank you.
LvZheng
2010/2/20 Ted Yu <yu...@gmail.com>
> Did the number of child processes increase over time ?
>
> On Friday, February 19, 2010, Zheng Lv <lv...@gmail.com> wrote:
> > Hello Edson,
> > Thank you for your reply. I don't want to kill them, I want to know why
> > these child processes don't exit, and to know how to make them exit
> > successfully when they finish. Any ideas? Thank you.
> > LvZheng
> >
> > 2010/2/18 Edson Ramiro <er...@gmail.com>
> >
> >> Do you want to kill them ?
> >>
> >> if yes, you can use
> >>
> >> ./bin/slaves.sh pkill java
> >>
> >> but it will kill the datanode and tasktracker processes
> >> in all slaves and you'll need to start these processes again.
> >>
> >> Edson Ramiro
> >>
> >>
> >> On 14 February 2010 22:09, Zheng Lv <lv...@gmail.com> wrote:
> >>
> >> > any idea?
> >> >
> >> > 2010/2/11 Zheng Lv <lv...@gmail.com>
> >> >
> >> > > Hello Everyone,
> >> > > We often find many child processes in datanodes, which have
> already
> >> > > finished for long time. And following are the jstack log:
> >> > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed
> >> > mode):
> >> > > "DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422 waiting on
> >> > > condition [0x0000000000000000]
> >> > > java.lang.Thread.State: RUNNABLE
> >> > > "NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826 runnable
> >> > > [0x000000004100a000]
> >> > > java.lang.Thread.State: RUNNABLE
> >> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >> > > at
> >> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> >> > > at
> >> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> >> > > at
> >> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> >> > > - locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1)
> >> > > - locked <0x00002aaab9b5f710> (a
> >> > > java.util.Collections$UnmodifiableSet)
> >> > > - locked <0x00002aaab9b5f680> (a
> sun.nio.ch.EPollSelectorImpl)
> >> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
> >> > > at
> >> > >
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> > > at
> >> > >
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> > > at java.lang.Thread.run(Thread.java:619)
> >> > > "pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825 waiting
> on
> >> > > condition [0x0000000041604000]
> >> > > java.lang.Thread.State: WAITING (parking)
> >> > > at sun.misc.Unsafe.park(Native Method)
> >> > > - parking to wait for <0x00002aaab9b61620> (a
> >> > >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >> > > at
> >> > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >> > > at
> >> > >
> >> >
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> >> > > at
> >> > >
> >> >
> >>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> >> > > at
> >> > >
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> >> > > at
> >> > >
> >> >
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa
>
Re: Many child processes dont exit
Posted by Ted Yu <yu...@gmail.com>.
Did the number of child processes increase over time ?
On Friday, February 19, 2010, Zheng Lv <lv...@gmail.com> wrote:
> Hello Edson,
> Thank you for your reply. I don't want to kill them, I want to know why
> these child processes don't exit, and to know how to make them exit
> successfully when they finish. Any ideas? Thank you.
> LvZheng
>
> 2010/2/18 Edson Ramiro <er...@gmail.com>
>
>> Do you want to kill them ?
>>
>> if yes, you can use
>>
>> ./bin/slaves.sh pkill java
>>
>> but it will kill the datanode and tasktracker processes
>> in all slaves and you'll need to start these processes again.
>>
>> Edson Ramiro
>>
>>
>> On 14 February 2010 22:09, Zheng Lv <lv...@gmail.com> wrote:
>>
>> > any idea?
>> >
>> > 2010/2/11 Zheng Lv <lv...@gmail.com>
>> >
>> > > Hello Everyone,
>> > > We often find many child processes in datanodes, which have already
>> > > finished for long time. And following are the jstack log:
>> > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed
>> > mode):
>> > > "DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422 waiting on
>> > > condition [0x0000000000000000]
>> > > java.lang.Thread.State: RUNNABLE
>> > > "NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826 runnable
>> > > [0x000000004100a000]
>> > > java.lang.Thread.State: RUNNABLE
>> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> > > at
>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>> > > at
>> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> > > at
>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> > > - locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1)
>> > > - locked <0x00002aaab9b5f710> (a
>> > > java.util.Collections$UnmodifiableSet)
>> > > - locked <0x00002aaab9b5f680> (a sun.nio.ch.EPollSelectorImpl)
>> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> > > at
>> > >
>> >
>> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
>> > > at
>> > >
>> >
>> org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672)
>> > > at
>> > >
>> >
>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
>> > > at
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > > at
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > > at java.lang.Thread.run(Thread.java:619)
>> > > "pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825 waiting on
>> > > condition [0x0000000041604000]
>> > > java.lang.Thread.State: WAITING (parking)
>> > > at sun.misc.Unsafe.park(Native Method)
>> > > - parking to wait for <0x00002aaab9b61620> (a
>> > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> > > at
>> > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>> > > at
>> > >
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>> > > at
>> > >
>> >
>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>> > > at
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>> > > at
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa
Re: Many child processes dont exit
Posted by Zheng Lv <lv...@gmail.com>.
Hello Edson,
Thank you for your reply. I don't want to kill them, I want to know why
these child processes don't exit, and to know how to make them exit
successfully when they finish. Any ideas? Thank you.
LvZheng
2010/2/18 Edson Ramiro <er...@gmail.com>
> Do you want to kill them ?
>
> if yes, you can use
>
> ./bin/slaves.sh pkill java
>
> but it will kill the datanode and tasktracker processes
> in all slaves and you'll need to start these processes again.
>
> Edson Ramiro
>
>
> On 14 February 2010 22:09, Zheng Lv <lv...@gmail.com> wrote:
>
> > any idea?
> >
> > 2010/2/11 Zheng Lv <lv...@gmail.com>
> >
> > > Hello Everyone,
> > > We often find many child processes in datanodes, which have already
> > > finished for long time. And following are the jstack log:
> > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed
> > mode):
> > > "DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422 waiting on
> > > condition [0x0000000000000000]
> > > java.lang.Thread.State: RUNNABLE
> > > "NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826 runnable
> > > [0x000000004100a000]
> > > java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> > > at
> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1)
> > > - locked <0x00002aaab9b5f710> (a
> > > java.util.Collections$UnmodifiableSet)
> > > - locked <0x00002aaab9b5f680> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at
> > >
> >
> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
> > > at
> > >
> >
> org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672)
> > > at
> > >
> >
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:619)
> > > "pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825 waiting on
> > > condition [0x0000000041604000]
> > > java.lang.Thread.State: WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for <0x00002aaab9b61620> (a
> > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at
> > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> > > at
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> > > at
> > >
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:619)
> > > "Attach Listener" daemon prio=10 tid=0x0000000043a22800 nid=0x2608
> > waiting
> > > on condition [0x0000000000000000]
> > > java.lang.Thread.State: RUNNABLE
> > > "pool-3-thread-1" prio=10 tid=0x00000000438ad000 nid=0x243f waiting on
> > > condition [0x0000000042575000]
> > > java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for <0x00002aaab9ae8770> (a
> > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at
> > > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> > > at
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
> > > at
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:619)
> > > "Thread for syncLogs" daemon prio=10 tid=0x00002aaac4446800 nid=0x2437
> > > waiting on condition [0x0000000040f09000]
> > > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > > at java.lang.Thread.sleep(Native Method)
> > > at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
> > > "Low Memory Detector" daemon prio=10 tid=0x0000000043771800 nid=0x242d
> > > runnable [0x0000000000000000]
> > > java.lang.Thread.State: RUNNABLE
> > > "CompilerThread1" daemon prio=10 tid=0x000000004376e800 nid=0x242c
> > waiting
> > > on condition [0x0000000000000000]
> > > java.lang.Thread.State: RUNNABLE
> > > "CompilerThread0" daemon prio=10 tid=0x000000004376a800 nid=0x242b
> > waiting
> > > on condition [0x0000000000000000]
> > > java.lang.Thread.State: RUNNABLE
> > > "Signal Dispatcher" daemon prio=10 tid=0x0000000043768800 nid=0x242a
> > > runnable [0x0000000000000000]
> > > java.lang.Thread.State: RUNNABLE
> > > "Finalizer" daemon prio=10 tid=0x0000000043746000 nid=0x2429 in
> > > Object.wait() [0x0000000042171000]
> > > java.lang.Thread.State: WAITING (on object monitor)
> > > at java.lang.Object.wait(Native Method)
> > > - waiting on <0x00002aaab9a8afe0> (a
> > > java.lang.ref.ReferenceQueue$Lock)
> > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > > - locked <0x00002aaab9a8afe0> (a
> > java.lang.ref.ReferenceQueue$Lock)
> > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > > at
> > java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> > > "Reference Handler" daemon prio=10 tid=0x0000000043744000 nid=0x2428 in
> > > Object.wait() [0x0000000042070000]
> > > java.lang.Thread.State: WAITING (on object monitor)
> > > at java.lang.Object.wait(Native Method)
> > > - waiting on <0x00002aaab96b13a0> (a
> > java.lang.ref.Reference$Lock)
> > > at java.lang.Object.wait(Object.java:485)
> > > at
> > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > > - locked <0x00002aaab96b13a0> (a java.lang.ref.Reference$Lock)
> > > "VM Thread" prio=10 tid=0x000000004373d800 nid=0x2427 runnable
> > > "GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000436eb800
> nid=0x2423
> > > runnable
> > > "GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000436ed800
> nid=0x2424
> > > runnable
> > > "GC task thread#2 (ParallelGC)" prio=10 tid=0x00000000436ef800
> > nid=0x2425
> > > runnable
> > > "GC task thread#3 (ParallelGC)" prio=10 tid=0x00000000436f1000
> nid=0x2426
> > > runnable
> > > "VM Periodic Task Thread" prio=10 tid=0x0000000043774000 nid=0x242e
> > waiting
> > > on condition
> > > JNI global references: 871
> > > any suggestion?
> > > best regards,
> > > LvZheng.
> > >
> >
>
Re: Many child processes dont exit
Posted by Edson Ramiro <er...@gmail.com>.
Do you want to kill them ?
if yes, you can use
./bin/slaves.sh pkill java
but it will kill the datanode and tasktracker processes
in all slaves and you'll need to start these processes again.
Edson Ramiro
On 14 February 2010 22:09, Zheng Lv <lv...@gmail.com> wrote:
> any idea?
>
> 2010/2/11 Zheng Lv <lv...@gmail.com>
>
> > Hello Everyone,
> > We often find many child processes in datanodes, which have already
> > finished for long time. And following are the jstack log:
> > Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed
> mode):
> > "DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422 waiting on
> > condition [0x0000000000000000]
> > java.lang.Thread.State: RUNNABLE
> > "NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826 runnable
> > [0x000000004100a000]
> > java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> > at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > - locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1)
> > - locked <0x00002aaab9b5f710> (a
> > java.util.Collections$UnmodifiableSet)
> > - locked <0x00002aaab9b5f680> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > at
> >
> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
> > at
> >
> org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672)
> > at
> >
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:619)
> > "pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825 waiting on
> > condition [0x0000000041604000]
> > java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <0x00002aaab9b61620> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > at java.lang.Thread.run(Thread.java:619)
> > "Attach Listener" daemon prio=10 tid=0x0000000043a22800 nid=0x2608
> waiting
> > on condition [0x0000000000000000]
> > java.lang.Thread.State: RUNNABLE
> > "pool-3-thread-1" prio=10 tid=0x00000000438ad000 nid=0x243f waiting on
> > condition [0x0000000042575000]
> > java.lang.Thread.State: TIMED_WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <0x00002aaab9ae8770> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at
> > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
> > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > at java.lang.Thread.run(Thread.java:619)
> > "Thread for syncLogs" daemon prio=10 tid=0x00002aaac4446800 nid=0x2437
> > waiting on condition [0x0000000040f09000]
> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> > at java.lang.Thread.sleep(Native Method)
> > at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
> > "Low Memory Detector" daemon prio=10 tid=0x0000000043771800 nid=0x242d
> > runnable [0x0000000000000000]
> > java.lang.Thread.State: RUNNABLE
> > "CompilerThread1" daemon prio=10 tid=0x000000004376e800 nid=0x242c
> waiting
> > on condition [0x0000000000000000]
> > java.lang.Thread.State: RUNNABLE
> > "CompilerThread0" daemon prio=10 tid=0x000000004376a800 nid=0x242b
> waiting
> > on condition [0x0000000000000000]
> > java.lang.Thread.State: RUNNABLE
> > "Signal Dispatcher" daemon prio=10 tid=0x0000000043768800 nid=0x242a
> > runnable [0x0000000000000000]
> > java.lang.Thread.State: RUNNABLE
> > "Finalizer" daemon prio=10 tid=0x0000000043746000 nid=0x2429 in
> > Object.wait() [0x0000000042171000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <0x00002aaab9a8afe0> (a
> > java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > - locked <0x00002aaab9a8afe0> (a
> java.lang.ref.ReferenceQueue$Lock)
> > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > at
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> > "Reference Handler" daemon prio=10 tid=0x0000000043744000 nid=0x2428 in
> > Object.wait() [0x0000000042070000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > - waiting on <0x00002aaab96b13a0> (a
> java.lang.ref.Reference$Lock)
> > at java.lang.Object.wait(Object.java:485)
> > at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > - locked <0x00002aaab96b13a0> (a java.lang.ref.Reference$Lock)
> > "VM Thread" prio=10 tid=0x000000004373d800 nid=0x2427 runnable
> > "GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000436eb800 nid=0x2423
> > runnable
> > "GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000436ed800 nid=0x2424
> > runnable
> > "GC task thread#2 (ParallelGC)" prio=10 tid=0x00000000436ef800
> nid=0x2425
> > runnable
> > "GC task thread#3 (ParallelGC)" prio=10 tid=0x00000000436f1000 nid=0x2426
> > runnable
> > "VM Periodic Task Thread" prio=10 tid=0x0000000043774000 nid=0x242e
> waiting
> > on condition
> > JNI global references: 871
> > any suggestion?
> > best regards,
> > LvZheng.
> >
>
Re: Many child processes dont exit
Posted by Zheng Lv <lv...@gmail.com>.
any idea?
2010/2/11 Zheng Lv <lv...@gmail.com>
> Hello Everyone,
> We often find many child processes in datanodes, which have already
> finished for long time. And following are the jstack log:
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode):
> "DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422 waiting on
> condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826 runnable
> [0x000000004100a000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1)
> - locked <0x00002aaab9b5f710> (a
> java.util.Collections$UnmodifiableSet)
> - locked <0x00002aaab9b5f680> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
> at
> org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672)
> at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> "pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825 waiting on
> condition [0x0000000041604000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00002aaab9b61620> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> "Attach Listener" daemon prio=10 tid=0x0000000043a22800 nid=0x2608 waiting
> on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "pool-3-thread-1" prio=10 tid=0x00000000438ad000 nid=0x243f waiting on
> condition [0x0000000042575000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00002aaab9ae8770> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> "Thread for syncLogs" daemon prio=10 tid=0x00002aaac4446800 nid=0x2437
> waiting on condition [0x0000000040f09000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
> "Low Memory Detector" daemon prio=10 tid=0x0000000043771800 nid=0x242d
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "CompilerThread1" daemon prio=10 tid=0x000000004376e800 nid=0x242c waiting
> on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "CompilerThread0" daemon prio=10 tid=0x000000004376a800 nid=0x242b waiting
> on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" daemon prio=10 tid=0x0000000043768800 nid=0x242a
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Finalizer" daemon prio=10 tid=0x0000000043746000 nid=0x2429 in
> Object.wait() [0x0000000042171000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab9a8afe0> (a
> java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x00002aaab9a8afe0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> "Reference Handler" daemon prio=10 tid=0x0000000043744000 nid=0x2428 in
> Object.wait() [0x0000000042070000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab96b13a0> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x00002aaab96b13a0> (a java.lang.ref.Reference$Lock)
> "VM Thread" prio=10 tid=0x000000004373d800 nid=0x2427 runnable
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000436eb800 nid=0x2423
> runnable
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000436ed800 nid=0x2424
> runnable
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00000000436ef800 nid=0x2425
> runnable
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00000000436f1000 nid=0x2426
> runnable
> "VM Periodic Task Thread" prio=10 tid=0x0000000043774000 nid=0x242e waiting
> on condition
> JNI global references: 871
> any suggestion?
> best regards,
> LvZheng.
>