You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by C J <c....@yahoo.com> on 2010/08/25 21:20:30 UTC
Child processes on datanodes/task trackers
Hi,
I wanted to know why I see running Child processes on my datanodes even though
there is no job running at that time. Are these left over from failed attempts?
Is there anything I can do to keep these clean?
Thanks,
Deepika
Re: Child processes on datanodes/task trackers
Posted by Steve Loughran <st...@apache.org>.
On 25/08/10 20:20, C J wrote:
> Hi,
>
> I wanted to know why I see running Child processes on my datanodes even though
> there is no job running at that time. Are these left over from failed attempts?
>
> Is there anything I can do to keep these clean?
>
> Thanks,
> Deepika
>
>
>
could be a bug somewhere in Hadoop or your code.
1. jps -v gives you the process ids,
2. jstack -l <pid> gives you the stack dumps and locks for the given
process, so you can see where the threads are -this will help you track
down the problem
3. It is, sadly, going to be you who gets to track down the problem. But
if you conclude that it is hadoop and not your code at fault, those
stack traces should be attached to the JIRA issue you can create
Re: Child processes on datanodes/task trackers
Posted by C J <c....@yahoo.com>.
Thanks Ted. Yes, I forgot to shutdown the CacheManager.
Appreciate your help.
________________________________
From: Ted Yu <yu...@gmail.com>
To: general@hadoop.apache.org
Sent: Wed, August 25, 2010 8:39:01 PM
Subject: Re: Child processes on datanodes/task trackers
I don't use ehcache.
Did you forget to close CacheManager at the end of your job by any chance ?
On Wed, Aug 25, 2010 at 7:59 PM, C J <c....@yahoo.com> wrote:
> Thanks Ted!
>
> I did a jstack and it seems there is an issue with ehcache that I am using
> in
> the mapper task.
>
>
> "net.sf.ehcache.CacheManager@57ac3379" daemon prio=10
> tid=0x0000000059180800
> nid=0x379e in Object.wait() [0x0000000041506000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaabb0b89a8> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> - locked <0x00002aaabb0b89a8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:462)
>
> Locked ownable synchronizers:
> - None
> .
> .
> .
>
>
> The hadoop version I am using is 0.20.2.
>
> Thanks.
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: general@hadoop.apache.org
> Sent: Wed, August 25, 2010 7:34:35 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> After you obtain pid, you can use jstack to see what the Child process was
> doing.
>
> What hadoop version are you using ?
>
> On Wed, Aug 25, 2010 at 7:28 PM, C J <c....@yahoo.com> wrote:
>
> > Thanks for your reply.
> >
> > Some of these child tasks belong to successful jobs. I am wondering why
> > they are
> > still hanging there for long finished jobs.
> >
> >
> >
> >
> >
> > ________________________________
> > From: Ted Yu <yu...@gmail.com>
> > To: general@hadoop.apache.org
> > Sent: Wed, August 25, 2010 4:17:38 PM
> > Subject: Re: Child processes on datanodes/task trackers
> >
> > Use jps to find out pid of the Child.
> > Then use this to find out which job the Child belongs to:
> > ps aux | grep <pid>
> >
> > On Wed, Aug 25, 2010 at 12:20 PM, C J <c....@yahoo.com> wrote:
> >
> > > Hi,
> > >
> > > I wanted to know why I see running Child processes on my datanodes even
> > > though
> > > there is no job running at that time. Are these left over from failed
> > > attempts?
> > >
> > > Is there anything I can do to keep these clean?
> > >
> > > Thanks,
> > > Deepika
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
Re: Child processes on datanodes/task trackers
Posted by Ted Yu <yu...@gmail.com>.
I don't use ehcache.
Did you forget to close CacheManager at the end of your job by any chance ?
On Wed, Aug 25, 2010 at 7:59 PM, C J <c....@yahoo.com> wrote:
> Thanks Ted!
>
> I did a jstack and it seems there is an issue with ehcache that I am using
> in
> the mapper task.
>
>
> "net.sf.ehcache.CacheManager@57ac3379" daemon prio=10
> tid=0x0000000059180800
> nid=0x379e in Object.wait() [0x0000000041506000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaabb0b89a8> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> - locked <0x00002aaabb0b89a8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:462)
>
> Locked ownable synchronizers:
> - None
> .
> .
> .
>
>
> The hadoop version I am using is 0.20.2.
>
> Thanks.
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: general@hadoop.apache.org
> Sent: Wed, August 25, 2010 7:34:35 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> After you obtain pid, you can use jstack to see what the Child process was
> doing.
>
> What hadoop version are you using ?
>
> On Wed, Aug 25, 2010 at 7:28 PM, C J <c....@yahoo.com> wrote:
>
> > Thanks for your reply.
> >
> > Some of these child tasks belong to successful jobs. I am wondering why
> > they are
> > still hanging there for long finished jobs.
> >
> >
> >
> >
> >
> > ________________________________
> > From: Ted Yu <yu...@gmail.com>
> > To: general@hadoop.apache.org
> > Sent: Wed, August 25, 2010 4:17:38 PM
> > Subject: Re: Child processes on datanodes/task trackers
> >
> > Use jps to find out pid of the Child.
> > Then use this to find out which job the Child belongs to:
> > ps aux | grep <pid>
> >
> > On Wed, Aug 25, 2010 at 12:20 PM, C J <c....@yahoo.com> wrote:
> >
> > > Hi,
> > >
> > > I wanted to know why I see running Child processes on my datanodes even
> > > though
> > > there is no job running at that time. Are these left over from failed
> > > attempts?
> > >
> > > Is there anything I can do to keep these clean?
> > >
> > > Thanks,
> > > Deepika
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
Re: Child processes on datanodes/task trackers
Posted by C J <c....@yahoo.com>.
Thanks Ted!
I did a jstack and it seems there is an issue with ehcache that I am using in
the mapper task.
"net.sf.ehcache.CacheManager@57ac3379" daemon prio=10 tid=0x0000000059180800
nid=0x379e in Object.wait() [0x0000000041506000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaabb0b89a8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00002aaabb0b89a8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
Locked ownable synchronizers:
- None
.
.
.
The hadoop version I am using is 0.20.2.
Thanks.
________________________________
From: Ted Yu <yu...@gmail.com>
To: general@hadoop.apache.org
Sent: Wed, August 25, 2010 7:34:35 PM
Subject: Re: Child processes on datanodes/task trackers
After you obtain pid, you can use jstack to see what the Child process was
doing.
What hadoop version are you using ?
On Wed, Aug 25, 2010 at 7:28 PM, C J <c....@yahoo.com> wrote:
> Thanks for your reply.
>
> Some of these child tasks belong to successful jobs. I am wondering why
> they are
> still hanging there for long finished jobs.
>
>
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: general@hadoop.apache.org
> Sent: Wed, August 25, 2010 4:17:38 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> Use jps to find out pid of the Child.
> Then use this to find out which job the Child belongs to:
> ps aux | grep <pid>
>
> On Wed, Aug 25, 2010 at 12:20 PM, C J <c....@yahoo.com> wrote:
>
> > Hi,
> >
> > I wanted to know why I see running Child processes on my datanodes even
> > though
> > there is no job running at that time. Are these left over from failed
> > attempts?
> >
> > Is there anything I can do to keep these clean?
> >
> > Thanks,
> > Deepika
> >
> >
> >
>
>
>
>
>
Re: Child processes on datanodes/task trackers
Posted by Ted Yu <yu...@gmail.com>.
After you obtain pid, you can use jstack to see what the Child process was
doing.
What hadoop version are you using ?
On Wed, Aug 25, 2010 at 7:28 PM, C J <c....@yahoo.com> wrote:
> Thanks for your reply.
>
> Some of these child tasks belong to successful jobs. I am wondering why
> they are
> still hanging there for long finished jobs.
>
>
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: general@hadoop.apache.org
> Sent: Wed, August 25, 2010 4:17:38 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> Use jps to find out pid of the Child.
> Then use this to find out which job the Child belongs to:
> ps aux | grep <pid>
>
> On Wed, Aug 25, 2010 at 12:20 PM, C J <c....@yahoo.com> wrote:
>
> > Hi,
> >
> > I wanted to know why I see running Child processes on my datanodes even
> > though
> > there is no job running at that time. Are these left over from failed
> > attempts?
> >
> > Is there anything I can do to keep these clean?
> >
> > Thanks,
> > Deepika
> >
> >
> >
>
>
>
>
>
Re: Child processes on datanodes/task trackers
Posted by C J <c....@yahoo.com>.
Thanks for your reply.
Some of these child tasks belong to successful jobs. I am wondering why they are
still hanging there for long finished jobs.
________________________________
From: Ted Yu <yu...@gmail.com>
To: general@hadoop.apache.org
Sent: Wed, August 25, 2010 4:17:38 PM
Subject: Re: Child processes on datanodes/task trackers
Use jps to find out pid of the Child.
Then use this to find out which job the Child belongs to:
ps aux | grep <pid>
On Wed, Aug 25, 2010 at 12:20 PM, C J <c....@yahoo.com> wrote:
> Hi,
>
> I wanted to know why I see running Child processes on my datanodes even
> though
> there is no job running at that time. Are these left over from failed
> attempts?
>
> Is there anything I can do to keep these clean?
>
> Thanks,
> Deepika
>
>
>
Re: Child processes on datanodes/task trackers
Posted by Ted Yu <yu...@gmail.com>.
Use jps to find out pid of the Child.
Then use this to find out which job the Child belongs to:
ps aux | grep <pid>
On Wed, Aug 25, 2010 at 12:20 PM, C J <c....@yahoo.com> wrote:
> Hi,
>
> I wanted to know why I see running Child processes on my datanodes even
> though
> there is no job running at that time. Are these left over from failed
> attempts?
>
> Is there anything I can do to keep these clean?
>
> Thanks,
> Deepika
>
>
>