You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Juwei Shi <sh...@gmail.com> on 2011/07/01 17:52:55 UTC
Jobs are still in running state after executing "hadoop job -kill jobId"
Hi,
I faced a problem that the jobs are still running after executing "hadoop
job -kill jobId". I rebooted the cluster but the job still can not be
killed.
The hadoop version is 0.20.2.
Any idea?
Thanks in advance!
--
- Juwei
Re: Jobs are still in running state after executing "hadoop job -kill jobId"
Posted by Juwei Shi <sh...@gmail.com>.
Harsh,
It works. Thanks a lot!!!
2011/7/2 Harsh J <ha...@cloudera.com>
> Juwei,
>
> Its odd that a killed job should get "recovered" back into running
> state. Can you not simply disable the JT recovery feature (I believe
> its turned off by default)?
>
> On Fri, Jul 1, 2011 at 10:47 PM, Juwei Shi <sh...@gmail.com> wrote:
> > Thanks Harsh.
> >
> > There are "recover" jobs after I re-boot mapreduce/hdfs.
> >
> > Is there any other way to delete the status records of the running jobs?
> > Then they will not recover after restarting JT?
> >
> > 2011/7/2 Harsh J <ha...@cloudera.com>
> >>
> >> Juwei,
> >>
> >> Please do not cross-post to multiple lists. I believe this question
> >> suits the mapreduce-user@ list so am replying only on there.
> >>
> >> On Fri, Jul 1, 2011 at 9:22 PM, Juwei Shi <sh...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I faced a problem that the jobs are still running after executing
> >> > "hadoop
> >> > job -kill jobId". I rebooted the cluster but the job still can not be
> >> > killed.
> >>
> >> What does the JT logs say after you attempt to kill a job ID? Does the
> >> same Job ID keep running even after or are you seeing other jobs
> >> continue to launch?
> >>
> >> --
> >> Harsh J
> >
> > --
> > - Juwei
> >
>
> --
> Harsh J
>
--
- Juwei
Re: Jobs are still in running state after executing "hadoop job -kill jobId"
Posted by Harsh J <ha...@cloudera.com>.
Juwei,
Its odd that a killed job should get "recovered" back into running
state. Can you not simply disable the JT recovery feature (I believe
its turned off by default)?
On Fri, Jul 1, 2011 at 10:47 PM, Juwei Shi <sh...@gmail.com> wrote:
> Thanks Harsh.
>
> There are "recover" jobs after I re-boot mapreduce/hdfs.
>
> Is there any other way to delete the status records of the running jobs?
> Then they will not recover after restarting JT?
>
> 2011/7/2 Harsh J <ha...@cloudera.com>
>>
>> Juwei,
>>
>> Please do not cross-post to multiple lists. I believe this question
>> suits the mapreduce-user@ list so am replying only on there.
>>
>> On Fri, Jul 1, 2011 at 9:22 PM, Juwei Shi <sh...@gmail.com> wrote:
>> > Hi,
>> >
>> > I faced a problem that the jobs are still running after executing
>> > "hadoop
>> > job -kill jobId". I rebooted the cluster but the job still can not be
>> > killed.
>>
>> What does the JT logs say after you attempt to kill a job ID? Does the
>> same Job ID keep running even after or are you seeing other jobs
>> continue to launch?
>>
>> --
>> Harsh J
>
> --
> - Juwei
>
--
Harsh J
Re: Jobs are still in running state after executing "hadoop job -kill jobId"
Posted by Juwei Shi <sh...@gmail.com>.
Thanks Harsh.
There are "recover" jobs after I re-boot mapreduce/hdfs.
Is there any other way to delete the status records of the running jobs?
Then they will not recover after restarting JT?
2011/7/2 Harsh J <ha...@cloudera.com>
> Juwei,
>
> Please do not cross-post to multiple lists. I believe this question
> suits the mapreduce-user@ list so am replying only on there.
>
> On Fri, Jul 1, 2011 at 9:22 PM, Juwei Shi <sh...@gmail.com> wrote:
> > Hi,
> >
> > I faced a problem that the jobs are still running after executing "hadoop
> > job -kill jobId". I rebooted the cluster but the job still can not be
> > killed.
>
> What does the JT logs say after you attempt to kill a job ID? Does the
> same Job ID keep running even after or are you seeing other jobs
> continue to launch?
>
> --
> Harsh J
>
--
- Juwei
Re: Jobs are still in running state after executing "hadoop job -kill jobId"
Posted by Harsh J <ha...@cloudera.com>.
Juwei,
Please do not cross-post to multiple lists. I believe this question
suits the mapreduce-user@ list so am replying only on there.
On Fri, Jul 1, 2011 at 9:22 PM, Juwei Shi <sh...@gmail.com> wrote:
> Hi,
>
> I faced a problem that the jobs are still running after executing "hadoop
> job -kill jobId". I rebooted the cluster but the job still can not be
> killed.
What does the JT logs say after you attempt to kill a job ID? Does the
same Job ID keep running even after or are you seeing other jobs
continue to launch?
--
Harsh J
Re: Jobs are still in running state after executing "hadoop job -kill jobId"
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Jul 5, 2011 at 10:05 AM, <Je...@shell.com> wrote:
> Um kill -9 "pid" ?
>
> -----Original Message-----
> From: Juwei Shi [mailto:shijuwei@gmail.com]
> Sent: Friday, July 01, 2011 10:53 AM
> To: common-user@hadoop.apache.org; mapreduce-user@hadoop.apache.org
> Subject: Jobs are still in running state after executing "hadoop job
> -kill jobId"
>
> Hi,
>
> I faced a problem that the jobs are still running after executing
> "hadoop
> job -kill jobId". I rebooted the cluster but the job still can not be
> killed.
>
> The hadoop version is 0.20.2.
>
> Any idea?
>
> Thanks in advance!
>
> --
> - Juwei
>
>
This happens sometimes. A task gets orphaned from the Task Tracker and never
goes away. It is a good idea to have a nagios check for very old tasks
because the orphans slowly such your memory away especially if the task
launches with a big Xmx. You really *should not* need to be nuking tasks
like this but occasionally it happens.
Edward
Re: Jobs are still in running state after executing "hadoop job -kill jobId"
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Jul 5, 2011 at 11:45 AM, Juwei Shi <sh...@gmail.com> wrote:
> We sometimes have hundreds of map or reduce tasks for a job. I think it is
> hard to find all of them and kill the corresponding jvm processes. If we do
> not want to restart hadoop, is there any automatic methods?
>
> 2011/7/5 <Je...@shell.com>
>
> > Um kill -9 "pid" ?
> >
> > -----Original Message-----
> > From: Juwei Shi [mailto:shijuwei@gmail.com]
> > Sent: Friday, July 01, 2011 10:53 AM
> > To: common-user@hadoop.apache.org; mapreduce-user@hadoop.apache.org
> > Subject: Jobs are still in running state after executing "hadoop job
> > -kill jobId"
> >
> > Hi,
> >
> > I faced a problem that the jobs are still running after executing
> > "hadoop
> > job -kill jobId". I rebooted the cluster but the job still can not be
> > killed.
> >
> > The hadoop version is 0.20.2.
> >
> > Any idea?
> >
> > Thanks in advance!
> >
> > --
> > - Juwei
> >
> >
>
I do not think they pop up very often but after days and months of running a
orphans can be alive. The way I would handle it is write a check that runs
over Nagios (NRPE) and looks for Hadoop task processes using ps, that are
older then a certain age such as 1 day or 1 week etc. Then you can decide if
want nagios to terminate these orphans or do it by hand.
Edward
Re: Jobs are still in running state after executing "hadoop job -kill jobId"
Posted by Juwei Shi <sh...@gmail.com>.
I am sorry that I cc to the common-user again. Please reply this mail to
avoid including common-user list agian.
2011/7/5 Juwei Shi <sh...@gmail.com>
> We sometimes have hundreds of map or reduce tasks for a job. I think it is
> hard to find all of them and kill the corresponding jvm processes. If we do
> not want to restart hadoop, is there any automatic methods?
>
> 2011/7/5 <Je...@shell.com>
>
> Um kill -9 "pid" ?
>>
>> -----Original Message-----
>> From: Juwei Shi [mailto:shijuwei@gmail.com]
>> Sent: Friday, July 01, 2011 10:53 AM
>> To: common-user@hadoop.apache.org; mapreduce-user@hadoop.apache.org
>> Subject: Jobs are still in running state after executing "hadoop job
>> -kill jobId"
>>
>> Hi,
>>
>> I faced a problem that the jobs are still running after executing
>> "hadoop
>> job -kill jobId". I rebooted the cluster but the job still can not be
>> killed.
>>
>> The hadoop version is 0.20.2.
>>
>> Any idea?
>>
>> Thanks in advance!
>>
>> --
>> - Juwei
>>
>>
>
>
>
>
Re: Jobs are still in running state after executing "hadoop job -kill jobId"
Posted by Juwei Shi <sh...@gmail.com>.
We sometimes have hundreds of map or reduce tasks for a job. I think it is
hard to find all of them and kill the corresponding jvm processes. If we do
not want to restart hadoop, is there any automatic methods?
2011/7/5 <Je...@shell.com>
> Um kill -9 "pid" ?
>
> -----Original Message-----
> From: Juwei Shi [mailto:shijuwei@gmail.com]
> Sent: Friday, July 01, 2011 10:53 AM
> To: common-user@hadoop.apache.org; mapreduce-user@hadoop.apache.org
> Subject: Jobs are still in running state after executing "hadoop job
> -kill jobId"
>
> Hi,
>
> I faced a problem that the jobs are still running after executing
> "hadoop
> job -kill jobId". I rebooted the cluster but the job still can not be
> killed.
>
> The hadoop version is 0.20.2.
>
> Any idea?
>
> Thanks in advance!
>
> --
> - Juwei
>
>
Re: Jobs are still in running state after executing "hadoop job -kill jobId"
Posted by Juwei Shi <sh...@gmail.com>.
We sometimes have hundreds of map or reduce tasks for a job. I think it is
hard to find all of them and kill the corresponding jvm processes. If we do
not want to restart hadoop, is there any automatic methods?
2011/7/5 <Je...@shell.com>
> Um kill -9 "pid" ?
>
> -----Original Message-----
> From: Juwei Shi [mailto:shijuwei@gmail.com]
> Sent: Friday, July 01, 2011 10:53 AM
> To: common-user@hadoop.apache.org; mapreduce-user@hadoop.apache.org
> Subject: Jobs are still in running state after executing "hadoop job
> -kill jobId"
>
> Hi,
>
> I faced a problem that the jobs are still running after executing
> "hadoop
> job -kill jobId". I rebooted the cluster but the job still can not be
> killed.
>
> The hadoop version is 0.20.2.
>
> Any idea?
>
> Thanks in advance!
>
> --
> - Juwei
>
>
RE: Jobs are still in running state after executing "hadoop job -kill jobId"
Posted by Je...@shell.com.
Um kill -9 "pid" ?
-----Original Message-----
From: Juwei Shi [mailto:shijuwei@gmail.com]
Sent: Friday, July 01, 2011 10:53 AM
To: common-user@hadoop.apache.org; mapreduce-user@hadoop.apache.org
Subject: Jobs are still in running state after executing "hadoop job
-kill jobId"
Hi,
I faced a problem that the jobs are still running after executing
"hadoop
job -kill jobId". I rebooted the cluster but the job still can not be
killed.
The hadoop version is 0.20.2.
Any idea?
Thanks in advance!
--
- Juwei