You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Pedro Costa <ps...@gmail.com> on 2011/06/08 18:40:04 UTC
cleanup task doesn't run always
Hi,
After I run the command "bin/hadoop job -history /temp/history/", I've
got these 2 task summary. In one of them, it run a cleanup task and in
the other hasn't run the cleanup task. This means that a cleanup task
doesn't run always. So, when a cleanup task should run?
Task Summary
============================
Kind Total Successful Failed Killed StartTime FinishTime
Setup 1 1 0 0 8-Jun-2011 15:54:50
8-Jun-2011 15:54:51 (0sec)
Map 600 600 0 0 8-Jun-2011 15:54:54
8-Jun-2011 15:57:05 (2mins, 11sec)
Reduce 252 250 0 2 8-Jun-2011 15:55:06
8-Jun-2011 15:57:48 (2mins, 42sec)
Cleanup 0 0 0 0
============================
Task Summary
============================
Kind Total Successful Failed Killed StartTime FinishTime
Setup 1 1 0 0 8-Jun-2011 16:04:19
8-Jun-2011 16:04:20 (1sec)
Map 500 383 0 117 8-Jun-2011 16:04:40
Reduce 0 0 0 0
Cleanup 1 1 0 0 8-Jun-2011 16:04:37
8-Jun-2011 16:04:38 (1sec)
============================
Thanks,
Re: cleanup task doesn't run always
Posted by Pedro Costa <ps...@gmail.com>.
Sorry, but what's the answer from Aaron? It's not in this mail.
And, I think that the cleanup task has nothing to do with HDFS. As far
as I can see, the hadoop contains a setup and a clean up task for map
and reduce tasks. I don't understand why, but it has. Here's the part
of the code that I'm talking about:
[code]
// cleanup map tip. This map doesn't use any splits. Just assign an empty
// split.
JobClient.RawSplit emptySplit = new JobClient.RawSplit();
cleanup[0] = new TaskInProgress(jobId, jobFile, emptySplit,
jobtracker, conf, this, numMapTasks);
cleanup[0].setJobCleanupTask();
// cleanup reduce tip.
cleanup[1] = new TaskInProgress(jobId, jobFile, numMapTasks,
numReduceTasks, jobtracker, conf, this);
cleanup[1].setJobCleanupTask();
// create two setup tips, one map and one reduce.
setup = new TaskInProgress[2];
// setup map tip. This map doesn't use any split. Just assign an empty
// split.
setup[0] = new TaskInProgress(jobId, jobFile, emptySplit,
jobtracker, conf, this, numMapTasks + 1 );
setup[0].setJobSetupTask();
// setup reduce tip.
setup[1] = new TaskInProgress(jobId, jobFile, numMapTasks,
numReduceTasks + 1, jobtracker, conf, this);
setup[1].setJobSetupTask();
[/code]
If anyone could explain also what's the purpose of these tasks I would
be very appreciated?
Thanks,
On Thu, Jun 9, 2011 at 9:05 AM, Laurent Hatier <la...@gmail.com> wrote:
> Oh i don't see that it was in the HDFS. Aaron has answered i think
>
> 2011/6/9 Laurent Hatier <la...@gmail.com>
>>
>> Have you try to restart your hadoop node ? (or all hadoop node). When you
>> go to restart, the namenode go to format the directory that you have choosen
>>
>> 2011/6/8 Pedro Costa <ps...@gmail.com>
>>>
>>> Hi,
>>>
>>> After I run the command "bin/hadoop job -history /temp/history/", I've
>>> got these 2 task summary. In one of them, it run a cleanup task and in
>>> the other hasn't run the cleanup task. This means that a cleanup task
>>> doesn't run always. So, when a cleanup task should run?
>>>
>>> Task Summary
>>> ============================
>>> Kind Total Successful Failed Killed StartTime
>>> FinishTime
>>>
>>> Setup 1 1 0 0 8-Jun-2011 15:54:50
>>> 8-Jun-2011 15:54:51 (0sec)
>>> Map 600 600 0 0 8-Jun-2011 15:54:54
>>> 8-Jun-2011 15:57:05 (2mins, 11sec)
>>> Reduce 252 250 0 2 8-Jun-2011 15:55:06
>>> 8-Jun-2011 15:57:48 (2mins, 42sec)
>>> Cleanup 0 0 0 0
>>> ============================
>>>
>>>
>>>
>>> Task Summary
>>> ============================
>>> Kind Total Successful Failed Killed StartTime
>>> FinishTime
>>>
>>> Setup 1 1 0 0 8-Jun-2011 16:04:19
>>> 8-Jun-2011 16:04:20 (1sec)
>>> Map 500 383 0 117 8-Jun-2011
>>> 16:04:40
>>> Reduce 0 0 0 0
>>> Cleanup 1 1 0 0 8-Jun-2011 16:04:37
>>> 8-Jun-2011 16:04:38 (1sec)
>>> ============================
>>>
>>> Thanks,
>>
>>
>>
>> --
>> Laurent HATIER
>> Étudiant en 2e année du Cycle Ingénieur à l'EISTI
>
>
>
> --
> Laurent HATIER
> Étudiant en 2e année du Cycle Ingénieur à l'EISTI
>
Re: cleanup task doesn't run always
Posted by Laurent Hatier <la...@gmail.com>.
Oh i don't see that it was in the HDFS. Aaron has answered i think
2011/6/9 Laurent Hatier <la...@gmail.com>
> Have you try to restart your hadoop node ? (or all hadoop node). When you
> go to restart, the namenode go to format the directory that you have choosen
>
>
> 2011/6/8 Pedro Costa <ps...@gmail.com>
>
>> Hi,
>>
>> After I run the command "bin/hadoop job -history /temp/history/", I've
>> got these 2 task summary. In one of them, it run a cleanup task and in
>> the other hasn't run the cleanup task. This means that a cleanup task
>> doesn't run always. So, when a cleanup task should run?
>>
>> Task Summary
>> ============================
>> Kind Total Successful Failed Killed StartTime FinishTime
>>
>> Setup 1 1 0 0 8-Jun-2011 15:54:50
>> 8-Jun-2011 15:54:51 (0sec)
>> Map 600 600 0 0 8-Jun-2011 15:54:54
>> 8-Jun-2011 15:57:05 (2mins, 11sec)
>> Reduce 252 250 0 2 8-Jun-2011 15:55:06
>> 8-Jun-2011 15:57:48 (2mins, 42sec)
>> Cleanup 0 0 0 0
>> ============================
>>
>>
>>
>> Task Summary
>> ============================
>> Kind Total Successful Failed Killed StartTime FinishTime
>>
>> Setup 1 1 0 0 8-Jun-2011 16:04:19
>> 8-Jun-2011 16:04:20 (1sec)
>> Map 500 383 0 117 8-Jun-2011
>> 16:04:40
>> Reduce 0 0 0 0
>> Cleanup 1 1 0 0 8-Jun-2011 16:04:37
>> 8-Jun-2011 16:04:38 (1sec)
>> ============================
>>
>> Thanks,
>>
>
>
>
> --
> Laurent HATIER
> Étudiant en 2e année du Cycle Ingénieur à l'EISTI
>
--
Laurent HATIER
Étudiant en 2e année du Cycle Ingénieur à l'EISTI
Re: cleanup task doesn't run always
Posted by Laurent Hatier <la...@gmail.com>.
Have you try to restart your hadoop node ? (or all hadoop node). When you go
to restart, the namenode go to format the directory that you have choosen
2011/6/8 Pedro Costa <ps...@gmail.com>
> Hi,
>
> After I run the command "bin/hadoop job -history /temp/history/", I've
> got these 2 task summary. In one of them, it run a cleanup task and in
> the other hasn't run the cleanup task. This means that a cleanup task
> doesn't run always. So, when a cleanup task should run?
>
> Task Summary
> ============================
> Kind Total Successful Failed Killed StartTime FinishTime
>
> Setup 1 1 0 0 8-Jun-2011 15:54:50
> 8-Jun-2011 15:54:51 (0sec)
> Map 600 600 0 0 8-Jun-2011 15:54:54
> 8-Jun-2011 15:57:05 (2mins, 11sec)
> Reduce 252 250 0 2 8-Jun-2011 15:55:06
> 8-Jun-2011 15:57:48 (2mins, 42sec)
> Cleanup 0 0 0 0
> ============================
>
>
>
> Task Summary
> ============================
> Kind Total Successful Failed Killed StartTime FinishTime
>
> Setup 1 1 0 0 8-Jun-2011 16:04:19
> 8-Jun-2011 16:04:20 (1sec)
> Map 500 383 0 117 8-Jun-2011 16:04:40
> Reduce 0 0 0 0
> Cleanup 1 1 0 0 8-Jun-2011 16:04:37
> 8-Jun-2011 16:04:38 (1sec)
> ============================
>
> Thanks,
>
--
Laurent HATIER
Étudiant en 2e année du Cycle Ingénieur à l'EISTI