You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Harish Mallipeddi <ha...@gmail.com> on 2009/08/03 14:50:14 UTC
Difference between "Killed Task Attempts" and "Killed Tasks"
Hi,
Anyone can tell me what's the difference between "Killed Task Attempts" and
"Killed Tasks"? I ran a big job (14820 maps and 0 reduces). In the
job-details page, the web GUI reports 62 "killed task attempts". I'm
assuming this is due to "speculative execution". Now when I go to the
job-history page for the job, it reports 54 "killed tasks" (and 14820
successful map-tasks as expected).
A few questions:
* Why 62 killed task attempts vs 54 killed tasks?
* Under speculative execution, does hadoop launch a new MapTask with new
task-id or does it just launch a new MapTaskAttempt with a new
task-attempt-id?
* When a MapTaskAttempt fails, and when hadoop tries to re-launch the
MapTask, does it create a new task-id or just a new task-attempt-id?
* Does 'mapred.map.max.attempts' include all attempts launched due to
speculative-execution?
Btw this job is basically a trivial no-op job - it just scans around 1TB of
data and does nothing else in the map. I looked at the killed tasks' syslog
output and I didn't see any errors.
--
Harish Mallipeddi
http://blog.poundbang.in
Re: Difference between "Killed Task Attempts" and "Killed Tasks"
Posted by Jason Venner <ja...@gmail.com>.
You only get the killed tasks, when speculative execution is enabled, when
one of a pair of identical tasks are running, finishes, the other task is
killed, but it is not considered an attempt.
On Mon, Aug 3, 2009 at 6:18 AM, Harish Mallipeddi <
harish.mallipeddi@gmail.com> wrote:
> Agreed. But how did I manage to get 54 killed tasks vs 62 killed
> task-attempts? I understand what a "failed task" is (a task for which
> 'mapred.map.max.attempts' attempts have failed). But what's a "killed
> task"?
>
> On Mon, Aug 3, 2009 at 6:41 PM, Enis Soztutar <en...@gmail.com> wrote:
>
> > Hi,
> >
> > Task attempt is an attempt to a task. At any given time, one or
> > more(speculative exec.) of task attempts can be running. For a task,
> there
> > can be many attempts at different nodes. A task is complete if any of its
> > attempts is complete. For a task to be marked as failed all of
> > mapred.map.max.attempts should fail. For every task in the job, a TaskID
> is
> > assigned. For every attempt, a TaskAttemptID is assigned (which ends with
> > _0, _1, etc).
> >
> >
> > Harish Mallipeddi wrote:
> >
> >> Hi,
> >>
> >> Anyone can tell me what's the difference between "Killed Task Attempts"
> >> and
> >> "Killed Tasks"? I ran a big job (14820 maps and 0 reduces). In the
> >> job-details page, the web GUI reports 62 "killed task attempts". I'm
> >> assuming this is due to "speculative execution". Now when I go to the
> >> job-history page for the job, it reports 54 "killed tasks" (and 14820
> >> successful map-tasks as expected).
> >>
> >> A few questions:
> >>
> >> * Why 62 killed task attempts vs 54 killed tasks?
> >> * Under speculative execution, does hadoop launch a new MapTask with new
> >> task-id or does it just launch a new MapTaskAttempt with a new
> >> task-attempt-id?
> >> * When a MapTaskAttempt fails, and when hadoop tries to re-launch the
> >> MapTask, does it create a new task-id or just a new task-attempt-id?
> >> * Does 'mapred.map.max.attempts' include all attempts launched due to
> >> speculative-execution?
> >>
> >> Btw this job is basically a trivial no-op job - it just scans around 1TB
> >> of
> >> data and does nothing else in the map. I looked at the killed tasks'
> >> syslog
> >> output and I didn't see any errors.
> >>
> >>
> >>
> >
> >
>
>
> --
> Harish Mallipeddi
> http://blog.poundbang.in
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals
Re: Difference between "Killed Task Attempts" and "Killed Tasks"
Posted by Harish Mallipeddi <ha...@gmail.com>.
Agreed. But how did I manage to get 54 killed tasks vs 62 killed
task-attempts? I understand what a "failed task" is (a task for which
'mapred.map.max.attempts' attempts have failed). But what's a "killed task"?
On Mon, Aug 3, 2009 at 6:41 PM, Enis Soztutar <en...@gmail.com> wrote:
> Hi,
>
> Task attempt is an attempt to a task. At any given time, one or
> more(speculative exec.) of task attempts can be running. For a task, there
> can be many attempts at different nodes. A task is complete if any of its
> attempts is complete. For a task to be marked as failed all of
> mapred.map.max.attempts should fail. For every task in the job, a TaskID is
> assigned. For every attempt, a TaskAttemptID is assigned (which ends with
> _0, _1, etc).
>
>
> Harish Mallipeddi wrote:
>
>> Hi,
>>
>> Anyone can tell me what's the difference between "Killed Task Attempts"
>> and
>> "Killed Tasks"? I ran a big job (14820 maps and 0 reduces). In the
>> job-details page, the web GUI reports 62 "killed task attempts". I'm
>> assuming this is due to "speculative execution". Now when I go to the
>> job-history page for the job, it reports 54 "killed tasks" (and 14820
>> successful map-tasks as expected).
>>
>> A few questions:
>>
>> * Why 62 killed task attempts vs 54 killed tasks?
>> * Under speculative execution, does hadoop launch a new MapTask with new
>> task-id or does it just launch a new MapTaskAttempt with a new
>> task-attempt-id?
>> * When a MapTaskAttempt fails, and when hadoop tries to re-launch the
>> MapTask, does it create a new task-id or just a new task-attempt-id?
>> * Does 'mapred.map.max.attempts' include all attempts launched due to
>> speculative-execution?
>>
>> Btw this job is basically a trivial no-op job - it just scans around 1TB
>> of
>> data and does nothing else in the map. I looked at the killed tasks'
>> syslog
>> output and I didn't see any errors.
>>
>>
>>
>
>
--
Harish Mallipeddi
http://blog.poundbang.in
Re: Difference between "Killed Task Attempts" and "Killed Tasks"
Posted by Enis Soztutar <en...@gmail.com>.
Hi,
Task attempt is an attempt to a task. At any given time, one or
more(speculative exec.) of task attempts can be running. For a task,
there can be many attempts at different nodes. A task is complete if any
of its attempts is complete. For a task to be marked as failed all of
mapred.map.max.attempts should fail. For every task in the job, a TaskID
is assigned. For every attempt, a TaskAttemptID is assigned (which ends
with _0, _1, etc).
Harish Mallipeddi wrote:
> Hi,
>
> Anyone can tell me what's the difference between "Killed Task Attempts" and
> "Killed Tasks"? I ran a big job (14820 maps and 0 reduces). In the
> job-details page, the web GUI reports 62 "killed task attempts". I'm
> assuming this is due to "speculative execution". Now when I go to the
> job-history page for the job, it reports 54 "killed tasks" (and 14820
> successful map-tasks as expected).
>
> A few questions:
>
> * Why 62 killed task attempts vs 54 killed tasks?
> * Under speculative execution, does hadoop launch a new MapTask with new
> task-id or does it just launch a new MapTaskAttempt with a new
> task-attempt-id?
> * When a MapTaskAttempt fails, and when hadoop tries to re-launch the
> MapTask, does it create a new task-id or just a new task-attempt-id?
> * Does 'mapred.map.max.attempts' include all attempts launched due to
> speculative-execution?
>
> Btw this job is basically a trivial no-op job - it just scans around 1TB of
> data and does nothing else in the map. I looked at the killed tasks' syslog
> output and I didn't see any errors.
>
>