Which component manages slow tasks?

[Merto Mertek <ma...@gmail.com>]

I am interested in which component of hadoop and how is managed slow task
execution.

Is this a tasktracker, jobtracker or a scheduler?

In fairscheduling (0.20.205) there exist a preemption mechanism but it just
handles min and fairshares and kills last submited tips..

Thanks.

Re: Which component manages slow tasks?

[Merto Mertek <ma...@gmail.com>]

Perfect, thank you Harsh, method that I was looking for was
markUnresponsiveTasks..

So in general if I sum up my understanding of fairscheduling is as follow:

- Pools assignments and ordering is done at tasktrackers requests and is
based primarily on how far is the pool bellow its minshare, afterwards on
runningTasks/weight. Fairness is provided by the preemption mechanism which
occurs every 15sec and checks the exact fair share that should be allocated
to every pool (Fairscheduler#preemptTasksIfNecessary()).

- After the pool a job is chosen based on the internal pool scheduling
policy (fair or fifo-by-arriving-time)

- Task assignment from a chosen job is based on task's data and the
tasktracker that is asking for a new task. Task  assignments cannot be
manipulated by the schedulers and the assignment procedure is the same for
all schedulers (first try to assign non running tasks, then local  and at
last speculative tasks).  Type of task locality is defined by delay
scheduling algorithm.

- Preemption mechanism for slow task is done on the tasktracker
(TaskTracker#markUnresponsiveTasks).


Other thing that I am just wondering is why fairsharing calculation
(SchedulingAlgorithms#computeFairShare) is not taken into account in the
sorting of pools at every tasktracker request? If I am not wrong the time
complexity of finding a right slotToWeight ratio R is O(log(R)*jobs). Is
maybe this the reason for not including it in every tasktracker request?


On 9 June 2012 15:38, Harsh J <ha...@cloudera.com> wrote:

> Merto,
>
> You seem to be looking for the speculative execution code?
>
> For MR1, in 1.x or 0.20.x and below, you can find that in the
> JobInProgress#findNewMapTask method (go down to the bit which does
> speculative execution logic for maps and reduces depending on if the
> job enabled it for itself).
>
> For MR2, in 2.x and above, since it offers a way to have pluggable
> speculative algorithms, the default implementation (a slightly
> advanced version of the one in 1.0) is available in the
> DefaultSpeculator class (under the mapreduce-client-app package). This
> is used by the MRAppMaster (and is event driven).
>
> If you are instead asking about the task timeout monitor (that kills
> tasks that haven't reported for a period):
>
> For MR1 it is in TaskTracker#markUnresponsiveTasks method (that uses
> timeout values set per task (from a job)).
>
> For MR2 similar code is in TaskHeartbeatHandler#PingChecker thread
> class. This is again in the mapreduce-client-app package.
>
> Preemption is a different operation than monitoring slow jobs/tasks.
>
> I believe you should be able to, with any decent IDE, follow the above
> entry points and then be able to drill up/down from there to analyze
> the code. In case you get stuck somewhere in understanding something
> around these features, feel free to ask us here!
>
> On Fri, Jun 8, 2012 at 9:14 PM, Merto Mertek <ma...@gmail.com> wrote:
> > I am interested in which component of hadoop and how is managed slow task
> > execution.
> >
> > Is this a tasktracker, jobtracker or a scheduler?
> >
> > In fairscheduling (0.20.205) there exist a preemption mechanism but it
> just
> > handles min and fairshares and kills last submited tips..
> >
> > Thanks.
>
>
>
> --
> Harsh J
>

Re: Which component manages slow tasks?

[Harsh J <ha...@cloudera.com>]

Merto,

You seem to be looking for the speculative execution code?

For MR1, in 1.x or 0.20.x and below, you can find that in the
JobInProgress#findNewMapTask method (go down to the bit which does
speculative execution logic for maps and reduces depending on if the
job enabled it for itself).

For MR2, in 2.x and above, since it offers a way to have pluggable
speculative algorithms, the default implementation (a slightly
advanced version of the one in 1.0) is available in the
DefaultSpeculator class (under the mapreduce-client-app package). This
is used by the MRAppMaster (and is event driven).

If you are instead asking about the task timeout monitor (that kills
tasks that haven't reported for a period):

For MR1 it is in TaskTracker#markUnresponsiveTasks method (that uses
timeout values set per task (from a job)).

For MR2 similar code is in TaskHeartbeatHandler#PingChecker thread
class. This is again in the mapreduce-client-app package.

Preemption is a different operation than monitoring slow jobs/tasks.

I believe you should be able to, with any decent IDE, follow the above
entry points and then be able to drill up/down from there to analyze
the code. In case you get stuck somewhere in understanding something
around these features, feel free to ask us here!

On Fri, Jun 8, 2012 at 9:14 PM, Merto Mertek <ma...@gmail.com> wrote:
> I am interested in which component of hadoop and how is managed slow task
> execution.
>
> Is this a tasktracker, jobtracker or a scheduler?
>
> In fairscheduling (0.20.205) there exist a preemption mechanism but it just
> handles min and fairshares and kills last submited tips..
>
> Thanks.



-- 
Harsh J