What happens after COMMIT_PENDING?

[Pedro Costa <ps...@gmail.com>]

Hi,

I'm running hadoop map-reduce in clustering, and I've a Reduce Task
that it remains in the state COMMIT_PENDING, and it doesn't finish.
This is happening because I've made some changes to the Hadoop MR. I'm
trying to solve my problem, but I don't understand what's happens
after the COMMIT_PENDING.

I know that when a task is in COMMIT_PENDING, is requesting something
to the TaskTracker or the JobTracker, for the task become SUCCESSFUL.
But, in detail, I don't know nothing. Can anyone explain in more
details what's happening?


Thanks,

-- 
xeon

Re: What happens after COMMIT_PENDING?

[Harsh J <qw...@gmail.com>]

Hello,

On Wed, Mar 9, 2011 at 3:17 PM, Pedro Costa <ps...@gmail.com> wrote:
> I know that when a task is in COMMIT_PENDING, is requesting something
> to the TaskTracker or the JobTracker, for the task become SUCCESSFUL.
> But, in detail, I don't know nothing. Can anyone explain in more
> details what's happening?

Every task sets its state to COMMIT_PENDING upon completion and
requests for a 'commit' status from the JobTracker. If the JT decides
that this task instance may be given a OK-to-commit, it does, else it
kills it. This is useful in speculative execution where multiple tasks
of the same ID run in parallel (only one ought to be committed, the
other(s) killed).

See JobTracker.updateTaskStatuses() and finishing-up parts of the
JobTracker.heartbeat() call that deals with the saving and killing
task actions, to see how COMMIT_PENDING is utilized.

-- 
Harsh J
www.harshj.com