You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amar Kamat (JIRA)" <ji...@apache.org> on 2009/02/24 11:56:01 UTC

[jira] Created: (HADOOP-5319) Inconsistency in handling lost trackers upon jobtracker restart

Inconsistency in handling lost trackers upon jobtracker restart
---------------------------------------------------------------

                 Key: HADOOP-5319
                 URL: https://issues.apache.org/jira/browse/HADOOP-5319
             Project: Hadoop Core
          Issue Type: Bug
          Components: mapred
            Reporter: Amar Kamat


If a tasktracker is lost, the jobtracker kills all the tasks that were successful on that tracker and re-executes it somewhere else. In-memory datastructures are all cleared up for the lost tracker. Now if the jobtracker restarts, the new jobtracker has no clue about the trackers that were lost and hence if the lost tracker join back, they will be accepted and all the tasks on those tracker will join back. Following are the issues
- If the running task on the lost tracker is killed, its cleanup attempt will be launched. Now the new jobtracker has no idea about this attempt. Also the lost tracker can join back and hence there are 2 attempts that are running with the same id, one which can move the tip to success and other which moves the tip to killed state.
- Ideally, the lost tracker should be asked to re-init which wont happen now.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5319) Inconsistency in handling lost trackers upon jobtracker restart

Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676294#action_12676294 ] 

Amar Kamat commented on HADOOP-5319:
------------------------------------

As of now there are 2 ways to achieve this
# Create a file for every tracker in _system-dir_. Upon a (re)join, simply create the file and upon a lost tracker, simply delete the file (as the first step before changing jobtracker's data structures). So, upon restart, there will be 2 lists, one obtained from the history and one from the _system-dir_. Lose all the tracker that have file missing in the _system-dir_. The trackers should be lost before opening up the jobtracker for the trackers. Note that the file will be created in memory (of namenode) and will be part of the heartbeat (once every tracker [re] join) while the deletes will be happening in a thread.
Benefits :  No sync required i.e faster per tracker update
Drawbacks : Too many files 
 
# Maintain in a file, a list currently available tracker  with the jobtracker. Assume that the jobtracker waits for X (say mapred.tasktracker.expiry.interval) units of time before creating the first file. After which on every tracker update (join/delete) the tracker-file is updated ( a new file will be written and then renamed to the old filename). Note that the file operation is not just in memory and requires writing the file. Hence this solution requires the access has to be batched up and not inline with heartbeat (i.e a thread which does this writing). 
Benefits : Less files
Drawbacks : Every update requires writing the current list of tracker to the file and replacing the old with new.

Both these solution (should) guarantee that the tracker that were lost in the old jobtracker should be lost in the new tracker. Thoughts?

> Inconsistency in handling lost trackers upon jobtracker restart
> ---------------------------------------------------------------
>
>                 Key: HADOOP-5319
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5319
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Amar Kamat
>
> If a tasktracker is lost, the jobtracker kills all the tasks that were successful on that tracker and re-executes it somewhere else. In-memory datastructures are all cleared up for the lost tracker. Now if the jobtracker restarts, the new jobtracker has no clue about the trackers that were lost and hence if the lost tracker join back, they will be accepted and all the tasks on those tracker will join back. Following are the issues
> - If the running task on the lost tracker is killed, its cleanup attempt will be launched. Now the new jobtracker has no idea about this attempt. Also the lost tracker can join back and hence there are 2 attempts that are running with the same id, one which can move the tip to success and other which moves the tip to killed state.
> - Ideally, the lost tracker should be asked to re-init which wont happen now.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.