[jira] [Commented] (MAPREDUCE-5196) CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing

["Carlo Curino (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646183#comment-13646183 ] 

Carlo Curino commented on MAPREDUCE-5196:
-----------------------------------------

The attached patch presents a checkpoint-based policy. This patch captures all the changes in the AM, and in the  TaskUmbilicalProtocol needed to support checkpoint-based task preemption. 
The CheckpointAMPreemptionPolicy is designed to do the following:
1) ignores the request for preemption of maps (as maps are typically short-run and not worth preempting). 
2) it propagates requests for premption for reducers, by looking up the taskAttemptId corresponding to the container we are asked to preempt
3) the policy also manages CheckpointIDs (i.e., references to the location of the checkpoint) on behalf of individual tasks (on restart a task ask whether it should start from scratch or form an existing checkpoint).

The umbilical protocol has been changed so that tasks will receive feedback from the AM whether they should preempt. This happens both on ping and statusUpdate (folded now in a single message type). 

We also enhance the TaskAttemptImpl state machine to include the notion of preemption.

The  
                
> CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing 
> ------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5196
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5196
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mr-am, mrv2
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: MAPREDUCE-5196.patch
>
>
> This JIRA tracks a checkpoint-based AM preemption policy. The policy handles propagation of the preemption requests received from the RM to the appropriate tasks, and bookeeping of checkpoints. Actual checkpointing of the task state is handled in upcoming JIRAs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira