[jira] [Commented] (MAPREDUCE-5196) CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13866815#comment-13866815 ] 

Jason Lowe commented on MAPREDUCE-5196:
---------------------------------------

I recently stumbled across the fact that this removed the ping() method from the umbilical and folded it into statusUpdate.  That's probably OK, but I noticed that it also changed the semantic behavior of what a ping means.  Previously pings would not be considered making progress, but now they are.  That's a problem if a task gets "stuck" where it's still connected to the AM but not really making any progress.  The status updater thread will be happy to continue pinging the AM yet the task has no hope of succeeding.  The progress watchdog in the AM ends up being defeated and is only useful when the task can't contact the AM at all (e.g.: network cut, overall process suspended, or crashed).

Was treating the ping as progress intentional and I'm missing something, or is this a bug?

> CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing 
> ------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5196
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5196
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mr-am, mrv2
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>             Fix For: 3.0.0
>
>         Attachments: MAPREDUCE-5196.1.patch, MAPREDUCE-5196.2.patch, MAPREDUCE-5196.3.patch, MAPREDUCE-5196.patch, MAPREDUCE-5196.patch
>
>
> This JIRA tracks a checkpoint-based AM preemption policy. The policy handles propagation of the preemption requests received from the RM to the appropriate tasks, and bookeeping of checkpoints. Actual checkpointing of the task state is handled in upcoming JIRAs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)