[jira] [Commented] (MAPREDUCE-5196) CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing

["Wangda Tan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016357#comment-14016357 ] 

Wangda Tan commented on MAPREDUCE-5196:
---------------------------------------


Hi [~curino], 
While trying to understand this part of change, I've several questions, hope you could share some idea of it,

my understand of the workflow, please forgive my current ignorance of this code,
1) CheckpointAMPreemptionPolicy will keep track of which containers (task-attempt) need to be preempted
2) TaskAttemptListener will set AMFeedback.preempted to true, when AMPreemptionPolicy.isPreempted(TaskID) returns true
3) Task get AMFeedback, and set mustPreempt. Task takes some action to do some checkpoints, etc. And call umbilical.preempted(taskId, taskStatus)
But a question here, I found the Task do nothing except printing some logs after mustPreempt set, is it expected behavior? 
If it's expected behavior, the task will keep running until it completed or killed by NM, resource cannot be proactively released by this task. I think we should call umbilical.preempted when we found mustPreempt is true, correct?

Another question is, I found in Task.java
{code}
  public void done(TaskUmbilicalProtocol umbilical,
                   TaskReporter reporter
                   ) throws IOException, InterruptedException {
    updateCounters();
    if (taskStatus.getRunState() == TaskStatus.State.PREEMPTED ) {
      // If we are preempted, do no output promotion; signal done and exit
      committer.commitTask(taskContext);
      umbilical.preempted(taskId, taskStatus);
      taskDone.set(true);
      reporter.stopCommunicationThread();
      return;
    }
    ...
  }
{code}
It relies on taskStatus.getRunState() == PREEMPTED, but I found nobody set taskStatus.runState to PREEMPTED. Could you please tell me which part of code set taskStatus. If nobody sets runState to PREEMPTED, it cannot invoke umbilical.preempted properly.

Thanks,
Wangda


> CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing 
> ------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5196
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5196
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mr-am, mrv2
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>             Fix For: 3.0.0
>
>         Attachments: MAPREDUCE-5196.1.patch, MAPREDUCE-5196.2.patch, MAPREDUCE-5196.3.patch, MAPREDUCE-5196.patch, MAPREDUCE-5196.patch
>
>
> This JIRA tracks a checkpoint-based AM preemption policy. The policy handles propagation of the preemption requests received from the RM to the appropriate tasks, and bookeeping of checkpoints. Actual checkpointing of the task state is handled in upcoming JIRAs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)