[jira] [Commented] (TEZ-1642) TestAMRecovery sometimes fail

["Jeff Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230843#comment-14230843 ] 

Jeff Zhang commented on TEZ-1642:
---------------------------------

bq. Is the intent to fail the vertex after 1 task is finished but not the second? This can be achieved by only controlling when these tasks get scheduled. We can stop deriving from the input ready vertex manager and control the scheduling in a custom manner. It can schedule task 1 but not task 2. Then wait for task 1 to finish. Then die. This way the vertex is partially finished. Then recovered AM schedules both tasks.
It can only guarantee that taks_1 is done, but can not guarantee that task_2 is running, The purpose of this test case is to make task_1 done while task_2 is running, and verify that task_1 is not re-run but task_2 is rerun when recover.
The purpose I make VM get notification from the task/taskattempt completed is that only in the main AsyncDispatcher the status of DAG/Vertex/Task/TaskAttempt is determined (state change will block on the main AsyncDispatcher). 
Or I can loose the test case scenario requirement, only guarantee that task_1 is done but not guarantee the state of task_2

> TestAMRecovery sometimes fail
> -----------------------------
>
>                 Key: TEZ-1642
>                 URL: https://issues.apache.org/jira/browse/TEZ-1642
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jeff Zhang
>            Assignee: Jeff Zhang
>         Attachments: TEZ-1642-2.patch, TEZ-1642-3.patch, TEZ-1642-4.patch, TEZ-1642.patch
>
>
> TestAMRecovery fails sometimes on testVertexPartiallyFinished_XXX.  
> The scenario is that we'd like kill AM when vertex is partially finished ( with 2 tasks, task_0 is finished and task_1 is running). When in recovery, task_0 should not rerun and task_1 should rerun. ( We use the recovery log(TaskAttemptFinishedEvent) to judge whether task is rerun)
> Currently, using VertexManager.onSourceTaskCompleted to control when to kill AM, but it is not perfect.  VertexManager.onSourceTaskCompleted is not invoked at the moment task attempt is finished ( TaskAttempt send event to Task to tell TaskAttempt is finsihed, and then Task send event to Vertex to trigger VM.onSourceTaskCompleted) 
> The following case is possible: task_0 finished -> task_1 finished -> VM.onSourceTaskCompleted -> VM.onSourceTaskCompleted
> In this case, we will take it as partially completed in the first VM.onSourceTaskCompleted, but actually the vertex is fully completed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)