[jira] [Commented] (TEZ-2249) Wait for all task attempt finished before moving Task to finished state

["Jonathan Eagles (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16835691#comment-16835691 ] 

Jonathan Eagles commented on TEZ-2249:
--------------------------------------

Looked at MapReduce for a similar feature, but there is none. It is susceptible to the same race condition. I have seen this occur recently and the outcome can be bad since temporary directory (and presumably files) can show up after the vertex stage commits. If subsequent stages are triggered based on a SUCCESS file being written, this can cause issues and contents change after the SUCCESS marker is created (a '_SUCCESS' file).

If there is still interest, I could help work on this patch (giving [~zjffdu] proper credit) as assignee isn't able to work on this.

> Wait for all task attempt finished before moving Task to finished state
> -----------------------------------------------------------------------
>
>                 Key: TEZ-2249
>                 URL: https://issues.apache.org/jira/browse/TEZ-2249
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jeff Zhang
>            Assignee: Jeff Zhang
>            Priority: Major
>         Attachments: TEZ-2249-1.patch
>
>
> 2 cases:
> * If Task needs to move the SUCCEEDED, then committing may happens while there's still task attempt running.
> * If Tasks needs to move to FAILED/KILLED/ERROD, then aborting may happens while there's still task attempt running.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)