You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Bikas Saha (JIRA)" <ji...@apache.org> on 2014/03/05 00:05:46 UTC

[jira] [Comment Edited] (TEZ-902) Fetch failure issues in shuffle Input

    [ https://issues.apache.org/jira/browse/TEZ-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920156#comment-13920156 ] 

Bikas Saha edited comment on TEZ-902 at 3/4/14 11:04 PM:
---------------------------------------------------------

I took another look at the code. The logic around remaining inputs is built around tracking what inputs are remaining to be copied. Dont see any place where its being used to cancel a failed input. I think leaving the remaining inputs logic as its stands is fine. Its possible to handle the failed input case by modifying the remaining list but then there would be 2 methods of cancelling bad inputs. 1) via removing them from the remaining list (new impl) 2) via adding them to obsolete inputs (existing impl). IMO, sticking to one method would be cleaner and resilient to errors down the road. Since obsolete inputs are already being used for InputFailed events, we could stick to that.

Hope this clarifies on your observations in the previous comment.

Given this, I would like to keep the patch as is for the following reasons
1) Fix the hung reducer case - Explained above on how failed inputs were not being handled earlier and the patch now handles them via obsoleting them
2) Patch adds a simple check to ShuffleHeader to cleanly identify an error in the input stream. Without this the ShuffleHeader object would produce a null InputAttemptIdentifier that was used in different methods. We are lucky that none of them NPE on the null object. The patch does an early check for this and throws an exception that goes down the input obsolete path
3) De-duplicating input sources and using the latest version - This is cheap to do and a good thing to have. At some point we will have speculation or some other feature/bug that may produce multiple versions of the input. This takes care of that in a defensive way.

Should the AM always invalidate a task attempt's outputs when its killed? Probably it should but for now it does not. We can track that in a separate jira and choose to invalidate on such state changes. However, we should look through the code and consider scenarios where a data movements event may be generated (and transmitted) by an attempt before it succeeds. The current code does not handle it either I think. Opened TEZ-914 to track this.



was (Author: bikassaha):
I took another look at the code. The logic around remaining inputs is built around tracking what inputs are remaining to be copied. Dont see any place where its being used to cancel a failed input. I think leaving the remaining inputs logic as its stands is fine. Its possible to handle the failed input case by modifying the remaining list but then there would be 2 methods of cancelling bad inputs. 1) via removing them from the remaining list (new impl) 2) via adding them to obsolete inputs (existing impl). IMO, sticking to one method would be cleaner and resilient to errors down the road. Since obsolete inputs are already being used for InputFailed events, we could stick to that.

Hope this clarifies on your observations in the previous comment.

Given this, I would like to keep the patch as is for the following reasons
1) Fix the hung reducer case - Explained above on how failed inputs were not being handled earlier and the patch now handles them via obsoleting them
2) Patch adds a simple check to ShuffleHeader to cleanly identify an error in the input stream. Without this the ShuffleHeader object would produce a null InputAttemptIdentifier that was used in different methods. We are lucky that none of them NPE on the null object. The patch does an early check for this and throws an exception that goes down the input obsolete path
3) De-duplicating input sources and using the latest version - This is cheap to do and a good thing to have. At some point we will have speculation or some other feature/bug that may produce multiple versions of the input. This takes care of that in a defensive way.

Should the AM always invalidate a task attempt's outputs when its killed? Probably it should but for now it does not. We can track that in a separate jira and choose to invalidate on such state changes. However, we should look through the code and consider scenarios where a data movements event may be generated (and transmitted) by an attempt before it succeeds. The current code does not handle it either I think.


> Fetch failure issues in shuffle Input
> -------------------------------------
>
>                 Key: TEZ-902
>                 URL: https://issues.apache.org/jira/browse/TEZ-902
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Bikas Saha
>            Assignee: Bikas Saha
>         Attachments: TEZ-902.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)