You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@tez.apache.org by "Saikat (JIRA)" <ji...@apache.org> on 2015/08/17 20:00:46 UTC

[jira] [Commented] (TEZ-2726) Handle invalid number of partitions for SCATTER-GATHER edge

    [ https://issues.apache.org/jira/browse/TEZ-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699942#comment-14699942 ] 

Saikat commented on TEZ-2726:
-----------------------------

Adding [~jlowe] [~rohini] [~jeagles] for watch and comments.

> Handle invalid number of partitions for SCATTER-GATHER edge
> -----------------------------------------------------------
>
>                 Key: TEZ-2726
>                 URL: https://issues.apache.org/jira/browse/TEZ-2726
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Saikat
>            Assignee: Saikat
>
> Encountered an issue where the source vertex has M task and sink vertex has N tasks (N > M), [e.g. M = 1, N = 3]and the edge is of type SCATTER -GATHER.
> This resulted in sink vertex receiving DMEs with non existent targetIds.
> The fetchers for the sink vertex tasks then try to retrieve the map outputs and retrieve invalid headers due to exception in the ShuffleHandler.
> Possible fixes:
> 1. raise proper Tez Exception to indicate this invalid scenario.
> 2. or write appropriate empty partition bits, for the missing partitions before sending out the DMEs to sink vertex. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)