[jira] [Commented] (TEZ-2193) Vertex with shuffle source edge and custom vertex manager should have a predefined parallelism

["Bikas Saha (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357301#comment-14357301 ] 

Bikas Saha commented on TEZ-2193:
---------------------------------

Jeff, have you seen this happen in practice?
It is semantically incorrect to have a shuffle edge without destination tasks defined to a valid value. This is a user error.
There are 2 possible solutions among others.
1) Check for this in shuffle edge manager code at runtime and enforce this. Similar to 1-1 edge manager enforcing equality of source/destination numTasks. A compile time check for this may not be generic enough since any edge manager can be used with a scatter gather edge type. E.g. the user could specify an edge manager that does not depend on the destination numTasks to specify the output partition spec for the source tasks.
2) Add a config knob for the scatter-gather built-in edge manager that allows a user-specified outputSpec value to be used when the destination does not have a value set.

> Vertex with shuffle source edge and custom vertex manager should have a predefined parallelism
> ----------------------------------------------------------------------------------------------
>
>                 Key: TEZ-2193
>                 URL: https://issues.apache.org/jira/browse/TEZ-2193
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jeff Zhang
>            Assignee: Jeff Zhang
>
> e.g. dag has vertices v1, v2 and shuffle edge between them, and v2 has custom vertex manager and -1 parallelism. In this case v1's output spec may be has -1 physical edge which will cause task hangs in TezChild.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)