You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2018/10/04 20:21:00 UTC
[jira] [Commented] (TEZ-3983) VertexGroup ONE_TO_ONE edges do not
produce the right graph
[ https://issues.apache.org/jira/browse/TEZ-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638815#comment-16638815 ]
Rohini Palaniswamy commented on TEZ-3983:
-----------------------------------------
Would be good to fix this case. Currently we have been switching to a unordered partitioned shuffle edge in Pig ([https://github.com/apache/pig/blob/5b0a0bfaefad0d57c6c9fbbc09d709d13ea9af93/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/UnionOptimizer.java#L635-L642)] for this scenario.
> VertexGroup ONE_TO_ONE edges do not produce the right graph
> -----------------------------------------------------------
>
> Key: TEZ-3983
> URL: https://issues.apache.org/jira/browse/TEZ-3983
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Adrian Nicoara
> Priority: Major
> Attachments: 0001-TEZ-3893.-VertexGroup-ONE_TO_ONE-DAG-verify-patched-.patch
>
>
> A VertexGroup is described to act as a union of multiple vertices.
> Consider a VertexGroup composed of 2 vertices, each with 2 tasks. If this is a union, then the expectation is that the VertexGroup has 4 tasks. A ONE_TO_ONE edge, to a downstream vertex, must then require that vertex to also have 4 tasks.
> This is not the case, as evidenced by the [following test|https://github.com/apache/tez/blob/261bbdd5929d562758deb31085b565db8e92d6a2/tez-api/src/test/java/org/apache/tez/dag/api/TestDAGVerify.java#L779-L831].
> Rather, what happens is that implementation (of unrolling the VertexGroup and directly creating edges between the components of the VertexGroup and the downstream vertices) is presented as the behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)