[jira] [Commented] (TEZ-800) One-one edge with parallelism -1 fails if source vertex parallelism is not -1 as well

["Hitesh Sharma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16342498#comment-16342498 ] 

Hitesh Sharma commented on TEZ-800:
-----------------------------------

I have a case where two 1:1 edges are coming into a vertex. One of the source vertex has parallelism specified (say it is set to 'n') while other doesn't (it is set to -1). Thus DAG::checkAndInferOneToOneParallelism infers the parallelism of the vertex to be 'n'.

The code then below fails to verify the DAG. 

 
{code:java}
// check for inconsistency and errors
for (Edge e : edges) {
  Vertex inputVertex = e.getInputVertex();
  Vertex outputVertex = e.getOutputVertex();
  
  if (e.getEdgeProperty().getDataMovementType() == DataMovementType.ONE_TO_ONE) {
    if (inputVertex.getParallelism() != outputVertex.getParallelism()) {
      // both should be equal or equal to -1.
      if (outputVertex.getParallelism() != -1) {
        throw new TezUncheckedException(
            "1-1 Edge. Destination vertex parallelism must match source vertex. "
            + "Vertex: " + inputVertex.getName() + " does not match vertex: " 
            + outputVertex.getName());
      }
    }
  }
}

{code}
It seems to me that we should be checking for the sourceVertex parallelism to be -1 as well over {color:#59afe1}here{color}.  

 

Will that be okay or some other feature may break?

 

 

> One-one edge with parallelism -1 fails if source vertex parallelism is not -1 as well
> -------------------------------------------------------------------------------------
>
>                 Key: TEZ-800
>                 URL: https://issues.apache.org/jira/browse/TEZ-800
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.3.0
>            Reporter: Rohini Palaniswamy
>            Assignee: Bikas Saha
>            Priority: Major
>             Fix For: 0.5.0
>
>         Attachments: TEZ-800.1.patch, TEZ-800.2.patch
>
>
> // no input initializers. At this moment, only other case is 1-1 edge
>           // with uninitialized sources
>           boolean hasOneToOneUninitedSource = false;
>           for (Map.Entry<Vertex, Edge> entry : vertex.sourceVertices.entrySet()) {
>             if (entry.getValue().getEdgeProperty().getDataMovementType() == 
>                 DataMovementType.ONE_TO_ONE) {
>               if (entry.getKey().getTotalTasks() == -1) {
>                 hasOneToOneUninitedSource = true;
>                 break;
>               }
>             }
>           }
> This checks for the source vertex which has the 1-1 edge to also have -1 parallelism.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)