[jira] [Resolved] (PIG-3876) Handle two outputs from split going to same input in MultiQueryOptimizer

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohini Palaniswamy resolved PIG-3876.
-------------------------------------
    Resolution: Duplicate
      Assignee: Rohini Palaniswamy

  Missed seeing that I had already created this jira and created PIG-4495 again. Tracking the fix in PIG-4495.

> Handle two outputs from split going to same input in MultiQueryOptimizer
> ------------------------------------------------------------------------
>
>                 Key: PIG-3876
>                 URL: https://issues.apache.org/jira/browse/PIG-3876
>             Project: Pig
>          Issue Type: Sub-task
>          Components: tez
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.15.0
>
>
> MultiQueryOptimizerTez.java
> {code}
> // Detect diamond shape, we cannot merge it into split, since Tez
>                 // does not handle double edge between vertexes
>                 boolean sharedSucc = false;
>                 if (getPlan().getSuccessors(successor)!=null) {
>                     for (TezOperator succ_successor : getPlan().getSuccessors(successor)) {
>                         if (succ_successors.contains(succ_successor)) {
>                             sharedSucc = true;
>                             break;
>                         }
>                     }
>                     succ_successors.addAll(getPlan().getSuccessors(successor));
>                 }
> {code}
> SPLIT A INTO B if <condition>, C if <condition>;
> D = JOIN B by x, C by x;
> We would like to do 
> V1 - Split (B -> V2, C -> V2)
> V2 - Join B and C
> Without the check for shared successors, above plan is created but B and C create two separate edges between V1 and V2 which is not supported by Tez. Since the splits are not merged into POSplit fully, we currently have
> V1 - Split ( B-> V3, C-> V2 with just POValueOutputTez)
> V2 -  LocalRearrange and -> V3
> V3 - Join B and C
>  We need to remove the check and merge them into the POSplit and fix this case to make B and C both write to same edge. Being more aggressive in multi-query increases performance.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)