[jira] [Updated] (PIG-5082) Tez UnionOptimizer creates vertex group with one member

["Travis Woodruff (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-5082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Travis Woodruff updated PIG-5082:
---------------------------------
    Attachment: PIG-5082.patch

Here's a patch that works around the issue. A better fix would probably be to detect the bad vertex group and remove it from the plan, but that's above my skill level.

> Tez UnionOptimizer creates vertex group with one member
> -------------------------------------------------------
>
>                 Key: PIG-5082
>                 URL: https://issues.apache.org/jira/browse/PIG-5082
>             Project: Pig
>          Issue Type: Bug
>          Components: tez
>    Affects Versions: 0.16.0
>            Reporter: Travis Woodruff
>            Priority: Minor
>         Attachments: PIG-5082.patch
>
>
> This script results in a vertex group with one member:
> {code}
> a = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (x:chararray);
> b = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (x:chararray);
> c = LOAD '/tmp/empty.txt' USING PigStorage('\t') AS (y:chararray);
> u1 = UNION ONSCHEMA a, b;
> SPLIT u1 INTO r IF x != '', s OTHERWISE;
> d = JOIN r BY x LEFT, c BY y;
> u2 = UNION ONSCHEMA d, s;
> e = FILTER u2 BY x == '';
> f = FILTER u2 BY x == 'm';
> u3 = UNION ONSCHEMA e, f;
> DUMP u3;
> {code}
> Which results in:
> {code}
> java.lang.IllegalArgumentException: VertexGroup must have at least 2 members
> 	at org.apache.tez.dag.api.VertexGroup.<init>(VertexGroup.java:77)
> 	at org.apache.tez.dag.api.DAG.createVertexGroup(DAG.java:202)
> 	at org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:396)
> 	at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:255)
> ...
> {code}
> This seems to be happening because {{UnionOptimizer}} is replacing a union with a vertex group and then optimizing away a predecessor union thus removing a node and resulting in a vertex group with one member.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)