[jira] [Created] (PIG-5309) Problem with tez + union + replicated join

["Will Oberman (JIRA)" <ji...@apache.org>]

Will Oberman created PIG-5309:
---------------------------------

             Summary: Problem with tez + union + replicated join
                 Key: PIG-5309
                 URL: https://issues.apache.org/jira/browse/PIG-5309
             Project: Pig
          Issue Type: Bug
          Components: tez
    Affects Versions: 0.17.0
            Reporter: Will Oberman
            Priority: Minor


I've been using Pig 0.12.1 for quite some time and am finally upgrading to 0.17.  One of my existing scripts failed.  I have a workaround (SET pig.tez.opt.union false), but I thought I'd pass on the problem I observed.  

In stdout: 
{noformat}
ERROR 2017: Internal error creating job configuration.
{noformat}

In the Pig log:
{noformat}
Caused by: java.lang.IllegalArgumentException: Edge [scope-93 : org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] -> [scope-83 : org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] ({ BROADCAST : org.apache.tez.runtime.library.input.UnorderedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.UnorderedKVOutput >> NullEdgeManager }) already defined!
	at org.apache.tez.dag.api.DAG.addEdge(DAG.java:272)
	at org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:404)
	at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:259)
	at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56)
	at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:87)
	at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46)
	at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.buildDAG(TezJobCompiler.java:69)
	at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:120)
	... 20 more
{noformat}

I played around with a minimum viable test script and can cause this to fail:
{noformat}
weblogs = LOAD '/tmp/in/weblogInfo' as (path:chararray, queryMap:map[chararray]); 
featureToExtraData = LOAD '/tmp/in/featureToExtraData' as (feature:chararray, extraData:chararray); 

oldA = FILTER weblogs BY path == '/A';
newA = FILTER weblogs BY path == '/somethingElse';
B = FILTER weblogs BY path == '/B';

oldAFeatures = FOREACH oldA GENERATE queryMap#'feature1' as feature1, queryMap#'feature2' as feature2;
newAFeatures = FOREACH newA GENERATE queryMap#'different1' as feature1, queryMap#'different2' as feature2;
AFeatures = UNION oldAFeatures, newAFeatures;
AFeaturesPlusMore = JOIN AFeatures BY feature1 LEFT, featureToExtraData BY feature USING 'replicated';

BFeatures = FOREACH B GENERATE queryMap#'somethingElseEntirely1' as feature1, queryMap#'somethingElseEntirely2' as feature2;
BFeaturesPlusMore = JOIN BFeatures BY feature1 LEFT, featureToExtraData BY feature USING 'replicated';

STORE AFeaturesPlusMore INTO '/tmp/out/1/AFeaturesPlusMore';
STORE BFeaturesPlusMore INTO '/tmp/out/1/BFeaturesPlusMore';
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)