Re: Review Request: UDF in nested plan results frontend exception

[Daniel Dai <da...@gmail.com>]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/439/
-----------------------------------------------------------

(Updated 2011-02-22 09:58:48.484463)


Review request for pig and Richard Ding.


Summary (updated)
-------

The below is my script :

register myanotherudf.jar;
A = load 'myinput' using PigStorage() as ( date:chararray,bcookie:chararray,count:int,avg:double,pvs:int);
B = foreach A generate (int)(avg / 100.0) * 100   as avg, pvs;
C = group B by ( avg );
D = foreach C {
        Pvs = order B by pvs;
        Const = org.vivek.MyAnotherUDF(Pvs.pvs).(count,sum);
        generate Const.sum as sum;
        };
store D into 'out_D';

The script is failing during compilation of the plan. The usage of the udf inside the foreach is causing the problem. The udf implements algebraic and the
output schema is also defined.
The below is the exception that I get :

ERROR 2042: Error in new logical plan. Try -Dpig.usenewlogicalplan=false.

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2042: Error in new logical plan. Try -Dpig.usenewlogicalplan=false.
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:309)
at org.apache.pig.PigServer.compilePp(PigServer.java:1364)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1206)
at org.apache.pig.PigServer.execute(PigServer.java:1200)
at org.apache.pig.PigServer.access$100(PigServer.java:128)
at org.apache.pig.PigServer$Graph.execute(PigServer.java:1527)
at org.apache.pig.PigServer.executeBatchEx(PigServer.java:372)
at org.apache.pig.PigServer.executeBatch(PigServer.java:339)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:500)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.lang.NullPointerException
at org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:105)
at org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:229)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:94)
at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:71)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:261)
... 13 more


This addresses bug PIG-1858.
    https://issues.apache.org/jira/browse/PIG-1858


Diffs
-----

  http://svn.apache.org/repos/asf/pig/branches/branch-0.8/src/org/apache/pig/impl/logicalLayer/parser/QueryParser.jjt 1072544 
  http://svn.apache.org/repos/asf/pig/branches/branch-0.8/test/org/apache/pig/test/TestEvalPipeline2.java 1072544 

Diff: https://reviews.apache.org/r/439/diff


Testing
-------

test-patch:
     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

unit test:
    all pass

end to end test:
    all pass


Thanks,

Daniel