You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Santhosh Srinivasan (JIRA)" <ji...@apache.org> on 2008/07/02 23:20:44 UTC
[jira] Created: (PIG-288) Null pointer exception with load as
schema - Optimizer
Null pointer exception with load as schema - Optimizer
------------------------------------------------------
Key: PIG-288
URL: https://issues.apache.org/jira/browse/PIG-288
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Santhosh Srinivasan
A new test case (testNestedPlan) added to TestEvalPipeline has the following query:
pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
pig.registerQuery("B = group A by $0;");
+ "C1 = filter A by $0 > -1;"
+ "C2 = distinct C1;"
+ "C3 = distinct A;"
+ "generate (int)group;"
+ "};";
Testcase: testNestedPlan took 0.913 sec
Caused an ERROR
Unable to open iterator for alias: C
java.io.IOException: Unable to open iterator for alias: C
at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
at org.apache.pig.PigServer.openIterator(PigServer.java:268)
at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
at org.apache.pig.PigServer.compileLp(PigServer.java:551)
at org.apache.pig.PigServer.execute(PigServer.java:477)
at org.apache.pig.PigServer.openIterator(PigServer.java:259)
... 16 more
Caused by: java.lang.NullPointerException
at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
... 20 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-288) Null pointer exception with load as
schema - Optimizer
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pi Song updated PIG-288:
------------------------
Attachment: PIG_288_OptimizerNPE.patch
The patch!
This is due to the fact that LOCOGroup also maintains internal plan relations. The optimizer also has to fix up the downstream operator's internal.
> Null pointer exception with load as schema - Optimizer
> ------------------------------------------------------
>
> Key: PIG-288
> URL: https://issues.apache.org/jira/browse/PIG-288
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Attachments: PIG_288_OptimizerNPE.patch
>
>
> A new test case (testNestedPlan) added to TestEvalPipeline has the following query:
> pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
> pig.registerQuery("B = group A by $0;");
> + "C1 = filter A by $0 > -1;"
> + "C2 = distinct C1;"
> + "C3 = distinct A;"
> + "generate (int)group;"
> + "};";
> Testcase: testNestedPlan took 0.913 sec
> Caused an ERROR
> Unable to open iterator for alias: C
> java.io.IOException: Unable to open iterator for alias: C
> at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
> at org.apache.pig.PigServer.openIterator(PigServer.java:268)
> at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
> at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
> at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
> at org.apache.pig.PigServer.compileLp(PigServer.java:551)
> at org.apache.pig.PigServer.execute(PigServer.java:477)
> at org.apache.pig.PigServer.openIterator(PigServer.java:259)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
> at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
> at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
> at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
> at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
> ... 20 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-288) Null pointer exception with load as
schema - Optimizer
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pi Song updated PIG-288:
------------------------
Attachment: PIG_288_OptimizerNPE_2.patch
Just found out that the thing I mentioned already exists so moved the COGroup fixing logic to the appropriate place.
The test is still failing though due to problem in JarManager instead now.
> Null pointer exception with load as schema - Optimizer
> ------------------------------------------------------
>
> Key: PIG-288
> URL: https://issues.apache.org/jira/browse/PIG-288
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Attachments: PIG_288_OptimizerNPE.patch, PIG_288_OptimizerNPE_2.patch
>
>
> A new test case (testNestedPlan) added to TestEvalPipeline has the following query:
> pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
> pig.registerQuery("B = group A by $0;");
> + "C1 = filter A by $0 > -1;"
> + "C2 = distinct C1;"
> + "C3 = distinct A;"
> + "generate (int)group;"
> + "};";
> Testcase: testNestedPlan took 0.913 sec
> Caused an ERROR
> Unable to open iterator for alias: C
> java.io.IOException: Unable to open iterator for alias: C
> at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
> at org.apache.pig.PigServer.openIterator(PigServer.java:268)
> at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
> at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
> at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
> at org.apache.pig.PigServer.compileLp(PigServer.java:551)
> at org.apache.pig.PigServer.execute(PigServer.java:477)
> at org.apache.pig.PigServer.openIterator(PigServer.java:259)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
> at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
> at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
> at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
> at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
> ... 20 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-288) Null pointer exception with load as
schema - Optimizer
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pi Song updated PIG-288:
------------------------
Status: Patch Available (was: Open)
> Null pointer exception with load as schema - Optimizer
> ------------------------------------------------------
>
> Key: PIG-288
> URL: https://issues.apache.org/jira/browse/PIG-288
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Attachments: PIG_288_OptimizerNPE.patch
>
>
> A new test case (testNestedPlan) added to TestEvalPipeline has the following query:
> pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
> pig.registerQuery("B = group A by $0;");
> + "C1 = filter A by $0 > -1;"
> + "C2 = distinct C1;"
> + "C3 = distinct A;"
> + "generate (int)group;"
> + "};";
> Testcase: testNestedPlan took 0.913 sec
> Caused an ERROR
> Unable to open iterator for alias: C
> java.io.IOException: Unable to open iterator for alias: C
> at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
> at org.apache.pig.PigServer.openIterator(PigServer.java:268)
> at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
> at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
> at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
> at org.apache.pig.PigServer.compileLp(PigServer.java:551)
> at org.apache.pig.PigServer.execute(PigServer.java:477)
> at org.apache.pig.PigServer.openIterator(PigServer.java:259)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
> at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
> at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
> at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
> at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
> ... 20 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-288) Null pointer exception with load as
schema - Optimizer
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pi Song updated PIG-288:
------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch checked in. Thanks to myself.
> Null pointer exception with load as schema - Optimizer
> ------------------------------------------------------
>
> Key: PIG-288
> URL: https://issues.apache.org/jira/browse/PIG-288
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Attachments: PIG_288_OptimizerNPE.patch, PIG_288_OptimizerNPE_2.patch
>
>
> A new test case (testNestedPlan) added to TestEvalPipeline has the following query:
> pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
> pig.registerQuery("B = group A by $0;");
> + "C1 = filter A by $0 > -1;"
> + "C2 = distinct C1;"
> + "C3 = distinct A;"
> + "generate (int)group;"
> + "};";
> Testcase: testNestedPlan took 0.913 sec
> Caused an ERROR
> Unable to open iterator for alias: C
> java.io.IOException: Unable to open iterator for alias: C
> at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
> at org.apache.pig.PigServer.openIterator(PigServer.java:268)
> at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
> at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
> at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
> at org.apache.pig.PigServer.compileLp(PigServer.java:551)
> at org.apache.pig.PigServer.execute(PigServer.java:477)
> at org.apache.pig.PigServer.openIterator(PigServer.java:259)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
> at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
> at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
> at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
> at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
> ... 20 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-288) Null pointer exception with load as
schema - Optimizer
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610593#action_12610593 ]
Pi Song commented on PIG-288:
-----------------------------
The similar issue can happen in type checker and plan optimizer (future) as well. In the meantime, this patch should work but we will have to come up with a guaranteed safe way to alter the plan.
For example, we may have utility methods for removing/inserting nodes in linear subgraph to start with:-
RemoveNode (a -> b -> c, b) --> (a -> c)
InsertNode(a -> c, b) --> a -> b -> c
> Null pointer exception with load as schema - Optimizer
> ------------------------------------------------------
>
> Key: PIG-288
> URL: https://issues.apache.org/jira/browse/PIG-288
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Santhosh Srinivasan
> Attachments: PIG_288_OptimizerNPE.patch
>
>
> A new test case (testNestedPlan) added to TestEvalPipeline has the following query:
> pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
> pig.registerQuery("B = group A by $0;");
> + "C1 = filter A by $0 > -1;"
> + "C2 = distinct C1;"
> + "C3 = distinct A;"
> + "generate (int)group;"
> + "};";
> Testcase: testNestedPlan took 0.913 sec
> Caused an ERROR
> Unable to open iterator for alias: C
> java.io.IOException: Unable to open iterator for alias: C
> at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
> at org.apache.pig.PigServer.openIterator(PigServer.java:268)
> at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
> at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
> at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
> at org.apache.pig.PigServer.compileLp(PigServer.java:551)
> at org.apache.pig.PigServer.execute(PigServer.java:477)
> at org.apache.pig.PigServer.openIterator(PigServer.java:259)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
> at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
> at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
> at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
> at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
> ... 20 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.