You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Santhosh Srinivasan (JIRA)" <ji...@apache.org> on 2008/07/02 23:20:44 UTC

[jira] Created: (PIG-288) Null pointer exception with load as schema - Optimizer

Null pointer exception with load as schema - Optimizer
------------------------------------------------------

                 Key: PIG-288
                 URL: https://issues.apache.org/jira/browse/PIG-288
             Project: Pig
          Issue Type: Bug
          Components: impl
    Affects Versions: types_branch
            Reporter: Santhosh Srinivasan


A new test case (testNestedPlan) added to TestEvalPipeline has the following query:

        pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
        pig.registerQuery("B = group A by $0;");
        + "C1 = filter A by $0 > -1;"
        + "C2 = distinct C1;"
        + "C3 = distinct A;"
        + "generate (int)group;"
        + "};";

Testcase: testNestedPlan took 0.913 sec
    Caused an ERROR
Unable to open iterator for alias: C
java.io.IOException: Unable to open iterator for alias: C
    at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
    at org.apache.pig.PigServer.openIterator(PigServer.java:268)
    at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
    at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
    at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
    at org.apache.pig.PigServer.compileLp(PigServer.java:551)
    at org.apache.pig.PigServer.execute(PigServer.java:477)
    at org.apache.pig.PigServer.openIterator(PigServer.java:259)
    ... 16 more
Caused by: java.lang.NullPointerException
    at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
    at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
    at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
    at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
    at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
    at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
    at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
    at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
    ... 20 more


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-288) Null pointer exception with load as schema - Optimizer

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pi Song updated PIG-288:
------------------------

    Attachment: PIG_288_OptimizerNPE.patch

The patch!

This is due to the fact that LOCOGroup also maintains internal plan relations. The optimizer also has to fix up the downstream operator's internal.

> Null pointer exception with load as schema - Optimizer
> ------------------------------------------------------
>
>                 Key: PIG-288
>                 URL: https://issues.apache.org/jira/browse/PIG-288
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Santhosh Srinivasan
>         Attachments: PIG_288_OptimizerNPE.patch
>
>
> A new test case (testNestedPlan) added to TestEvalPipeline has the following query:
>         pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
>         pig.registerQuery("B = group A by $0;");
>         + "C1 = filter A by $0 > -1;"
>         + "C2 = distinct C1;"
>         + "C3 = distinct A;"
>         + "generate (int)group;"
>         + "};";
> Testcase: testNestedPlan took 0.913 sec
>     Caused an ERROR
> Unable to open iterator for alias: C
> java.io.IOException: Unable to open iterator for alias: C
>     at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
>     at org.apache.pig.PigServer.openIterator(PigServer.java:268)
>     at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
>     at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
>     at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
>     at org.apache.pig.PigServer.compileLp(PigServer.java:551)
>     at org.apache.pig.PigServer.execute(PigServer.java:477)
>     at org.apache.pig.PigServer.openIterator(PigServer.java:259)
>     ... 16 more
> Caused by: java.lang.NullPointerException
>     at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
>     at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
>     at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
>     at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
>     at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
>     at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
>     at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
>     at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
>     ... 20 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-288) Null pointer exception with load as schema - Optimizer

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pi Song updated PIG-288:
------------------------

    Attachment: PIG_288_OptimizerNPE_2.patch

Just found out that the thing I mentioned already exists so moved the COGroup fixing logic to the appropriate place.
The test is still failing though due to problem in JarManager instead now.

> Null pointer exception with load as schema - Optimizer
> ------------------------------------------------------
>
>                 Key: PIG-288
>                 URL: https://issues.apache.org/jira/browse/PIG-288
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Santhosh Srinivasan
>         Attachments: PIG_288_OptimizerNPE.patch, PIG_288_OptimizerNPE_2.patch
>
>
> A new test case (testNestedPlan) added to TestEvalPipeline has the following query:
>         pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
>         pig.registerQuery("B = group A by $0;");
>         + "C1 = filter A by $0 > -1;"
>         + "C2 = distinct C1;"
>         + "C3 = distinct A;"
>         + "generate (int)group;"
>         + "};";
> Testcase: testNestedPlan took 0.913 sec
>     Caused an ERROR
> Unable to open iterator for alias: C
> java.io.IOException: Unable to open iterator for alias: C
>     at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
>     at org.apache.pig.PigServer.openIterator(PigServer.java:268)
>     at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
>     at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
>     at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
>     at org.apache.pig.PigServer.compileLp(PigServer.java:551)
>     at org.apache.pig.PigServer.execute(PigServer.java:477)
>     at org.apache.pig.PigServer.openIterator(PigServer.java:259)
>     ... 16 more
> Caused by: java.lang.NullPointerException
>     at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
>     at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
>     at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
>     at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
>     at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
>     at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
>     at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
>     at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
>     ... 20 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-288) Null pointer exception with load as schema - Optimizer

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pi Song updated PIG-288:
------------------------

    Status: Patch Available  (was: Open)

> Null pointer exception with load as schema - Optimizer
> ------------------------------------------------------
>
>                 Key: PIG-288
>                 URL: https://issues.apache.org/jira/browse/PIG-288
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Santhosh Srinivasan
>         Attachments: PIG_288_OptimizerNPE.patch
>
>
> A new test case (testNestedPlan) added to TestEvalPipeline has the following query:
>         pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
>         pig.registerQuery("B = group A by $0;");
>         + "C1 = filter A by $0 > -1;"
>         + "C2 = distinct C1;"
>         + "C3 = distinct A;"
>         + "generate (int)group;"
>         + "};";
> Testcase: testNestedPlan took 0.913 sec
>     Caused an ERROR
> Unable to open iterator for alias: C
> java.io.IOException: Unable to open iterator for alias: C
>     at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
>     at org.apache.pig.PigServer.openIterator(PigServer.java:268)
>     at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
>     at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
>     at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
>     at org.apache.pig.PigServer.compileLp(PigServer.java:551)
>     at org.apache.pig.PigServer.execute(PigServer.java:477)
>     at org.apache.pig.PigServer.openIterator(PigServer.java:259)
>     ... 16 more
> Caused by: java.lang.NullPointerException
>     at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
>     at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
>     at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
>     at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
>     at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
>     at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
>     at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
>     at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
>     ... 20 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-288) Null pointer exception with load as schema - Optimizer

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pi Song updated PIG-288:
------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Patch checked in. Thanks to myself.

> Null pointer exception with load as schema - Optimizer
> ------------------------------------------------------
>
>                 Key: PIG-288
>                 URL: https://issues.apache.org/jira/browse/PIG-288
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Santhosh Srinivasan
>         Attachments: PIG_288_OptimizerNPE.patch, PIG_288_OptimizerNPE_2.patch
>
>
> A new test case (testNestedPlan) added to TestEvalPipeline has the following query:
>         pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
>         pig.registerQuery("B = group A by $0;");
>         + "C1 = filter A by $0 > -1;"
>         + "C2 = distinct C1;"
>         + "C3 = distinct A;"
>         + "generate (int)group;"
>         + "};";
> Testcase: testNestedPlan took 0.913 sec
>     Caused an ERROR
> Unable to open iterator for alias: C
> java.io.IOException: Unable to open iterator for alias: C
>     at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
>     at org.apache.pig.PigServer.openIterator(PigServer.java:268)
>     at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
>     at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
>     at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
>     at org.apache.pig.PigServer.compileLp(PigServer.java:551)
>     at org.apache.pig.PigServer.execute(PigServer.java:477)
>     at org.apache.pig.PigServer.openIterator(PigServer.java:259)
>     ... 16 more
> Caused by: java.lang.NullPointerException
>     at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
>     at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
>     at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
>     at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
>     at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
>     at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
>     at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
>     at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
>     ... 20 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-288) Null pointer exception with load as schema - Optimizer

Posted by "Pi Song (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610593#action_12610593 ] 

Pi Song commented on PIG-288:
-----------------------------

The similar issue can happen in type checker and plan optimizer (future) as well. In the meantime, this patch should work but we will have to come up with a guaranteed safe way to alter the plan.

For example, we may have utility methods for removing/inserting nodes in linear subgraph to start with:-

RemoveNode (a -> b -> c,  b)   -->  (a -> c) 

InsertNode(a -> c, b)   -->   a -> b -> c


> Null pointer exception with load as schema - Optimizer
> ------------------------------------------------------
>
>                 Key: PIG-288
>                 URL: https://issues.apache.org/jira/browse/PIG-288
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Santhosh Srinivasan
>         Attachments: PIG_288_OptimizerNPE.patch
>
>
> A new test case (testNestedPlan) added to TestEvalPipeline has the following query:
>         pig.registerQuery("A = LOAD 'file:" + tmpFile + "'as (a:int, b:int);");
>         pig.registerQuery("B = group A by $0;");
>         + "C1 = filter A by $0 > -1;"
>         + "C2 = distinct C1;"
>         + "C3 = distinct A;"
>         + "generate (int)group;"
>         + "};";
> Testcase: testNestedPlan took 0.913 sec
>     Caused an ERROR
> Unable to open iterator for alias: C
> java.io.IOException: Unable to open iterator for alias: C
>     at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:34)
>     at org.apache.pig.PigServer.openIterator(PigServer.java:268)
>     at org.apache.pig.test.TestEvalPipeline.testNestedPlan(TestEvalPipeline.java:376)
> Caused by: org.apache.pig.impl.plan.optimizer.OptimizerException: Unable to insert type casts into plan
>     at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:144)
>     at org.apache.pig.impl.plan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:63)
>     at org.apache.pig.PigServer.compileLp(PigServer.java:551)
>     at org.apache.pig.PigServer.execute(PigServer.java:477)
>     at org.apache.pig.PigServer.openIterator(PigServer.java:259)
>     ... 16 more
> Caused by: java.lang.NullPointerException
>     at org.apache.pig.impl.logicalLayer.LOVisitor.visit(LOVisitor.java:121)
>     at org.apache.pig.impl.logicalLayer.optimizer.SchemaRemover.visit(SchemaRemover.java:65)
>     at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:273)
>     at org.apache.pig.impl.logicalLayer.LOCogroup.visit(LOCogroup.java:37)
>     at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
>     at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
>     at org.apache.pig.impl.logicalLayer.optimizer.LogicalTransformer.rebuildSchemas(LogicalTransformer.java:57)
>     at org.apache.pig.impl.logicalLayer.optimizer.TypeCastInserter.transform(TypeCastInserter.java:141)
>     ... 20 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.