You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Jing Huang (JIRA)" <ji...@apache.org> on 2009/12/10 01:05:18 UTC

[jira] Created: (PIG-1142) Got NullPointerException merge join with pruning

Got NullPointerException merge join with pruning
------------------------------------------------

                 Key: PIG-1142
                 URL: https://issues.apache.org/jira/browse/PIG-1142
             Project: Pig
          Issue Type: Bug
    Affects Versions: 0.6.0
            Reporter: Jing Huang
             Fix For: 0.7.0


Here is my pig script:
register $zebraJar;
--fs -rmr $outputDir


a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');

sort1 = order a1 by str2;
sort2 = order a2 by str2;

--store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
--store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');

rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();

joina = join rec1 by str2, rec2 by str2 using "merge" ;

E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;

--limitedVals = LIMIT E 5;
--dump limitedVals;

store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');



============
Here is the stacktrace:
java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789215#action_12789215 ] 

Hadoop QA commented on PIG-1142:
--------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12427671/PIG-1142-2.patch
  against trunk revision 889346.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/115/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/115/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/115/console

This message is automatically generated.

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1142-1.patch, PIG-1142-2.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-1142:
----------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Patch committed to both trunk and 0.6 branch.

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1142-1.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788458#action_12788458 ] 

Daniel Dai commented on PIG-1142:
---------------------------------

To make it clear, this issue happens when we prune more than one columns in front of join key.

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.7.0
>
>         Attachments: PIG-1142-1.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-1142:
----------------------------

    Hadoop Flags:   (was: [Reviewed])
          Status: Patch Available  (was: Reopened)

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1142-1.patch, PIG-1142-2.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788415#action_12788415 ] 

Daniel Dai commented on PIG-1142:
---------------------------------

I can reproduce it in regular join as well. Here is the script:
a = LOAD '1.txt' as (a0, a1, a2);
b = LOAD '2.txt' as (b0, b1, b2);
c = join a by a2, b by b2;
d = foreach c generate $0,  $1,  $2;
dump d;

The logical plan is wrong:
{code}
.........
    |---LOJoin 1-17 Schema: {a::a0: bytearray,a::a1: bytearray,a::a2: bytearray,b::b2: bytearray} Type: bag
        |   |
        |   Project 1-15 Projections: [2] Overloaded: false FieldSchema: a2: bytearray Type: bytearray
        |   Input: Load 1-13
        |   |
        |   Project 1-16 Projections: [1] Overloaded: false FieldSchema: Caught Exception: Attempt to fetch field 1 from schema of size 1 Type: Unknown
        |   Input: Load 1-14
        |
        |---Load 1-13 Schema: {a0: bytearray,a1: bytearray,a2: bytearray} Type: bag
        |
        |---Load 1-14 Schema: {b2: bytearray} Type: bag
{code}

The second project of LOJoin should project column 0

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>             Fix For: 0.7.0
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai reassigned PIG-1142:
-------------------------------

    Assignee: Daniel Dai

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.7.0
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788511#action_12788511 ] 

Olga Natkovich commented on PIG-1142:
-------------------------------------

+1. Changes look good.

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.7.0
>
>         Attachments: PIG-1142-1.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-1142:
----------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

New patch committed to both trunk and 0.6 branch.

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1142-1.patch, PIG-1142-2.patch, PIG-1142-3.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-1142:
----------------------------

    Status: Patch Available  (was: Open)

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.7.0
>
>         Attachments: PIG-1142-1.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Jing Huang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788949#action_12788949 ] 

Jing Huang commented on PIG-1142:
---------------------------------

Verified fix on pig 0.6.0 branch. 
Fix works.

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1142-1.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Reopened: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai reopened PIG-1142:
-----------------------------


Find several lines of code share the same nature as the initial one. Should fix them all. 

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1142-1.patch, PIG-1142-2.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789388#action_12789388 ] 

Olga Natkovich commented on PIG-1142:
-------------------------------------

+1, Daniel, code changes look good but do we need to add more unit tests to cover them?

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1142-1.patch, PIG-1142-2.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-1142:
----------------------------

    Attachment: PIG-1142-1.patch

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.7.0
>
>         Attachments: PIG-1142-1.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-1142:
----------------------------

    Attachment: PIG-1142-3.patch

No code change in PIG-1142-3.patch, except for more test cases.

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1142-1.patch, PIG-1142-2.patch, PIG-1142-3.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Olga Natkovich updated PIG-1142:
--------------------------------

    Fix Version/s:     (was: 0.7.0)
                   0.6.0

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1142-1.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789440#action_12789440 ] 

Alan Gates commented on PIG-1142:
---------------------------------

Additional test cases look good.  +1

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1142-1.patch, PIG-1142-2.patch, PIG-1142-3.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-1142:
----------------------------

    Attachment: PIG-1142-2.patch

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1142-1.patch, PIG-1142-2.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-1142) Got NullPointerException merge join with pruning

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788505#action_12788505 ] 

Hadoop QA commented on PIG-1142:
--------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12427539/PIG-1142-1.patch
  against trunk revision 888852.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/110/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/110/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/110/console

This message is automatically generated.

> Got NullPointerException merge join with pruning
> ------------------------------------------------
>
>                 Key: PIG-1142
>                 URL: https://issues.apache.org/jira/browse/PIG-1142
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Jing Huang
>            Assignee: Daniel Dai
>             Fix For: 0.7.0
>
>         Attachments: PIG-1142-1.patch
>
>
> Here is my pig script:
> register $zebraJar;
> --fs -rmr $outputDir
> a1 = LOAD '$inputDir/small1' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> a2 = LOAD '$inputDir/small2' USING org.apache.hadoop.zebra.pig.TableLoader('count,seed,int1,str2');
> sort1 = order a1 by str2;
> sort2 = order a2 by str2;
> --store sort1 into '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> --store sort2 into '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableStorer('[count,seed,int1,str2]');
> rec1 = load '$outputDir/smallsorted11' using org.apache.hadoop.zebra.pig.TableLoader();
> rec2 = load '$outputDir/smallsorted21' using org.apache.hadoop.zebra.pig.TableLoader();
> joina = join rec1 by str2, rec2 by str2 using "merge" ;
> E = foreach joina  generate $0 as count,  $1 as seed,  $2 as int1,  $3 as str2;
> --limitedVals = LIMIT E 5;
> --dump limitedVals;
> store E into '$outputDir/smalljoin2' using org.apache.hadoop.zebra.pig.TableStorer('');
> ============
> Here is the stacktrace:
> java.lang.NullPointerException at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:312) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.extractKeysFromTuple(POMergeJoin.java:464) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNext(POMergeJoin.java:341) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:260) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:237) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:253) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.close(PigMapBase.java:107) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.Child.main(Child.java:159) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.