You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Doug Daniels (Created) (JIRA)" <ji...@apache.org> on 2011/10/03 19:41:34 UTC

[jira] [Created] (PIG-2313) NPE in ILLUSTRATE trying to get StatusReporter in STORE

NPE in ILLUSTRATE trying to get StatusReporter in STORE
-------------------------------------------------------

                 Key: PIG-2313
                 URL: https://issues.apache.org/jira/browse/PIG-2313
             Project: Pig
          Issue Type: Bug
    Affects Versions: 0.9.0, 0.10
            Reporter: Doug Daniels


I'm seeing an NPE trying to do an illustrate on a script.  So far the simplest version of the script that exhibits the issue is:

{code}
raw = LOAD 'data.txt' USING PigStorage() AS (x:int, y:int);
filtered = FILTER raw BY x < 5;
grouped = GROUP filtered BY x;
counted = FOREACH grouped
         GENERATE group AS x,
                  COUNT(filtered) AS the_count;
rmf output;
STORE counted INTO 'output';
{code}

I had to pass a few nested Exceptions along to get it, but the bottom stack trace looks like:

{code}
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88)
	at org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:60)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createRecordCounter(MapReducePOStoreImpl.java:121)
	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:108)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.cleanup(PigGenericMapReduce.java:525)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:222)
	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:275)
	at org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:418)
	... 21 more
{code}

It looks like the IllustratorContext in the hadoop20 PigMapReduce.java shim (line 73) is getting setup with a null StatusReporter.  This seems to be for the Reduce phase.  On the other hand, the PigMapBase.java sets up the IllustratorContext with an IllustratorDummyReporter for the Map phase.

Eventually when the code in MapReducePOStoreImpl line 121 tries to get the reporter, it fails with the NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2313) NPE in ILLUSTRATE trying to get StatusReporter in STORE

Posted by "Doug Daniels (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119528#comment-13119528 ] 

Doug Daniels commented on PIG-2313:
-----------------------------------

I created a patch that passes an IllustratorDummyReporter into the IlustratorContext in hadoop20 PigMapReduce.Reduce.  This fixes the NPE, but yields a new Exception about being unable to setup the store function:

{code}
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 0: Exception: Unable to setup the store function.
	at org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:420)
	at org.apache.pig.pen.LineageTrimmingVisitor.processLoad(LineageTrimmingVisitor.java:374)
	at org.apache.pig.pen.LineageTrimmingVisitor.processOperator(LineageTrimmingVisitor.java:438)
	... 19 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2081: Unable to setup the store function.
	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:111)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.cleanup(PigGenericMapReduce.java:525)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:222)
	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:260)
	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:278)
	at org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:418)
	... 21 more
Caused by: java.io.IOException: File already exists:file:/Users/ddaniels/code/sandbox/jira/output_new/_temporary/_attempt__0000_r_000000_0/part-r-00000
	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:228)
	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:368)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:465)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:372)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextOutputFormat.getRecordWriter(PigTextOutputFormat.java:98)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createStoreFunc(MapReducePOStoreImpl.java:85)
	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:103)
	... 27 more
================================================================================
{code}

Does anyone understand what would cause that?  The Exception above is from running it in local mode, but the same problem happens in MapReduce mode (I checked that the output dir does not exist too).

Also, I tried to add a test in TestExampleGenerator.java in the patch, but it complains "Internal error. Did not find roots in the physical plan."  Is there something special that needs to be done to get the store to appear in the plan?
                
> NPE in ILLUSTRATE trying to get StatusReporter in STORE
> -------------------------------------------------------
>
>                 Key: PIG-2313
>                 URL: https://issues.apache.org/jira/browse/PIG-2313
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.9.0, 0.10
>            Reporter: Doug Daniels
>         Attachments: PIG-2313-example.tar.gz, PIG-2313-first_shot.patch
>
>
> I'm seeing an NPE trying to do an illustrate on a script.  So far the simplest version of the script that exhibits the issue is:
> {code}
> raw = LOAD 'data.txt' USING PigStorage() AS (x:int, y:int);
> filtered = FILTER raw BY x < 5;
> grouped = GROUP filtered BY x;
> counted = FOREACH grouped
>          GENERATE group AS x,
>                   COUNT(filtered) AS the_count;
> rmf output;
> STORE counted INTO 'output';
> {code}
> I had to pass a few nested Exceptions along to get it, but the bottom stack trace looks like:
> {code}
> Caused by: java.lang.NullPointerException
> 	at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88)
> 	at org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:60)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createRecordCounter(MapReducePOStoreImpl.java:121)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:108)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.cleanup(PigGenericMapReduce.java:525)
> 	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
> 	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:222)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:275)
> 	at org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:418)
> 	... 21 more
> {code}
> It looks like the IllustratorContext in the hadoop20 PigMapReduce.java shim (line 73) is getting setup with a null StatusReporter.  This seems to be for the Reduce phase.  On the other hand, the PigMapBase.java sets up the IllustratorContext with an IllustratorDummyReporter for the Map phase.
> Eventually when the code in MapReducePOStoreImpl line 121 tries to get the reporter, it fails with the NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2313) NPE in ILLUSTRATE trying to get StatusReporter in STORE

Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2313:
----------------------------

    Attachment: PIG-2313-3.patch

PIG-2313-3.patch resync with trunk.
                
> NPE in ILLUSTRATE trying to get StatusReporter in STORE
> -------------------------------------------------------
>
>                 Key: PIG-2313
>                 URL: https://issues.apache.org/jira/browse/PIG-2313
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.9.0, 0.10
>            Reporter: Doug Daniels
>         Attachments: PIG-2313-2.patch, PIG-2313-3.patch, PIG-2313-example.tar.gz, PIG-2313-first_shot.patch
>
>
> I'm seeing an NPE trying to do an illustrate on a script.  So far the simplest version of the script that exhibits the issue is:
> {code}
> raw = LOAD 'data.txt' USING PigStorage() AS (x:int, y:int);
> filtered = FILTER raw BY x < 5;
> grouped = GROUP filtered BY x;
> counted = FOREACH grouped
>          GENERATE group AS x,
>                   COUNT(filtered) AS the_count;
> rmf output;
> STORE counted INTO 'output';
> {code}
> I had to pass a few nested Exceptions along to get it, but the bottom stack trace looks like:
> {code}
> Caused by: java.lang.NullPointerException
> 	at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88)
> 	at org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:60)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createRecordCounter(MapReducePOStoreImpl.java:121)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:108)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.cleanup(PigGenericMapReduce.java:525)
> 	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
> 	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:222)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:275)
> 	at org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:418)
> 	... 21 more
> {code}
> It looks like the IllustratorContext in the hadoop20 PigMapReduce.java shim (line 73) is getting setup with a null StatusReporter.  This seems to be for the Reduce phase.  On the other hand, the PigMapBase.java sets up the IllustratorContext with an IllustratorDummyReporter for the Map phase.
> Eventually when the code in MapReducePOStoreImpl line 121 tries to get the reporter, it fails with the NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2313) NPE in ILLUSTRATE trying to get StatusReporter in STORE

Posted by "Thejas M Nair (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141738#comment-13141738 ] 

Thejas M Nair commented on PIG-2313:
------------------------------------

+1
                
> NPE in ILLUSTRATE trying to get StatusReporter in STORE
> -------------------------------------------------------
>
>                 Key: PIG-2313
>                 URL: https://issues.apache.org/jira/browse/PIG-2313
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.9.0, 0.10
>            Reporter: Doug Daniels
>         Attachments: PIG-2313-2.patch, PIG-2313-example.tar.gz, PIG-2313-first_shot.patch
>
>
> I'm seeing an NPE trying to do an illustrate on a script.  So far the simplest version of the script that exhibits the issue is:
> {code}
> raw = LOAD 'data.txt' USING PigStorage() AS (x:int, y:int);
> filtered = FILTER raw BY x < 5;
> grouped = GROUP filtered BY x;
> counted = FOREACH grouped
>          GENERATE group AS x,
>                   COUNT(filtered) AS the_count;
> rmf output;
> STORE counted INTO 'output';
> {code}
> I had to pass a few nested Exceptions along to get it, but the bottom stack trace looks like:
> {code}
> Caused by: java.lang.NullPointerException
> 	at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88)
> 	at org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:60)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createRecordCounter(MapReducePOStoreImpl.java:121)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:108)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.cleanup(PigGenericMapReduce.java:525)
> 	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
> 	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:222)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:275)
> 	at org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:418)
> 	... 21 more
> {code}
> It looks like the IllustratorContext in the hadoop20 PigMapReduce.java shim (line 73) is getting setup with a null StatusReporter.  This seems to be for the Reduce phase.  On the other hand, the PigMapBase.java sets up the IllustratorContext with an IllustratorDummyReporter for the Map phase.
> Eventually when the code in MapReducePOStoreImpl line 121 tries to get the reporter, it fails with the NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2313) NPE in ILLUSTRATE trying to get StatusReporter in STORE

Posted by "Doug Daniels (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Daniels updated PIG-2313:
------------------------------

    Attachment: PIG-2313-first_shot.patch
                PIG-2313-example.tar.gz
    
> NPE in ILLUSTRATE trying to get StatusReporter in STORE
> -------------------------------------------------------
>
>                 Key: PIG-2313
>                 URL: https://issues.apache.org/jira/browse/PIG-2313
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.9.0, 0.10
>            Reporter: Doug Daniels
>         Attachments: PIG-2313-example.tar.gz, PIG-2313-first_shot.patch
>
>
> I'm seeing an NPE trying to do an illustrate on a script.  So far the simplest version of the script that exhibits the issue is:
> {code}
> raw = LOAD 'data.txt' USING PigStorage() AS (x:int, y:int);
> filtered = FILTER raw BY x < 5;
> grouped = GROUP filtered BY x;
> counted = FOREACH grouped
>          GENERATE group AS x,
>                   COUNT(filtered) AS the_count;
> rmf output;
> STORE counted INTO 'output';
> {code}
> I had to pass a few nested Exceptions along to get it, but the bottom stack trace looks like:
> {code}
> Caused by: java.lang.NullPointerException
> 	at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88)
> 	at org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:60)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createRecordCounter(MapReducePOStoreImpl.java:121)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:108)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.cleanup(PigGenericMapReduce.java:525)
> 	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
> 	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:222)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:275)
> 	at org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:418)
> 	... 21 more
> {code}
> It looks like the IllustratorContext in the hadoop20 PigMapReduce.java shim (line 73) is getting setup with a null StatusReporter.  This seems to be for the Reduce phase.  On the other hand, the PigMapBase.java sets up the IllustratorContext with an IllustratorDummyReporter for the Map phase.
> Eventually when the code in MapReducePOStoreImpl line 121 tries to get the reporter, it fails with the NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Resolved] (PIG-2313) NPE in ILLUSTRATE trying to get StatusReporter in STORE

Posted by "Daniel Dai (Resolved) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai resolved PIG-2313.
-----------------------------

       Resolution: Fixed
    Fix Version/s: 0.11
                   0.10
         Assignee: Daniel Dai
     Hadoop Flags: Reviewed

Patch committed to both trunk and 0.10 branch.
                
> NPE in ILLUSTRATE trying to get StatusReporter in STORE
> -------------------------------------------------------
>
>                 Key: PIG-2313
>                 URL: https://issues.apache.org/jira/browse/PIG-2313
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.9.0, 0.10
>            Reporter: Doug Daniels
>            Assignee: Daniel Dai
>             Fix For: 0.10, 0.11
>
>         Attachments: PIG-2313-2.patch, PIG-2313-3.patch, PIG-2313-example.tar.gz, PIG-2313-first_shot.patch
>
>
> I'm seeing an NPE trying to do an illustrate on a script.  So far the simplest version of the script that exhibits the issue is:
> {code}
> raw = LOAD 'data.txt' USING PigStorage() AS (x:int, y:int);
> filtered = FILTER raw BY x < 5;
> grouped = GROUP filtered BY x;
> counted = FOREACH grouped
>          GENERATE group AS x,
>                   COUNT(filtered) AS the_count;
> rmf output;
> STORE counted INTO 'output';
> {code}
> I had to pass a few nested Exceptions along to get it, but the bottom stack trace looks like:
> {code}
> Caused by: java.lang.NullPointerException
> 	at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88)
> 	at org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:60)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createRecordCounter(MapReducePOStoreImpl.java:121)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:108)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.cleanup(PigGenericMapReduce.java:525)
> 	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
> 	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:222)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:275)
> 	at org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:418)
> 	... 21 more
> {code}
> It looks like the IllustratorContext in the hadoop20 PigMapReduce.java shim (line 73) is getting setup with a null StatusReporter.  This seems to be for the Reduce phase.  On the other hand, the PigMapBase.java sets up the IllustratorContext with an IllustratorDummyReporter for the Map phase.
> Eventually when the code in MapReducePOStoreImpl line 121 tries to get the reporter, it fails with the NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2313) NPE in ILLUSTRATE trying to get StatusReporter in STORE

Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-2313:
----------------------------

    Attachment: PIG-2313-2.patch

Hi, Daniels,
Your fix for Reporter is right. We didn't pass reporter to reducer in illustrate. I attached PIG-2313-2.patch also fix the "store func setup" issue. I include your test case in patch.
                
> NPE in ILLUSTRATE trying to get StatusReporter in STORE
> -------------------------------------------------------
>
>                 Key: PIG-2313
>                 URL: https://issues.apache.org/jira/browse/PIG-2313
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.9.0, 0.10
>            Reporter: Doug Daniels
>         Attachments: PIG-2313-2.patch, PIG-2313-example.tar.gz, PIG-2313-first_shot.patch
>
>
> I'm seeing an NPE trying to do an illustrate on a script.  So far the simplest version of the script that exhibits the issue is:
> {code}
> raw = LOAD 'data.txt' USING PigStorage() AS (x:int, y:int);
> filtered = FILTER raw BY x < 5;
> grouped = GROUP filtered BY x;
> counted = FOREACH grouped
>          GENERATE group AS x,
>                   COUNT(filtered) AS the_count;
> rmf output;
> STORE counted INTO 'output';
> {code}
> I had to pass a few nested Exceptions along to get it, but the bottom stack trace looks like:
> {code}
> Caused by: java.lang.NullPointerException
> 	at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88)
> 	at org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:60)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createRecordCounter(MapReducePOStoreImpl.java:121)
> 	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:108)
> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.cleanup(PigGenericMapReduce.java:525)
> 	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
> 	at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:222)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> 	at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:275)
> 	at org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:418)
> 	... 21 more
> {code}
> It looks like the IllustratorContext in the hadoop20 PigMapReduce.java shim (line 73) is getting setup with a null StatusReporter.  This seems to be for the Reduce phase.  On the other hand, the PigMapBase.java sets up the IllustratorContext with an IllustratorDummyReporter for the Map phase.
> Eventually when the code in MapReducePOStoreImpl line 121 tries to get the reporter, it fails with the NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira