You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org> on 2011/11/03 00:35:32 UTC
[jira] [Updated] (PIG-2313) NPE in ILLUSTRATE trying to get
StatusReporter in STORE
[ https://issues.apache.org/jira/browse/PIG-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2313:
----------------------------
Attachment: PIG-2313-3.patch
PIG-2313-3.patch resync with trunk.
> NPE in ILLUSTRATE trying to get StatusReporter in STORE
> -------------------------------------------------------
>
> Key: PIG-2313
> URL: https://issues.apache.org/jira/browse/PIG-2313
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.10
> Reporter: Doug Daniels
> Attachments: PIG-2313-2.patch, PIG-2313-3.patch, PIG-2313-example.tar.gz, PIG-2313-first_shot.patch
>
>
> I'm seeing an NPE trying to do an illustrate on a script. So far the simplest version of the script that exhibits the issue is:
> {code}
> raw = LOAD 'data.txt' USING PigStorage() AS (x:int, y:int);
> filtered = FILTER raw BY x < 5;
> grouped = GROUP filtered BY x;
> counted = FOREACH grouped
> GENERATE group AS x,
> COUNT(filtered) AS the_count;
> rmf output;
> STORE counted INTO 'output';
> {code}
> I had to pass a few nested Exceptions along to get it, but the bottom stack trace looks like:
> {code}
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.TaskInputOutputContext.getCounter(TaskInputOutputContext.java:88)
> at org.apache.pig.tools.pigstats.PigStatusReporter.getCounter(PigStatusReporter.java:60)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReducePOStoreImpl.createRecordCounter(MapReducePOStoreImpl.java:121)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.setUp(POStore.java:108)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.cleanup(PigGenericMapReduce.java:525)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:178)
> at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:222)
> at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
> at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:275)
> at org.apache.pig.pen.LineageTrimmingVisitor.checkNewBaseData(LineageTrimmingVisitor.java:418)
> ... 21 more
> {code}
> It looks like the IllustratorContext in the hadoop20 PigMapReduce.java shim (line 73) is getting setup with a null StatusReporter. This seems to be for the Reduce phase. On the other hand, the PigMapBase.java sets up the IllustratorContext with an IllustratorDummyReporter for the Map phase.
> Eventually when the code in MapReducePOStoreImpl line 121 tries to get the reporter, it fails with the NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira