You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Pradeep Kamath (JIRA)" <ji...@apache.org> on 2008/10/24 00:26:46 UTC
[jira] Created: (PIG-508) Query with a cogroup have one of its
inputs coming from a group fails
Query with a cogroup have one of its inputs coming from a group fails
---------------------------------------------------------------------
Key: PIG-508
URL: https://issues.apache.org/jira/browse/PIG-508
Project: Pig
Issue Type: Bug
Affects Versions: types_branch
Reporter: Pradeep Kamath
Assignee: Pradeep Kamath
Fix For: types_branch
Script which fails:
{code}
a = load '/user/pig/tests/data/singlefile/studenttab10k';
b = group a by $0;
c = load '/user/pig/tests/data/singlefile/studenttab10k';
d = cogroup b by $0, c by $0;
e = foreach d generate group, c.$1, SUM(c.$1), COUNT(c);
dump e;
{code}
Error message produced:
{noformat}
08/10/23 15:23:54 ERROR mapReduceLayer.MapReduceLauncher: Job failed!
08/10/23 15:23:54 ERROR mapReduceLayer.Launcher: Error message from task (reduce) task_200810231521_0007_r_000000java.lang.NullPointerException
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage.getNext(POPackage.java:218)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:208)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-508) Query with a cogroup have one of its
inputs coming from a group fails
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-508:
-------------------------------
Status: Patch Available (was: Open)
> Query with a cogroup have one of its inputs coming from a group fails
> ---------------------------------------------------------------------
>
> Key: PIG-508
> URL: https://issues.apache.org/jira/browse/PIG-508
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: types_branch
>
>
> Script which fails:
> {code}
> a = load '/user/pig/tests/data/singlefile/studenttab10k';
> b = group a by $0;
> c = load '/user/pig/tests/data/singlefile/studenttab10k';
> d = cogroup b by $0, c by $0;
> e = foreach d generate group, c.$1, SUM(c.$1), COUNT(c);
> dump e;
> {code}
> Error message produced:
> {noformat}
> 08/10/23 15:23:54 ERROR mapReduceLayer.MapReduceLauncher: Job failed!
> 08/10/23 15:23:54 ERROR mapReduceLayer.Launcher: Error message from task (reduce) task_200810231521_0007_r_000000java.lang.NullPointerException
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage.getNext(POPackage.java:218)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:208)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:134)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-508) Query with a cogroup have one of its
inputs coming from a group fails
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-508:
-------------------------------
Attachment: PIG-508.patch
Attached patch - details on the fix:
POPackageAnnotator is a visitor which looks for "POPackage" and annotates it with "keyInfo" from each of the LocalRearranges which provide input to the POPackage. The keyinfo essentially has information about what part of the "value" for a given input is present in the "key" and hence ommitted from the "value". The visitor was incorrectly assuming that if a local rearrange corresponding to the package is found in the given MROper's map plan, then the annotation is done. This breaks in the case of the script in this issue - the POPackage has one of its Local rearranges in the map plan of the same MROper as the POPackage and the other local rearrange in the reduce plan of the predecessor MROper. Hence the visitor was changed to ensure that POPackage is annotated with information from *all* Local rearranges.
> Query with a cogroup have one of its inputs coming from a group fails
> ---------------------------------------------------------------------
>
> Key: PIG-508
> URL: https://issues.apache.org/jira/browse/PIG-508
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: types_branch
>
> Attachments: PIG-508.patch
>
>
> Script which fails:
> {code}
> a = load '/user/pig/tests/data/singlefile/studenttab10k';
> b = group a by $0;
> c = load '/user/pig/tests/data/singlefile/studenttab10k';
> d = cogroup b by $0, c by $0;
> e = foreach d generate group, c.$1, SUM(c.$1), COUNT(c);
> dump e;
> {code}
> Error message produced:
> {noformat}
> 08/10/23 15:23:54 ERROR mapReduceLayer.MapReduceLauncher: Job failed!
> 08/10/23 15:23:54 ERROR mapReduceLayer.Launcher: Error message from task (reduce) task_200810231521_0007_r_000000java.lang.NullPointerException
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage.getNext(POPackage.java:218)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:208)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:134)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-508) Query with a cogroup have one of its
inputs coming from a group fails
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-508:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
patch committed; thanks Pradeep
> Query with a cogroup have one of its inputs coming from a group fails
> ---------------------------------------------------------------------
>
> Key: PIG-508
> URL: https://issues.apache.org/jira/browse/PIG-508
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: types_branch
>
> Attachments: PIG-508.patch
>
>
> Script which fails:
> {code}
> a = load '/user/pig/tests/data/singlefile/studenttab10k';
> b = group a by $0;
> c = load '/user/pig/tests/data/singlefile/studenttab10k';
> d = cogroup b by $0, c by $0;
> e = foreach d generate group, c.$1, SUM(c.$1), COUNT(c);
> dump e;
> {code}
> Error message produced:
> {noformat}
> 08/10/23 15:23:54 ERROR mapReduceLayer.MapReduceLauncher: Job failed!
> 08/10/23 15:23:54 ERROR mapReduceLayer.Launcher: Error message from task (reduce) task_200810231521_0007_r_000000java.lang.NullPointerException
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPackage.getNext(POPackage.java:218)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:208)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:134)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.