You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Pradeep Kamath (JIRA)" <ji...@apache.org> on 2008/09/24 03:01:47 UTC
[jira] Created: (PIG-454) group by followed by group ALL causes
error in reduce
group by followed by group ALL causes error in reduce
-----------------------------------------------------
Key: PIG-454
URL: https://issues.apache.org/jira/browse/PIG-454
Project: Pig
Issue Type: Bug
Affects Versions: types_branch
Reporter: Pradeep Kamath
Fix For: types_branch
Script:
{code}
a = load 'st10k' as (name, age, gpa);
b = group a by name;
c = foreach b generate flatten(group), COUNT(a) as cnt;
d = group c all;
e = foreach d generate AVG(c.cnt);
dump e;
{code}
Error:
{noformat}
2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
...
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-454) group by followed by group ALL causes
error in reduce
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-454:
---------------------------
Attachment: PIG-454.patch
CombinerOptimizer is a visitor that walks the entire plan of MapReduceOpers. It was not resetting state as it visited each operator, causing it to get confused on the key to set in the combiner in cases where there were multiple ops that could use the combiner.
> group by followed by group ALL causes error in reduce
> -----------------------------------------------------
>
> Key: PIG-454
> URL: https://issues.apache.org/jira/browse/PIG-454
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Reporter: Pradeep Kamath
> Assignee: Alan Gates
> Fix For: types_branch
>
> Attachments: PIG-454.patch
>
>
> Script:
> {code}
> a = load 'st10k' as (name, age, gpa);
> b = group a by name;
> c = foreach b generate flatten(group), COUNT(a) as cnt;
> d = group c all;
> e = foreach d generate AVG(c.cnt);
> dump e;
> {code}
> Error:
> {noformat}
> 2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
> 2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
> at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-454) group by followed by group ALL causes
error in reduce
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich reassigned PIG-454:
----------------------------------
Assignee: Alan Gates
> group by followed by group ALL causes error in reduce
> -----------------------------------------------------
>
> Key: PIG-454
> URL: https://issues.apache.org/jira/browse/PIG-454
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Reporter: Pradeep Kamath
> Assignee: Alan Gates
> Fix For: types_branch
>
>
> Script:
> {code}
> a = load 'st10k' as (name, age, gpa);
> b = group a by name;
> c = foreach b generate flatten(group), COUNT(a) as cnt;
> d = group c all;
> e = foreach d generate AVG(c.cnt);
> dump e;
> {code}
> Error:
> {noformat}
> 2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
> 2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
> at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-454) group by followed by group ALL causes
error in reduce
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634353#action_12634353 ]
Olga Natkovich commented on PIG-454:
------------------------------------
+1
> group by followed by group ALL causes error in reduce
> -----------------------------------------------------
>
> Key: PIG-454
> URL: https://issues.apache.org/jira/browse/PIG-454
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Reporter: Pradeep Kamath
> Assignee: Alan Gates
> Fix For: types_branch
>
> Attachments: PIG-454.patch
>
>
> Script:
> {code}
> a = load 'st10k' as (name, age, gpa);
> b = group a by name;
> c = foreach b generate flatten(group), COUNT(a) as cnt;
> d = group c all;
> e = foreach d generate AVG(c.cnt);
> dump e;
> {code}
> Error:
> {noformat}
> 2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
> 2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
> at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-454) group by followed by group ALL causes
error in reduce
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-454:
---------------------------
Status: Patch Available (was: Open)
> group by followed by group ALL causes error in reduce
> -----------------------------------------------------
>
> Key: PIG-454
> URL: https://issues.apache.org/jira/browse/PIG-454
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Reporter: Pradeep Kamath
> Assignee: Alan Gates
> Fix For: types_branch
>
> Attachments: PIG-454.patch
>
>
> Script:
> {code}
> a = load 'st10k' as (name, age, gpa);
> b = group a by name;
> c = foreach b generate flatten(group), COUNT(a) as cnt;
> d = group c all;
> e = foreach d generate AVG(c.cnt);
> dump e;
> {code}
> Error:
> {noformat}
> 2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
> 2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
> at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-454) group by followed by group ALL causes
error in reduce
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-454:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
patch is in
> group by followed by group ALL causes error in reduce
> -----------------------------------------------------
>
> Key: PIG-454
> URL: https://issues.apache.org/jira/browse/PIG-454
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Reporter: Pradeep Kamath
> Assignee: Alan Gates
> Fix For: types_branch
>
> Attachments: PIG-454.patch
>
>
> Script:
> {code}
> a = load 'st10k' as (name, age, gpa);
> b = group a by name;
> c = foreach b generate flatten(group), COUNT(a) as cnt;
> d = group c all;
> e = foreach d generate AVG(c.cnt);
> dump e;
> {code}
> Error:
> {noformat}
> 2008-09-23 17:58:12,002 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job failed!
> 2008-09-23 17:58:12,004 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0117_m_000000java.io.IOException: wrong key class: org.apache.pig.impl.io.NullableTuple is not class org.apache.pig.impl.io.NullableText
> at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:995)
> at org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1079)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:155)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:56)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:872)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:779)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:691)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.