You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Viraj Bhat (JIRA)" <ji...@apache.org> on 2008/11/18 05:23:44 UTC
[jira] Updated: (PIG-537) Failure in Hadoop map collect stage due
to type mismatch in the keys used in cogroup
[ https://issues.apache.org/jira/browse/PIG-537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-537:
---------------------------
Description:
Consider the following pig query, which demonstrates various problems during the Logical Plan creation and the subsequent execution of the M/R job. In this query we do two cogroups, one between A and B to generate an alias ABtemptable. Then we again cogroup A with ABtemptable based on marks which was read in as an int.
==================================================================================
{code}
A = load 'mymarks.txt' as (username:chararray,marks:int);
B = load 'mygrades.txt' as (username:chararray,grade:chararray);
ABtemp = cogroup A by username, B by username;
ABtemptable = foreach ABtemp generate
group as username,
flatten(A.marks) as newmarks;
--describe ABtemptable;
C = cogroup A by marks, ABtemptable by newmarks;
--describe C;
explain C;
dump C;
{code}
==================================================================================
The schema for C and ABtemptable which pig reports:
==================================================================================
{code}describe ABtemptable;{code} ABtemptable: {username: chararray,newmarks: int}
{code}describe C;{code} C: {group: int,A: {username: chararray,marks: int},ABtemptable: {username: chararray,newmarks: int}}
==================================================================================
If you run the above query you get the following error:
==================================================================================
2008-11-18 03:57:14,372 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) task_200810152105_0156_m_000000java.io.IOException: Type mismatch in key from map: expected org.apache.pig.impl.io.NullableText, recieved org.apache.pig.impl.io.NullableIntWritable
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:415)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:97)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:82)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209)
==================================================================================
Looking at the {code}explain C{code} output, you see that newmarks has become a chararray (surprising!!)
==================================================================================
|---CoGroup viraj-Tue Nov 18 03:49:42 UTC 2008-25 Schema: {group: Unknown,{username: bytearray,marks: int},ABtemptable: {username: chararray,newmarks: chararray}} Type: bag
| |
| Project viraj-Tue Nov 18 03:49:42 UTC 2008-23 Projections: [1] Overloaded: false FieldSchema: marks: int Type: int
| Input: SplitOutput[null] viraj-Tue Nov 18 03:49:42 UTC 2008-29
| |
| Project viraj-Tue Nov 18 03:49:42 UTC 2008-24 Projections: [1] Overloaded: false FieldSchema: newmarks: chararray Type: chararray
| Input: ForEach viraj-Tue Nov 18 03:49:42 UTC 2008-22
|---ForEach viraj-Tue Nov 18 03:49:42 UTC 2008-22 Schema: {username: chararray,newmarks: chararray} Type: bag
==================================================================================
In Summary this script demonstrates the following problems:
1) Logical Plan creation
2) When cogrouping with fields of different types which results in group unknown is not caught during compile phase.
Additionally I am enclosing the explain output of alias C and testfiles to run the script which is on this jira!!
Viraj
was:
Consider the following pig query, which demonstrates various problems during the Logical Plan creation and the subsequent execution of the M/R job. In this query we do two cogroups, one between A and B to generate an alias ABtemptable. Then we again cogroup A with ABtemptable based on marks which was read in as an int.
==================================================================================
{code}
A = load 'mymarks.txt' as (username:chararray,marks:int);
B = load 'mygrades.txt' as (username:chararray,grade:chararray);
ABtemp = cogroup A by username, B by username;
ABtemptable = foreach ABtemp generate
group as username,
flatten(A.marks) as newmarks;
--describe ABtemptable;
C = cogroup A by marks, ABtemptable by newmarks;
--describe C;
explain C;
dump C;
{code}
==================================================================================
The schema for C and ABtemptable which pig reports:
==================================================================================
{code}describe ABtemptable{code} = ABtemptable: {username: chararray,newmarks: int}
{code}describe C{code} = C: {group: int,A: {username: chararray,marks: int},ABtemptable: {username: chararray,newmarks: int}}
==================================================================================
If you run the above query you get the following error:
==================================================================================
2008-11-18 03:57:14,372 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) task_200810152105_0156_m_000000java.io.IOException: Type mismatch in key from map: expected org.apache.pig.impl.io.NullableText, recieved org.apache.pig.impl.io.NullableIntWritable
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:415)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:97)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:82)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209)
==================================================================================
Looking at the {code}explain C{code} output, you see that newmarks has become a chararray (surprising!!)
==================================================================================
|---CoGroup viraj-Tue Nov 18 03:49:42 UTC 2008-25 Schema: {group: Unknown,{username: bytearray,marks: int},ABtemptable: {username: chararray,newmarks: chararray}} Type: bag
| |
| Project viraj-Tue Nov 18 03:49:42 UTC 2008-23 Projections: [1] Overloaded: false FieldSchema: marks: int Type: int
| Input: SplitOutput[null] viraj-Tue Nov 18 03:49:42 UTC 2008-29
| |
| Project viraj-Tue Nov 18 03:49:42 UTC 2008-24 Projections: [1] Overloaded: false FieldSchema: newmarks: chararray Type: chararray
| Input: ForEach viraj-Tue Nov 18 03:49:42 UTC 2008-22
|---ForEach viraj-Tue Nov 18 03:49:42 UTC 2008-22 Schema: {username: chararray,newmarks: chararray} Type: bag
==================================================================================
In Summary this script demonstrates the following problems:
1) Logical Plan creation
2) When cogrouping with fields of different types which results in group unknown is not caught during compile phase.
Additionally I am enclosing the explain output of alias C and testfiles to run the script which is on this jira!!
Viraj
> Failure in Hadoop map collect stage due to type mismatch in the keys used in cogroup
> ------------------------------------------------------------------------------------
>
> Key: PIG-537
> URL: https://issues.apache.org/jira/browse/PIG-537
> Project: Pig
> Issue Type: Bug
> Affects Versions: types_branch
> Reporter: Viraj Bhat
> Priority: Critical
> Fix For: types_branch
>
>
> Consider the following pig query, which demonstrates various problems during the Logical Plan creation and the subsequent execution of the M/R job. In this query we do two cogroups, one between A and B to generate an alias ABtemptable. Then we again cogroup A with ABtemptable based on marks which was read in as an int.
> ==================================================================================
> {code}
> A = load 'mymarks.txt' as (username:chararray,marks:int);
> B = load 'mygrades.txt' as (username:chararray,grade:chararray);
> ABtemp = cogroup A by username, B by username;
> ABtemptable = foreach ABtemp generate
> group as username,
> flatten(A.marks) as newmarks;
> --describe ABtemptable;
> C = cogroup A by marks, ABtemptable by newmarks;
> --describe C;
> explain C;
> dump C;
> {code}
> ==================================================================================
> The schema for C and ABtemptable which pig reports:
> ==================================================================================
> {code}describe ABtemptable;{code} ABtemptable: {username: chararray,newmarks: int}
> {code}describe C;{code} C: {group: int,A: {username: chararray,marks: int},ABtemptable: {username: chararray,newmarks: int}}
> ==================================================================================
> If you run the above query you get the following error:
> ==================================================================================
> 2008-11-18 03:57:14,372 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) task_200810152105_0156_m_000000java.io.IOException: Type mismatch in key from map: expected org.apache.pig.impl.io.NullableText, recieved org.apache.pig.impl.io.NullableIntWritable
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:415)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:97)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:82)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209)
> ==================================================================================
> Looking at the {code}explain C{code} output, you see that newmarks has become a chararray (surprising!!)
> ==================================================================================
> |---CoGroup viraj-Tue Nov 18 03:49:42 UTC 2008-25 Schema: {group: Unknown,{username: bytearray,marks: int},ABtemptable: {username: chararray,newmarks: chararray}} Type: bag
> | |
> | Project viraj-Tue Nov 18 03:49:42 UTC 2008-23 Projections: [1] Overloaded: false FieldSchema: marks: int Type: int
> | Input: SplitOutput[null] viraj-Tue Nov 18 03:49:42 UTC 2008-29
> | |
> | Project viraj-Tue Nov 18 03:49:42 UTC 2008-24 Projections: [1] Overloaded: false FieldSchema: newmarks: chararray Type: chararray
> | Input: ForEach viraj-Tue Nov 18 03:49:42 UTC 2008-22
> |---ForEach viraj-Tue Nov 18 03:49:42 UTC 2008-22 Schema: {username: chararray,newmarks: chararray} Type: bag
> ==================================================================================
> In Summary this script demonstrates the following problems:
> 1) Logical Plan creation
> 2) When cogrouping with fields of different types which results in group unknown is not caught during compile phase.
> Additionally I am enclosing the explain output of alias C and testfiles to run the script which is on this jira!!
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.