You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Vivek Padmanabhan (JIRA)" <ji...@apache.org> on 2011/07/13 13:42:01 UTC
[jira] [Created] (PIG-2159) New logical plan uses incorrect class
for SUM causing for ClassCastException
New logical plan uses incorrect class for SUM causing for ClassCastException
-----------------------------------------------------------------------------
Key: PIG-2159
URL: https://issues.apache.org/jira/browse/PIG-2159
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Vivek Padmanabhan
Fix For: 0.9.0
The below is my script;
{code}
A = load 'input1' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
B = load 'input2' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
C = load 'input_Main' using PigStorage(',') as (f1:int,f2:int,f3:int);
U = UNION ONSCHEMA A,B;
J = join C by (f1,f2,f3) LEFT OUTER, U by (f1,f2,f3);
Porj = foreach J generate C::f1 as f1 ,C::f2 as f2,C::f3 as f3,U::f4 as f4,U::f5 as f5;
G = GROUP Porj by (f1,f2,f3,f5);
Final = foreach G generate SUM(Porj.f4) as total;
dump Final;
{code}
The script fails at while computing the sum with class cast exception.
Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
at org.apache.pig.builtin.DoubleSum$Initial.exec(DoubleSum.java:82)
... 19 more
This is clearly a bug in the logical plan created in 0.9. The sum operation should have processed using org.apache.pig.builtin.LongSum, but instead 0.9 logical plan have used org.apache.pig.builtin.DoubleSum which is meant for sum of doubles. And hence the ClassCastException.
The same script works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2159) New logical plan uses incorrect class
for SUM causing for ClassCastException
Posted by "Vivek Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064498#comment-13064498 ]
Vivek Padmanabhan commented on PIG-2159:
----------------------------------------
The issue could be reproduced using the above mentioned script. A sample input is provided below ;
{code}
input1
100,101,102,103,104,105
110,111,112,113,114,115
input2
200,201,202,203,204,205
210,211,212,213,214,215
input0
100,101,102,103,104,105
200,201,202,203,204,205
{code}
The logical plan from explain for 0.9;
#-----------------------------------------------
# New Logical Plan:
#-----------------------------------------------
Final: (Name: LOStore Schema: total#95:double)
|
|---Final: (Name: LOForEach Schema: total#95:double)
| |
| (Name: LOGenerate[false] Schema:
total#95:double)ColumnPrune:InputUids=[91]ColumnPrune:OutputUids=[95]
| | |
| | (Name: UserFunc(org.apache.pig.builtin.DoubleSum) Type:
double Uid: 95)
The logical plan from explain for 0.8;
#-----------------------------------------------
# New Logical Plan:
#-----------------------------------------------
fake: (Name: LOStore Schema: total#68:long)
|
|---Final: (Name: LOForEach Schema: total#68:long)
| |
| (Name: LOGenerate[false] Schema:
total#68:long)ColumnPrune:InputUids=[66]ColumnPrune:OutputUids=[68]
| | |
| | (Name: UserFunc(org.apache.pig.builtin.LongSum) Type: long
Uid: 68)
| | |
| | |---(Name: Dereference Type: bag Uid: 67 Column:[3])
| | |
| | |---Porj:(Name: Project Type: bag Uid: 66 Input: 0
Column: (*))
| |
| |---Porj: (Name: LOInnerLoad[1] Schema:
f1#43:int,f2#44:int,f3#45:int,f4#59:long,f5#60:double)
> New logical plan uses incorrect class for SUM causing for ClassCastException
> -----------------------------------------------------------------------------
>
> Key: PIG-2159
> URL: https://issues.apache.org/jira/browse/PIG-2159
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Fix For: 0.9.0
>
>
> The below is my script;
> {code}
> A = load 'input1' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> B = load 'input2' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> C = load 'input_Main' using PigStorage(',') as (f1:int,f2:int,f3:int);
> U = UNION ONSCHEMA A,B;
> J = join C by (f1,f2,f3) LEFT OUTER, U by (f1,f2,f3);
> Porj = foreach J generate C::f1 as f1 ,C::f2 as f2,C::f3 as f3,U::f4 as f4,U::f5 as f5;
> G = GROUP Porj by (f1,f2,f3,f5);
> Final = foreach G generate SUM(Porj.f4) as total;
> dump Final;
> {code}
> The script fails at while computing the sum with class cast exception.
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
> at org.apache.pig.builtin.DoubleSum$Initial.exec(DoubleSum.java:82)
> ... 19 more
> This is clearly a bug in the logical plan created in 0.9. The sum operation should have processed using org.apache.pig.builtin.LongSum, but instead 0.9 logical plan have used org.apache.pig.builtin.DoubleSum which is meant for sum of doubles. And hence the ClassCastException.
> The same script works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2159) New logical plan uses incorrect class
for SUM causing for ClassCastException
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2159:
----------------------------
Resolution: Fixed
Assignee: Daniel Dai
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Patch committed to both 0.9 branch and trunk
> New logical plan uses incorrect class for SUM causing for ClassCastException
> -----------------------------------------------------------------------------
>
> Key: PIG-2159
> URL: https://issues.apache.org/jira/browse/PIG-2159
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Daniel Dai
> Priority: Blocker
> Fix For: 0.9.0
>
> Attachments: PIG-2159-1.patch, PIG-2159-2.patch
>
>
> The below is my script;
> {code}
> A = load 'input1' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> B = load 'input2' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> C = load 'input_Main' using PigStorage(',') as (f1:int,f2:int,f3:int);
> U = UNION ONSCHEMA A,B;
> J = join C by (f1,f2,f3) LEFT OUTER, U by (f1,f2,f3);
> Porj = foreach J generate C::f1 as f1 ,C::f2 as f2,C::f3 as f3,U::f4 as f4,U::f5 as f5;
> G = GROUP Porj by (f1,f2,f3,f5);
> Final = foreach G generate SUM(Porj.f4) as total;
> dump Final;
> {code}
> The script fails at while computing the sum with class cast exception.
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
> at org.apache.pig.builtin.DoubleSum$Initial.exec(DoubleSum.java:82)
> ... 19 more
> This is clearly a bug in the logical plan created in 0.9. The sum operation should have processed using org.apache.pig.builtin.LongSum, but instead 0.9 logical plan have used org.apache.pig.builtin.DoubleSum which is meant for sum of doubles. And hence the ClassCastException.
> The same script works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2159) New logical plan uses incorrect class
for SUM causing for ClassCastException
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2159:
----------------------------
Attachment: PIG-2159-1.patch
> New logical plan uses incorrect class for SUM causing for ClassCastException
> -----------------------------------------------------------------------------
>
> Key: PIG-2159
> URL: https://issues.apache.org/jira/browse/PIG-2159
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Priority: Blocker
> Fix For: 0.9.0
>
> Attachments: PIG-2159-1.patch
>
>
> The below is my script;
> {code}
> A = load 'input1' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> B = load 'input2' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> C = load 'input_Main' using PigStorage(',') as (f1:int,f2:int,f3:int);
> U = UNION ONSCHEMA A,B;
> J = join C by (f1,f2,f3) LEFT OUTER, U by (f1,f2,f3);
> Porj = foreach J generate C::f1 as f1 ,C::f2 as f2,C::f3 as f3,U::f4 as f4,U::f5 as f5;
> G = GROUP Porj by (f1,f2,f3,f5);
> Final = foreach G generate SUM(Porj.f4) as total;
> dump Final;
> {code}
> The script fails at while computing the sum with class cast exception.
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
> at org.apache.pig.builtin.DoubleSum$Initial.exec(DoubleSum.java:82)
> ... 19 more
> This is clearly a bug in the logical plan created in 0.9. The sum operation should have processed using org.apache.pig.builtin.LongSum, but instead 0.9 logical plan have used org.apache.pig.builtin.DoubleSum which is meant for sum of doubles. And hence the ClassCastException.
> The same script works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2159) New logical plan uses incorrect class
for SUM causing for ClassCastException
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066082#comment-13066082 ]
Alan Gates commented on PIG-2159:
---------------------------------
Dmitry, I don't see this as a blocker for 0.9. It does not produce wrong results and users can rewrite their scripts to work around it. I agree it should go on the 0.9 branch and be part of the anticipated 0.9.1 release.
> New logical plan uses incorrect class for SUM causing for ClassCastException
> -----------------------------------------------------------------------------
>
> Key: PIG-2159
> URL: https://issues.apache.org/jira/browse/PIG-2159
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Priority: Blocker
> Fix For: 0.9.0
>
> Attachments: PIG-2159-1.patch, PIG-2159-2.patch
>
>
> The below is my script;
> {code}
> A = load 'input1' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> B = load 'input2' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> C = load 'input_Main' using PigStorage(',') as (f1:int,f2:int,f3:int);
> U = UNION ONSCHEMA A,B;
> J = join C by (f1,f2,f3) LEFT OUTER, U by (f1,f2,f3);
> Porj = foreach J generate C::f1 as f1 ,C::f2 as f2,C::f3 as f3,U::f4 as f4,U::f5 as f5;
> G = GROUP Porj by (f1,f2,f3,f5);
> Final = foreach G generate SUM(Porj.f4) as total;
> dump Final;
> {code}
> The script fails at while computing the sum with class cast exception.
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
> at org.apache.pig.builtin.DoubleSum$Initial.exec(DoubleSum.java:82)
> ... 19 more
> This is clearly a bug in the logical plan created in 0.9. The sum operation should have processed using org.apache.pig.builtin.LongSum, but instead 0.9 logical plan have used org.apache.pig.builtin.DoubleSum which is meant for sum of doubles. And hence the ClassCastException.
> The same script works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2159) New logical plan uses incorrect class
for SUM causing for ClassCastException
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2159:
----------------------------
Status: Patch Available (was: Open)
> New logical plan uses incorrect class for SUM causing for ClassCastException
> -----------------------------------------------------------------------------
>
> Key: PIG-2159
> URL: https://issues.apache.org/jira/browse/PIG-2159
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Priority: Blocker
> Fix For: 0.9.0
>
> Attachments: PIG-2159-1.patch, PIG-2159-2.patch
>
>
> The below is my script;
> {code}
> A = load 'input1' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> B = load 'input2' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> C = load 'input_Main' using PigStorage(',') as (f1:int,f2:int,f3:int);
> U = UNION ONSCHEMA A,B;
> J = join C by (f1,f2,f3) LEFT OUTER, U by (f1,f2,f3);
> Porj = foreach J generate C::f1 as f1 ,C::f2 as f2,C::f3 as f3,U::f4 as f4,U::f5 as f5;
> G = GROUP Porj by (f1,f2,f3,f5);
> Final = foreach G generate SUM(Porj.f4) as total;
> dump Final;
> {code}
> The script fails at while computing the sum with class cast exception.
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
> at org.apache.pig.builtin.DoubleSum$Initial.exec(DoubleSum.java:82)
> ... 19 more
> This is clearly a bug in the logical plan created in 0.9. The sum operation should have processed using org.apache.pig.builtin.LongSum, but instead 0.9 logical plan have used org.apache.pig.builtin.DoubleSum which is meant for sum of doubles. And hence the ClassCastException.
> The same script works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2159) New logical plan uses incorrect class
for SUM causing for ClassCastException
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065444#comment-13065444 ]
Daniel Dai commented on PIG-2159:
---------------------------------
The error is caused by schema generated by unionOnSchema, which only has empty uid. This would impact many queries containing unionOnSchema.
> New logical plan uses incorrect class for SUM causing for ClassCastException
> -----------------------------------------------------------------------------
>
> Key: PIG-2159
> URL: https://issues.apache.org/jira/browse/PIG-2159
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Fix For: 0.9.0
>
>
> The below is my script;
> {code}
> A = load 'input1' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> B = load 'input2' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> C = load 'input_Main' using PigStorage(',') as (f1:int,f2:int,f3:int);
> U = UNION ONSCHEMA A,B;
> J = join C by (f1,f2,f3) LEFT OUTER, U by (f1,f2,f3);
> Porj = foreach J generate C::f1 as f1 ,C::f2 as f2,C::f3 as f3,U::f4 as f4,U::f5 as f5;
> G = GROUP Porj by (f1,f2,f3,f5);
> Final = foreach G generate SUM(Porj.f4) as total;
> dump Final;
> {code}
> The script fails at while computing the sum with class cast exception.
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
> at org.apache.pig.builtin.DoubleSum$Initial.exec(DoubleSum.java:82)
> ... 19 more
> This is clearly a bug in the logical plan created in 0.9. The sum operation should have processed using org.apache.pig.builtin.LongSum, but instead 0.9 logical plan have used org.apache.pig.builtin.DoubleSum which is meant for sum of doubles. And hence the ClassCastException.
> The same script works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2159) New logical plan uses incorrect class
for SUM causing for ClassCastException
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067380#comment-13067380 ]
Thejas M Nair commented on PIG-2159:
------------------------------------
+1
> New logical plan uses incorrect class for SUM causing for ClassCastException
> -----------------------------------------------------------------------------
>
> Key: PIG-2159
> URL: https://issues.apache.org/jira/browse/PIG-2159
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Priority: Blocker
> Fix For: 0.9.0
>
> Attachments: PIG-2159-1.patch, PIG-2159-2.patch
>
>
> The below is my script;
> {code}
> A = load 'input1' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> B = load 'input2' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> C = load 'input_Main' using PigStorage(',') as (f1:int,f2:int,f3:int);
> U = UNION ONSCHEMA A,B;
> J = join C by (f1,f2,f3) LEFT OUTER, U by (f1,f2,f3);
> Porj = foreach J generate C::f1 as f1 ,C::f2 as f2,C::f3 as f3,U::f4 as f4,U::f5 as f5;
> G = GROUP Porj by (f1,f2,f3,f5);
> Final = foreach G generate SUM(Porj.f4) as total;
> dump Final;
> {code}
> The script fails at while computing the sum with class cast exception.
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
> at org.apache.pig.builtin.DoubleSum$Initial.exec(DoubleSum.java:82)
> ... 19 more
> This is clearly a bug in the logical plan created in 0.9. The sum operation should have processed using org.apache.pig.builtin.LongSum, but instead 0.9 logical plan have used org.apache.pig.builtin.DoubleSum which is meant for sum of doubles. And hence the ClassCastException.
> The same script works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2159) New logical plan uses incorrect class
for SUM causing for ClassCastException
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2159:
----------------------------
Attachment: PIG-2159-2.patch
Fix test failure on TestUnionOnSchemaSetter.
> New logical plan uses incorrect class for SUM causing for ClassCastException
> -----------------------------------------------------------------------------
>
> Key: PIG-2159
> URL: https://issues.apache.org/jira/browse/PIG-2159
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Priority: Blocker
> Fix For: 0.9.0
>
> Attachments: PIG-2159-1.patch, PIG-2159-2.patch
>
>
> The below is my script;
> {code}
> A = load 'input1' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> B = load 'input2' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> C = load 'input_Main' using PigStorage(',') as (f1:int,f2:int,f3:int);
> U = UNION ONSCHEMA A,B;
> J = join C by (f1,f2,f3) LEFT OUTER, U by (f1,f2,f3);
> Porj = foreach J generate C::f1 as f1 ,C::f2 as f2,C::f3 as f3,U::f4 as f4,U::f5 as f5;
> G = GROUP Porj by (f1,f2,f3,f5);
> Final = foreach G generate SUM(Porj.f4) as total;
> dump Final;
> {code}
> The script fails at while computing the sum with class cast exception.
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
> at org.apache.pig.builtin.DoubleSum$Initial.exec(DoubleSum.java:82)
> ... 19 more
> This is clearly a bug in the logical plan created in 0.9. The sum operation should have processed using org.apache.pig.builtin.LongSum, but instead 0.9 logical plan have used org.apache.pig.builtin.DoubleSum which is meant for sum of doubles. And hence the ClassCastException.
> The same script works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2159) New logical plan uses incorrect class
for SUM causing for ClassCastException
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy updated PIG-2159:
-----------------------------------
Priority: Blocker (was: Major)
Sounds like a blocker for the 0.9 release, changing the priority accordingly.
Nice catch.
> New logical plan uses incorrect class for SUM causing for ClassCastException
> -----------------------------------------------------------------------------
>
> Key: PIG-2159
> URL: https://issues.apache.org/jira/browse/PIG-2159
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Priority: Blocker
> Fix For: 0.9.0
>
>
> The below is my script;
> {code}
> A = load 'input1' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> B = load 'input2' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> C = load 'input_Main' using PigStorage(',') as (f1:int,f2:int,f3:int);
> U = UNION ONSCHEMA A,B;
> J = join C by (f1,f2,f3) LEFT OUTER, U by (f1,f2,f3);
> Porj = foreach J generate C::f1 as f1 ,C::f2 as f2,C::f3 as f3,U::f4 as f4,U::f5 as f5;
> G = GROUP Porj by (f1,f2,f3,f5);
> Final = foreach G generate SUM(Porj.f4) as total;
> dump Final;
> {code}
> The script fails at while computing the sum with class cast exception.
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
> at org.apache.pig.builtin.DoubleSum$Initial.exec(DoubleSum.java:82)
> ... 19 more
> This is clearly a bug in the logical plan created in 0.9. The sum operation should have processed using org.apache.pig.builtin.LongSum, but instead 0.9 logical plan have used org.apache.pig.builtin.DoubleSum which is meant for sum of doubles. And hence the ClassCastException.
> The same script works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2159) New logical plan uses incorrect class
for SUM causing for ClassCastException
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066784#comment-13066784 ]
jiraposter@reviews.apache.org commented on PIG-2159:
----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1138/
-----------------------------------------------------------
Review request for pig and thejas.
Summary
-------
See PIG-2159
This addresses bug PIG-2159.
https://issues.apache.org/jira/browse/PIG-2159
Diffs
-----
trunk/src/org/apache/pig/newplan/logical/relational/LOUnion.java 1146183
trunk/test/org/apache/pig/parser/TestUnionOnSchemaSetter.java 1146183
trunk/test/org/apache/pig/test/TestEvalPipeline2.java 1146183
Diff: https://reviews.apache.org/r/1138/diff
Testing
-------
Unit-test:
all pass
Test-patch:
all pass
Thanks,
Daniel
> New logical plan uses incorrect class for SUM causing for ClassCastException
> -----------------------------------------------------------------------------
>
> Key: PIG-2159
> URL: https://issues.apache.org/jira/browse/PIG-2159
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Vivek Padmanabhan
> Priority: Blocker
> Fix For: 0.9.0
>
> Attachments: PIG-2159-1.patch, PIG-2159-2.patch
>
>
> The below is my script;
> {code}
> A = load 'input1' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> B = load 'input2' using PigStorage(',') as (f1:int,f2:int,f3:int,f4:long,f5:double);
> C = load 'input_Main' using PigStorage(',') as (f1:int,f2:int,f3:int);
> U = UNION ONSCHEMA A,B;
> J = join C by (f1,f2,f3) LEFT OUTER, U by (f1,f2,f3);
> Porj = foreach J generate C::f1 as f1 ,C::f2 as f2,C::f3 as f3,U::f4 as f4,U::f5 as f5;
> G = GROUP Porj by (f1,f2,f3,f5);
> Final = foreach G generate SUM(Porj.f4) as total;
> dump Final;
> {code}
> The script fails at while computing the sum with class cast exception.
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
> at org.apache.pig.builtin.DoubleSum$Initial.exec(DoubleSum.java:82)
> ... 19 more
> This is clearly a bug in the logical plan created in 0.9. The sum operation should have processed using org.apache.pig.builtin.LongSum, but instead 0.9 logical plan have used org.apache.pig.builtin.DoubleSum which is meant for sum of doubles. And hence the ClassCastException.
> The same script works fine with Pig 0.8.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira