You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2014/06/20 03:35:24 UTC
[jira] [Resolved] (PIG-3987) Unjustified cast error when performing
a UNION
[ https://issues.apache.org/jira/browse/PIG-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai resolved PIG-3987.
-----------------------------
Resolution: Won't Fix
The script runs in recent version of Pig (tried on 0.12.1). We anyway won't fix 0.10, close the Jira.
> Unjustified cast error when performing a UNION
> ----------------------------------------------
>
> Key: PIG-3987
> URL: https://issues.apache.org/jira/browse/PIG-3987
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.1
> Reporter: Giovanni Botta
> Attachments: .pig_header, .pig_schema, part-r-00000
>
>
> I ran into a very strange issue with one of my pig scripts. I described it in this SO: http://stackoverflow.com/questions/24047572/strange-cast-error-in-pig-hadoop
> Here it is:
> I have the following script:
> {code}
> br = LOAD 'cfs:///somedata';
> SPLIT br INTO s0 IF (sp == 1), not_s0 OTHERWISE;
> SPLIT not_s0 INTO s1 IF (adp >= 1.0), not_s1 OTHERWISE;
> SPLIT not_s1 INTO s2 IF (p > 1L), not_s2 OTHERWISE;
> SPLIT not_s2 INTO s3 IF (s > 0L), s4 OTHERWISE;
> tmp0 = FOREACH s0 GENERATE b, 'x' as seg;
> tmp1 = FOREACH s1 GENERATE b, 'y' as seg;
> tmp2 = FOREACH s2 GENERATE b, 'z' as seg;
> tmp3 = FOREACH s3 GENERATE b, 'w' as seg;
> tmp4 = FOREACH s4 GENERATE b, 't' as seg;
> out = UNION ONSCHEMA tmp0, tmp1, tmp2, tmp3, tmp4;
> dump out;
> {code}
> Where the file loaded in br was generated by a previous Pig script and has an embedded schema (a .pig_schema file):
> {code}
> describe br
> br: {b: chararray,p: long,afternoon: long,ddv: long,pa: long,s: long,t0002: long,t0204: long,t0406: long,t0608: long,t0810: long,t1012: long,t1214: long,t1416: long,t1618: long,t1820: long,t2022: long,t2200: long,browser_software: chararray,first_timestamp: long,last_timestamp: long,os: chararray,platform: chararray,sp: int,adp: double}
> {code}
> Some irrelevant fields were edited from the above (I can't fully disclose the nature of the data at this time).
> The script fails with the following error:
> {code}
> ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR: java.lang.Integer cannot be cast to java.lang.Long
> {code}
> However, dumping tmp0, tmp1, tmp2, tmp3, tmp4 works flawlessly.
> The Hadoop job tracker shows the following error 4 times:
> {code}
> java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
> at java.lang.Long.compareTo(Long.java:50)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.EqualToExpr.doComparison(EqualToExpr.java:116)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.EqualToExpr.getNext(EqualToExpr.java:83)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.PONot.getNext(PONot.java:71)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFilter.getNext(POFilter.java:148)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:233)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.getNext(POSplit.java:214)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.runPipeline(POSplit.java:254)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.processPlan(POSplit.java:236)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.getNext(POSplit.java:228)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:271)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:266)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:260)
> {code}
> I also tried this snippet (instead of the original dump):
> {code}
> x = UNION s1,s2;
> y = FOREACH x GENERATE b;
> dump y;
> {code}
> and I get a different (but I assume related) error:
> {code}
> ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR: java.lang.Double cannot be cast to java.lang.Long
> {code}
> with the job tracker error (repeated 4 times):
> {code}
> java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.Long
> at java.lang.Long.compareTo(Long.java:50)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.GTOrEqualToExpr.doComparison(GTOrEqualToExpr.java:111)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.GTOrEqualToExpr.getNext(GTOrEqualToExpr.java:78)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.PONot.getNext(PONot.java:71)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFilter.getNext(POFilter.java:148)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:233)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getNext(POStore.java:141)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.runPipeline(POSplit.java:254)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.processPlan(POSplit.java:236)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.getNext(POSplit.java:228)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:271)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:266)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:260)
> {code}
> I don't think I have a data quality issue. I successfully ran the following snippet (taking it from after the definition of s0, s1, ...):
> {code}
> tmp0 = FOREACH s0 GENERATE *, 'x' as seg;
> tmp1 = FOREACH s1 GENERATE *, 'y' as seg;
> tmp2 = FOREACH s2 GENERATE *, 'z' as seg;
> tmp3 = FOREACH s3 GENERATE *, 'w' as seg;
> tmp4 = FOREACH s4 GENERATE *, 't' as seg;
> br_seg = UNION ONSCHEMA tmp0, tmp1, tmp2, tmp3, tmp4;
> breakdown = FOREACH( GROUP br_seg BY seg ){
> ddb = FILTER br_seg BY (ddv > 0L);
> desktop = FILTER br_seg BY (platform == 'd');
> mobile = FILTER br_seg BY (platform == 'm');
> p_br = FILTER br_seg BY (sp == 1);
> tablet = FILTER br_seg BY (platform == 't');
> GENERATE group as seg,
> COUNT(br_seg) as br,
> SUM(br_seg.p) as p,
> COUNT(ddb) as ddb,
> COUNT(desktop) as desktop,
> COUNT(mobile) as mobile,
> COUNT(p_br) as p_br,
> COUNT(tablet) as tablet,
> SUM(br_seg.ddv) as ddv,
> SUM(br_seg.pa) as pa,
> SUM(br_seg.t0002) as t0002,
> SUM(br_seg.t0204) as t0204,
> SUM(br_seg.t0406) as t0406,
> SUM(br_seg.t0608) as t0608,
> SUM(br_seg.t0810) as t0810,
> SUM(br_seg.t1012) as t1012,
> SUM(br_seg.t1214) as t1214,
> SUM(br_seg.t1416) as t1416,
> SUM(br_seg.t1618) as t1618,
> SUM(br_seg.t1820) as t1820,
> SUM(br_seg.t2022) as t2022,
> SUM(br_seg.t2200) as t2200;
> }
> dump breakdown
> {code}
> And got as output:
> {code}
> (t,43,43,0,30,7,0,6,0,0,2,5,9,3,2,3,1,4,1,4,4,5)
> (w,17,17,0,10,3,0,4,0,0,1,1,1,0,1,2,1,6,1,1,1,1)
> (x,17,243,0,12,2,17,3,0,243,1,6,4,9,20,55,40,37,21,23,8,19)
> (y,17,108,0,14,2,0,1,0,0,7,3,5,5,3,11,29,4,16,6,13,6)
> (z,6,12,0,4,1,0,1,0,0,0,0,0,2,3,1,0,4,2,0,0,0)
> {code}
> Is this a known bug or a new one? Is there a work around?
--
This message was sent by Atlassian JIRA
(v6.2#6252)