You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Scott Carey (JIRA)" <ji...@apache.org> on 2010/07/09 23:39:58 UTC
[jira] Created: (PIG-1491) Failure planning nested FOREACH with
DISTINCT, POLoad cannot be cast to POLocalRearrange
Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
----------------------------------------------------------------------------------------
Key: PIG-1491
URL: https://issues.apache.org/jira/browse/PIG-1491
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Scott Carey
I have a failure that occurs during planning while using DISTINCT in a nested FOREACH.
Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1491) Failure planning nested FOREACH with
DISTINCT, POLoad cannot be cast to POLocalRearrange
Posted by "Scott Carey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886908#action_12886908 ]
Scott Carey commented on PIG-1491:
----------------------------------
more readable:
{code}
DATA_G = COGROUP A by (a, b, c, d) OUTER, B by (a, b, c, d) OUTER;
DATA = FOREACH DATA_G {
a_items = DISTINCT A.x;
b_items = DISTINCT B.x;
GENERATE
FLATTEN(group) as (a,b,c,d),
SUM(A.m) as m, SUM(A.n) as n,
COUNT(a_items) as a_item_count,
(long)(SUM(B.u) + (double)0.5) as u,
(long)(SUM(B.v) + (double)0.5) as v,
COUNT(b_items) as b_item_count;
}
{code}
> Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
> ----------------------------------------------------------------------------------------
>
> Key: PIG-1491
> URL: https://issues.apache.org/jira/browse/PIG-1491
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Scott Carey
>
> I have a failure that occurs during planning while using DISTINCT in a nested FOREACH.
> Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1491) Failure planning nested FOREACH with
DISTINCT, POLoad cannot be cast to POLocalRearrange
Posted by "Scott Carey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886907#action_12886907 ]
Scott Carey commented on PIG-1491:
----------------------------------
Full stack trace at the end of this comment.
The pig script used is a couple hundred lines long. But the individual chunk that I can change to trigger the issue is the following:
DATA_G = COGROUP A by (a, b, c, d) OUTER, B by (a, b, c, d) OUTER;
DATA = FOREACH DATA_G {
a_items = DISTINCT A.x;
b_items = DISTINCT B.x;
GENERATE
FLATTEN(group) as (a,b,c,d),
SUM(A.m) as m,
SUM(A.n) as n,
COUNT(a_items) as a_item_count,
(long)(SUM(B.u) + (double)0.5) as u,
(long)(SUM(B.v) + (double)0.5) as v,
COUNT(b_items) as b_item_count;
}
Removing both of the DISTINCT temporary aliases and not generating those counts works fine. Adding either one of them causes it to fail.
ERROR 2043: Unexpected error during execution.
org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:318)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1016)
at org.apache.pig.PigServer.execute(PigServer.java:1009)
at org.apache.pig.PigServer.access$100(PigServer.java:114)
at org.apache.pig.PigServer$Graph.execute(PigServer.java:1261)
at org.apache.pig.PigServer.executeBatch(PigServer.java:326)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:110)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:167)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.main(Main.java:335)
Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69)
at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:446)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:108)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:294)
... 10 more
> Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
> ----------------------------------------------------------------------------------------
>
> Key: PIG-1491
> URL: https://issues.apache.org/jira/browse/PIG-1491
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Scott Carey
>
> I have a failure that occurs during planning while using DISTINCT in a nested FOREACH.
> Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1491) Failure planning nested FOREACH with
DISTINCT, POLoad cannot be cast to POLocalRearrange
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1491:
----------------------------
Fix Version/s: 0.9.0
> Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
> ----------------------------------------------------------------------------------------
>
> Key: PIG-1491
> URL: https://issues.apache.org/jira/browse/PIG-1491
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Scott Carey
> Fix For: 0.9.0
>
>
> I have a failure that occurs during planning while using DISTINCT in a nested FOREACH.
> Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1491) Failure planning nested FOREACH with
DISTINCT, POLoad cannot be cast to POLocalRearrange
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886906#action_12886906 ]
Ashutosh Chauhan commented on PIG-1491:
---------------------------------------
Scott,
It will be useful if you can also paste the Pig script which produced this exception.
> Failure planning nested FOREACH with DISTINCT, POLoad cannot be cast to POLocalRearrange
> ----------------------------------------------------------------------------------------
>
> Key: PIG-1491
> URL: https://issues.apache.org/jira/browse/PIG-1491
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Scott Carey
>
> I have a failure that occurs during planning while using DISTINCT in a nested FOREACH.
> Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad cannot be cast to org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizer.visitMROp(SecondaryKeyOptimizer.java:352)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:218)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper.visit(MapReduceOper.java:40)
> at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.