You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2007/12/06 20:02:43 UTC

[jira] Commented: (PIG-43) Combiner queries are not repeatable using the same alias

    [ https://issues.apache.org/jira/browse/PIG-43?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549134 ] 

Olga Natkovich commented on PIG-43:
-----------------------------------

looks good. +1

> Combiner queries are not repeatable using the same alias
> --------------------------------------------------------
>
>                 Key: PIG-43
>                 URL: https://issues.apache.org/jira/browse/PIG-43
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.0.0
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>            Priority: Minor
>         Attachments: pig43.patch
>
>
> In general, it is possible to define an alias in pig and use it multiple times:
> a = load 'file';
> b = group a by $0;
> c = foreach b generate group, COUNT($1);
> dump c;
> dump c;
> Queries that choose to use the combiner give the following error on the second dump:
> java.lang.AssertionError: Can't convert non-algebraic function to final.
>         at org.apache.pig.impl.eval.FuncEvalSpec.resetFuncToFinal(FuncEvalSpec.java:285)
>         at org.apache.pig.impl.physicalLayer.MapreducePlanCompiler$ReduceAdjuster.visitFuncEval(MapreducePlanCompiler.java:393)
>         at org.apache.pig.impl.eval.FuncEvalSpec.visit(FuncEvalSpec.java:238)
>         at org.apache.pig.impl.physicalLayer.MapreducePlanCompiler$ReduceAdjuster.visitGenerate(MapreducePlanCompiler.java:367)
>         at org.apache.pig.impl.eval.GenerateSpec.visit(GenerateSpec.java:374)
>         at org.apache.pig.impl.physicalLayer.MapreducePlanCompiler.pushInto(MapreducePlanCompiler.java:223)
>         at org.apache.pig.impl.physicalLayer.MapreducePlanCompiler.compile(MapreducePlanCompiler.java:79)
>         at org.apache.pig.impl.physicalLayer.PhysicalPlan.<init>(PhysicalPlan.java:56)
>         at org.apache.pig.impl.physicalLayer.IntermedResult.compile(IntermedResult.java:95)
>         at org.apache.pig.PigServer.openIterator(PigServer.java:319)
>         at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:120)
>         at org.apache.pig.tools.grunt.GruntParser.parse(GruntParser.java:334)
>         at org.apache.pig.tools.grunt.GruntParser.parseContOnError(GruntParser.java:38)
>         at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:48)
>         at org.apache.pig.Main.main(Main.java:211)
> This is because the logic that chooses the combiner modifies the EvalSpecs in the logical plan.  Pig does not preserve the physical plan built for the first run of the query, but tries to reconstruct it. When it does so the modified EvalSpecs cause it a problem.  The combiner logic needs to be changed to make a copy of the EvalSpecs rather than modify them.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.