You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Ayon Sinha <ay...@yahoo.com> on 2011/12/06 03:10:56 UTC

Re: Pig 0.9.1's not so helpful error message about ColumnMapKeyPrune

Thanks Jonathan, that works but now I want to know why. I'm guessing I'm loosing some optimization. The thing is the previous version of the script was working fine without the -t ColumnMapKeyPrune. All I did was added a new column to my tuple that needs to get flattened.

Anyone know what's going on inside?
 
-Ayon
See My Photos on Flickr
Also check out my Blog for answers to commonly asked questions.



________________________________
 From: Jonathan Coveney <jc...@gmail.com>
To: user@pig.apache.org; Ayon Sinha <ay...@yahoo.com> 
Sent: Monday, December 5, 2011 5:58 PM
Subject: Re: Pig 0.9.1's not so helpful error message
 
There are some known issues with ColumnMapKeyPrune AFAIK, so instead of
doing:

pig script.pig

do

pig -t ColumnMapKeyPrune script.pig

does that work?

2011/12/5 Ayon Sinha <ay...@yahoo.com>

> This is all I got during the run. What does it mean? Now I have to debug
> line by line. Any hint is greatly appreciated.
>
> 2011-12-05 17:28:01,854 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t
> ColumnMapKeyPrune
> Details at logfile: /home/ayon/pig_1323134879161.log
> $ cat pig_1323134879161.log
> Pig Stack Trace
> ---------------
> ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t
> ColumnMapKeyPrune
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error
> processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
> at
> org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
> at org.apache.pig.PigServer.compilePp(PigServer.java:1360)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297)
> at org.apache.pig.PigServer.execute(PigServer.java:1286)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:360)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:553)
> at org.apache.pig.Main.main(Main.java:108)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at
> org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.removeSubTree(ColumnPruneVisitor.java:441)
> at
> org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.visit(ColumnPruneVisitor.java:421)
> at
> org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> at
> org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> at
> org.apache.pig.newplan.logical.rules.ColumnMapKeyPrune$ColumnMapKeyPruneTransformer.transform(ColumnMapKeyPrune.java:141)
> at
> org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> ... 16 more
>
> ================================================================================
>
> -Ayon
> See My Photos on Flickr
> Also check out my Blog for answers to commonly asked questions.
>

Re: Pig 0.9.1's not so helpful error message about ColumnMapKeyPrune

Posted by Ayon Sinha <ay...@yahoo.com>.
This is Pig 9.1.

Script is like :

register /home/ayon/udfs.jar;

a = load '$input' using PigStorage() as (a:chararray, b:chararray, c:int, d:chararray);

g = group a by (a, b);

g = foreach g {
      x = order $1 by c;
      generate group.a, group.b, x;
    };


u = foreach g generate myudf($1) as triplet; -- the triplet is a Bag of tuples of (chararray, chararray, int)
describe u;

z = foreach u generate flatten(triplet);
z = foreach z generate $0 as p, $1 as q, $2 as r;

dump z;

 
-Ayon
See My Photos on Flickr
Also check out my Blog for answers to commonly asked questions.



________________________________
 From: Daniel Dai <da...@hortonworks.com>
To: user@pig.apache.org; Ayon Sinha <ay...@yahoo.com> 
Sent: Monday, December 5, 2011 6:39 PM
Subject: Re: Pig 0.9.1's not so helpful error message about ColumnMapKeyPrune
 
Hi, Ayon,
Which version of Pig are you using? Can you share your script?

Thanks,
Daniel

On Mon, Dec 5, 2011 at 6:10 PM, Ayon Sinha <ay...@yahoo.com> wrote:

> Thanks Jonathan, that works but now I want to know why. I'm guessing I'm
> loosing some optimization. The thing is the previous version of the script
> was working fine without the -t ColumnMapKeyPrune. All I did was added a
> new column to my tuple that needs to get flattened.
>
> Anyone know what's going on inside?
>
> -Ayon
> See My Photos on Flickr
> Also check out my Blog for answers to commonly asked questions.
>
>
>
> ________________________________
>  From: Jonathan Coveney <jc...@gmail.com>
> To: user@pig.apache.org; Ayon Sinha <ay...@yahoo.com>
> Sent: Monday, December 5, 2011 5:58 PM
> Subject: Re: Pig 0.9.1's not so helpful error message
>
> There are some known issues with ColumnMapKeyPrune AFAIK, so instead of
> doing:
>
> pig script.pig
>
> do
>
> pig -t ColumnMapKeyPrune script.pig
>
> does that work?
>
> 2011/12/5 Ayon Sinha <ay...@yahoo.com>
>
> > This is all I got during the run. What does it mean? Now I have to debug
> > line by line. Any hint is greatly appreciated.
> >
> > 2011-12-05 17:28:01,854 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> > ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t
> > ColumnMapKeyPrune
> > Details at logfile: /home/ayon/pig_1323134879161.log
> > $ cat pig_1323134879161.log
> > Pig Stack Trace
> > ---------------
> > ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t
> > ColumnMapKeyPrune
> >
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error
> > processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
> > at
> >
> org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
> > at org.apache.pig.PigServer.compilePp(PigServer.java:1360)
> > at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297)
> > at org.apache.pig.PigServer.execute(PigServer.java:1286)
> > at org.apache.pig.PigServer.executeBatch(PigServer.java:360)
> > at
> > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> > at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> > at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> > at org.apache.pig.Main.run(Main.java:553)
> > at org.apache.pig.Main.main(Main.java:108)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > Caused by: java.util.ConcurrentModificationException
> > at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> > at java.util.AbstractList$Itr.next(AbstractList.java:343)
> > at
> >
> org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.removeSubTree(ColumnPruneVisitor.java:441)
> > at
> >
> org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.visit(ColumnPruneVisitor.java:421)
> > at
> >
> org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> > at
> >
> org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
> > at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> > at
> >
> org.apache.pig.newplan.logical.rules.ColumnMapKeyPrune$ColumnMapKeyPruneTransformer.transform(ColumnMapKeyPrune.java:141)
> > at
> >
> org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> > ... 16 more
> >
> >
> ================================================================================
> >
> > -Ayon
> > See My Photos on Flickr
> > Also check out my Blog for answers to commonly asked questions.
> >

Re: Pig 0.9.1's not so helpful error message about ColumnMapKeyPrune

Posted by Daniel Dai <da...@hortonworks.com>.
Hi, Ayon,
Which version of Pig are you using? Can you share your script?

Thanks,
Daniel

On Mon, Dec 5, 2011 at 6:10 PM, Ayon Sinha <ay...@yahoo.com> wrote:

> Thanks Jonathan, that works but now I want to know why. I'm guessing I'm
> loosing some optimization. The thing is the previous version of the script
> was working fine without the -t ColumnMapKeyPrune. All I did was added a
> new column to my tuple that needs to get flattened.
>
> Anyone know what's going on inside?
>
> -Ayon
> See My Photos on Flickr
> Also check out my Blog for answers to commonly asked questions.
>
>
>
> ________________________________
>  From: Jonathan Coveney <jc...@gmail.com>
> To: user@pig.apache.org; Ayon Sinha <ay...@yahoo.com>
> Sent: Monday, December 5, 2011 5:58 PM
> Subject: Re: Pig 0.9.1's not so helpful error message
>
> There are some known issues with ColumnMapKeyPrune AFAIK, so instead of
> doing:
>
> pig script.pig
>
> do
>
> pig -t ColumnMapKeyPrune script.pig
>
> does that work?
>
> 2011/12/5 Ayon Sinha <ay...@yahoo.com>
>
> > This is all I got during the run. What does it mean? Now I have to debug
> > line by line. Any hint is greatly appreciated.
> >
> > 2011-12-05 17:28:01,854 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> > ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t
> > ColumnMapKeyPrune
> > Details at logfile: /home/ayon/pig_1323134879161.log
> > $ cat pig_1323134879161.log
> > Pig Stack Trace
> > ---------------
> > ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t
> > ColumnMapKeyPrune
> >
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error
> > processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
> > at
> >
> org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
> > at org.apache.pig.PigServer.compilePp(PigServer.java:1360)
> > at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297)
> > at org.apache.pig.PigServer.execute(PigServer.java:1286)
> > at org.apache.pig.PigServer.executeBatch(PigServer.java:360)
> > at
> > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:131)
> > at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:192)
> > at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> > at org.apache.pig.Main.run(Main.java:553)
> > at org.apache.pig.Main.main(Main.java:108)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > Caused by: java.util.ConcurrentModificationException
> > at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> > at java.util.AbstractList$Itr.next(AbstractList.java:343)
> > at
> >
> org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.removeSubTree(ColumnPruneVisitor.java:441)
> > at
> >
> org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.visit(ColumnPruneVisitor.java:421)
> > at
> >
> org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:74)
> > at
> >
> org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
> > at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
> > at
> >
> org.apache.pig.newplan.logical.rules.ColumnMapKeyPrune$ColumnMapKeyPruneTransformer.transform(ColumnMapKeyPrune.java:141)
> > at
> >
> org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
> > ... 16 more
> >
> >
> ================================================================================
> >
> > -Ayon
> > See My Photos on Flickr
> > Also check out my Blog for answers to commonly asked questions.
> >