You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Iman Elghandour <ie...@yahoo.com> on 2009/03/19 17:27:04 UTC
multiquery execution plans using explain
Hello,
I am having trouble checking the execution plans using the explain command. I sent before this email about errors when using explain in a script: http://www.mail-archive.com/pig-user@hadoop.apache.org/msg00374.html
I noticed later that in the multiquery specification http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification, the explain is not used in batch mode. So I tried the interactive mode but it did not work. The error issued is as follows
ERROR 2998: Unhandled internal error. Ceci n'est pas un bug.
java.lang.AssertionError: Ceci n'est pas un bug.
at org.apache.pig.PigServer.getStorePlan(PigServer.java:694)
at org.apache.pig.PigServer.explain(PigServer.java:537)
at org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:229)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.Explain(PigScriptParser.java:478)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:211)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:133)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:72)
at org.apache.pig.Main.main(Main.java:296)
This error is returned because there are more than one leaf in the script. The execution of the scripts work fine. The problem is mainly with their explanation.
These are the two scripts that I have tried:
A = load 'data' as (a, b, c);
B = filter A by a > 5;
store B into 'operatorResults/queryOpImplicitSplit1';
C = group B by b;
store C into 'operatorResults/queryOpImplicitSplit2';
A = load 'data' as (a, b, c);
split A into B if a > 0, C if a>0;
B1 = filter B by a > 5;
store B1 into 'operatorResults/queryOpExplicitSplit1';
C1 = group C by b;
store C1 into 'operatorResults/queryOpExplicitSplit2';
I tried to replace the "store"s with "explain"s in the scripts, or to remove them but it never worked. I have tried to execute the scripts and they work fine.
Would someone help me by providing an example of how to generate the execution plans.
Thanks,
Iman
__________________________________________________________________
Looking for the perfect gift? Give the gift of Flickr!
http://www.flickr.com/gift/
Re: multiquery execution plans using explain
Posted by Gunther Hagleitner <ha...@yahoo-inc.com>.
Iman,
The multi-query branch is a fairly unstable dev branch and I am afraid
you'll have to expect things like you saw for now. If you're experimenting
or want to contribute that's fine, but if you are looking for something more
stable, you probably want to switch to the trunk.
For your explain:
I copied your script into foo.pig and then ran:
grunt:> explain -out . -script foo.pig
That'll give you for the execution plan:
(yes, the store into tmp is unnecessary, that'll change)
--------------------------------------------------
| Map Reduce Plan |
--------------------------------------------------
MapReduce node 1-41
Map Plan
Split - 1-47
| |
|
Store(operatorResults/queryOpImplicitSplit1:org.apache.pig.builtin.PigStorag
e) - 1-40
| |
| |---Filter[tuple] - 1-38
| | |
| | Constant(true) - 1-39
| |
| Local Rearrange[tuple]{bytearray}(false) - 1-35
| | |
| | Project[bytearray][1] - 1-36
| |
| |---Filter[tuple] - 1-31
| | |
| | Constant(true) - 1-32
| |
|
Store(/tmp/temp356542017/tmp1047134161:org.apache.pig.builtin.BinStorage) -
1-42
|
|---Filter[bag] - 1-25
| |
| Greater Than[boolean] - 1-29
| |
| |---Constant(5) - 1-26
| |
| |---Cast[int] - 1-28
| |
| |---Project[bytearray][0] - 1-27
|
|---Load(data:org.apache.pig.builtin.PigStorage) - 1-24--------
Reduce Plan
Store(operatorResults/queryOpImplicitSplit2:org.apache.pig.builtin.PigStorag
e) - 1-37
|
|---Package[tuple]{bytearray} - 1-34--------
Global sort: false
----------------
Thanks,
Gunther.
On 3/19/09 9:27 AM, "Iman Elghandour" <ie...@yahoo.com> wrote:
> Hello,
> I am having trouble checking the execution plans using the explain command. I
> sent before this email about errors when using explain in a script:
> http://www.mail-archive.com/pig-user@hadoop.apache.org/msg00374.html
>
> I noticed later that in the multiquery specification
> http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification, the explain
> is not used in batch mode. So I tried the interactive mode but it did not
> work. The error issued is as follows
>
> ERROR 2998: Unhandled internal error. Ceci n'est pas un bug.
> java.lang.AssertionError: Ceci n'est pas un bug.
> at org.apache.pig.PigServer.getStorePlan(PigServer.java:694)
> at org.apache.pig.PigServer.explain(PigServer.java:537)
> at
> org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:229)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Explain(PigScriptParser.
> java:478)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.ja
> va:211)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:133)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:72)
> at org.apache.pig.Main.main(Main.java:296)
>
> This error is returned because there are more than one leaf in the script. The
> execution of the scripts work fine. The problem is mainly with their
> explanation.
>
> These are the two scripts that I have tried:
>
> A = load 'data' as (a, b, c);
> B = filter A by a > 5;
> store B into 'operatorResults/queryOpImplicitSplit1';
> C = group B by b;
> store C into 'operatorResults/queryOpImplicitSplit2';
>
> A = load 'data' as (a, b, c);
> split A into B if a > 0, C if a>0;
> B1 = filter B by a > 5;
> store B1 into 'operatorResults/queryOpExplicitSplit1';
> C1 = group C by b;
> store C1 into 'operatorResults/queryOpExplicitSplit2';
>
> I tried to replace the "store"s with "explain"s in the scripts, or to remove
> them but it never worked. I have tried to execute the scripts and they work
> fine.
>
> Would someone help me by providing an example of how to generate the execution
> plans.
>
> Thanks,
> Iman
>
>
>
>
> __________________________________________________________________
> Looking for the perfect gift? Give the gift of Flickr!
>
> http://www.flickr.com/gift/