You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Petar Jovanovic <pe...@essi.upc.edu> on 2013/02/18 17:18:16 UTC
run pig explain command over the entire script in java
Hi,
I am trying to find the way to run the explain command over the entire
pig script in java. I was using PigServer but it offers only to do
explain over the single query (alias) not the entire script. Is there a
way to do something like:
$ pig -x local -e 'explain -script Temp1/TPC_test.pig -out explain-out9.txt'
but from my Java code?
Thanks.
Re: run pig explain command over the entire script in java
Posted by Petar Jovanovic <pe...@essi.upc.edu>.
I would like to do it from Java automatically and also for the scripts
with multiple stores.
On 2/18/2013 11:51 PM, Jonathan Coveney wrote:
> so what is the issue with just doing "explain final_alias;" ? Or do you
> want it to do explain on the final alias of any given script?
>
>
> 2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
>
>> Well no. I want to do explain over the entire script as I show you in the
>> command I wrote.
>> That would produce the compact execution plan for the whole script and not
>> the several separate ones (one for each alias).
>>
>>
>>
>> On 2/18/2013 10:21 PM, Jonathan Coveney wrote:
>>
>>> I guess I'm confused at what you want then.
>>>
>>> So we have a script:
>>>
>>> a = ..
>>> b = ...
>>> c = ..
>>>
>>> my impression was that you essentially wanted to print "explain a; explain
>>> b; explain c;" is this not what you mean/
>>>
>>>
>>> 2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
>>>
>>> But pig already has this, it's jut that the java API seems not to have
>>>> it.
>>>> The point would be to have the execution flow from the entire script.
>>>> Indeed, the output is not exactly the same if the explain is called over
>>>> the entire script than if it is called for each query (alias).
>>>>
>>>> If from the shell we execute command
>>>>
>>>> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
>>>> explain-out9.txt'
>>>> it will not give the same output as if we did it for each operation
>>>> separately.
>>>>
>>>>
>>>> On 2/18/2013 7:04 PM, Jonathan Coveney wrote:
>>>>
>>>> Hacky way: grep for "^\S =", pull out the names, and then do the
>>>>> explains.
>>>>>
>>>>> Why is doing the progressive explains useful? it wouldn't be too hard to
>>>>> build this into pig but the results would be pretty unwieldy, it'd be
>>>>> really big, and pretty redundant.
>>>>>
>>>>>
>>>>> 2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
>>>>>
>>>>> Hi,
>>>>>
>>>>>> I am trying to find the way to run the explain command over the entire
>>>>>> pig
>>>>>> script in java. I was using PigServer but it offers only to do explain
>>>>>> over
>>>>>> the single query (alias) not the entire script. Is there a way to do
>>>>>> something like:
>>>>>> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
>>>>>> explain-out9.txt'
>>>>>> but from my Java code?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
Re: run pig explain command over the entire script in java
Posted by Jonathan Coveney <jc...@gmail.com>.
so what is the issue with just doing "explain final_alias;" ? Or do you
want it to do explain on the final alias of any given script?
2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
> Well no. I want to do explain over the entire script as I show you in the
> command I wrote.
> That would produce the compact execution plan for the whole script and not
> the several separate ones (one for each alias).
>
>
>
> On 2/18/2013 10:21 PM, Jonathan Coveney wrote:
>
>> I guess I'm confused at what you want then.
>>
>> So we have a script:
>>
>> a = ..
>> b = ...
>> c = ..
>>
>> my impression was that you essentially wanted to print "explain a; explain
>> b; explain c;" is this not what you mean/
>>
>>
>> 2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
>>
>> But pig already has this, it's jut that the java API seems not to have
>>> it.
>>> The point would be to have the execution flow from the entire script.
>>> Indeed, the output is not exactly the same if the explain is called over
>>> the entire script than if it is called for each query (alias).
>>>
>>> If from the shell we execute command
>>>
>>> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
>>> explain-out9.txt'
>>> it will not give the same output as if we did it for each operation
>>> separately.
>>>
>>>
>>> On 2/18/2013 7:04 PM, Jonathan Coveney wrote:
>>>
>>> Hacky way: grep for "^\S =", pull out the names, and then do the
>>>> explains.
>>>>
>>>> Why is doing the progressive explains useful? it wouldn't be too hard to
>>>> build this into pig but the results would be pretty unwieldy, it'd be
>>>> really big, and pretty redundant.
>>>>
>>>>
>>>> 2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
>>>>
>>>> Hi,
>>>>
>>>>> I am trying to find the way to run the explain command over the entire
>>>>> pig
>>>>> script in java. I was using PigServer but it offers only to do explain
>>>>> over
>>>>> the single query (alias) not the entire script. Is there a way to do
>>>>> something like:
>>>>> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
>>>>> explain-out9.txt'
>>>>> but from my Java code?
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>>
>
Re: run pig explain command over the entire script in java
Posted by Petar Jovanovic <pe...@essi.upc.edu>.
Well no. I want to do explain over the entire script as I show you in
the command I wrote.
That would produce the compact execution plan for the whole script and
not the several separate ones (one for each alias).
On 2/18/2013 10:21 PM, Jonathan Coveney wrote:
> I guess I'm confused at what you want then.
>
> So we have a script:
>
> a = ..
> b = ...
> c = ..
>
> my impression was that you essentially wanted to print "explain a; explain
> b; explain c;" is this not what you mean/
>
>
> 2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
>
>> But pig already has this, it's jut that the java API seems not to have it.
>> The point would be to have the execution flow from the entire script.
>> Indeed, the output is not exactly the same if the explain is called over
>> the entire script than if it is called for each query (alias).
>>
>> If from the shell we execute command
>>
>> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
>> explain-out9.txt'
>> it will not give the same output as if we did it for each operation
>> separately.
>>
>>
>> On 2/18/2013 7:04 PM, Jonathan Coveney wrote:
>>
>>> Hacky way: grep for "^\S =", pull out the names, and then do the explains.
>>>
>>> Why is doing the progressive explains useful? it wouldn't be too hard to
>>> build this into pig but the results would be pretty unwieldy, it'd be
>>> really big, and pretty redundant.
>>>
>>>
>>> 2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
>>>
>>> Hi,
>>>> I am trying to find the way to run the explain command over the entire
>>>> pig
>>>> script in java. I was using PigServer but it offers only to do explain
>>>> over
>>>> the single query (alias) not the entire script. Is there a way to do
>>>> something like:
>>>> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
>>>> explain-out9.txt'
>>>> but from my Java code?
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>
Re: run pig explain command over the entire script in java
Posted by Jonathan Coveney <jc...@gmail.com>.
I guess I'm confused at what you want then.
So we have a script:
a = ..
b = ...
c = ..
my impression was that you essentially wanted to print "explain a; explain
b; explain c;" is this not what you mean/
2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
> But pig already has this, it's jut that the java API seems not to have it.
> The point would be to have the execution flow from the entire script.
> Indeed, the output is not exactly the same if the explain is called over
> the entire script than if it is called for each query (alias).
>
> If from the shell we execute command
>
> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
> explain-out9.txt'
> it will not give the same output as if we did it for each operation
> separately.
>
>
> On 2/18/2013 7:04 PM, Jonathan Coveney wrote:
>
>> Hacky way: grep for "^\S =", pull out the names, and then do the explains.
>>
>> Why is doing the progressive explains useful? it wouldn't be too hard to
>> build this into pig but the results would be pretty unwieldy, it'd be
>> really big, and pretty redundant.
>>
>>
>> 2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
>>
>> Hi,
>>> I am trying to find the way to run the explain command over the entire
>>> pig
>>> script in java. I was using PigServer but it offers only to do explain
>>> over
>>> the single query (alias) not the entire script. Is there a way to do
>>> something like:
>>> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
>>> explain-out9.txt'
>>> but from my Java code?
>>>
>>> Thanks.
>>>
>>>
>>>
>
Re: run pig explain command over the entire script in java
Posted by Petar Jovanovic <pe...@essi.upc.edu>.
But pig already has this, it's jut that the java API seems not to have it.
The point would be to have the execution flow from the entire script.
Indeed, the output is not exactly the same if the explain is called over
the entire script than if it is called for each query (alias).
If from the shell we execute command
$ pig -x local -e 'explain -script Temp1/TPC_test.pig -out explain-out9.txt'
it will not give the same output as if we did it for each operation
separately.
On 2/18/2013 7:04 PM, Jonathan Coveney wrote:
> Hacky way: grep for "^\S =", pull out the names, and then do the explains.
>
> Why is doing the progressive explains useful? it wouldn't be too hard to
> build this into pig but the results would be pretty unwieldy, it'd be
> really big, and pretty redundant.
>
>
> 2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
>
>> Hi,
>> I am trying to find the way to run the explain command over the entire pig
>> script in java. I was using PigServer but it offers only to do explain over
>> the single query (alias) not the entire script. Is there a way to do
>> something like:
>> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
>> explain-out9.txt'
>> but from my Java code?
>>
>> Thanks.
>>
>>
Re: run pig explain command over the entire script in java
Posted by Jonathan Coveney <jc...@gmail.com>.
Hacky way: grep for "^\S =", pull out the names, and then do the explains.
Why is doing the progressive explains useful? it wouldn't be too hard to
build this into pig but the results would be pretty unwieldy, it'd be
really big, and pretty redundant.
2013/2/18 Petar Jovanovic <pe...@essi.upc.edu>
> Hi,
> I am trying to find the way to run the explain command over the entire pig
> script in java. I was using PigServer but it offers only to do explain over
> the single query (alias) not the entire script. Is there a way to do
> something like:
> $ pig -x local -e 'explain -script Temp1/TPC_test.pig -out
> explain-out9.txt'
> but from my Java code?
>
> Thanks.
>
>