You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by prasenjit mukherjee <pm...@quattrowireless.com> on 2009/12/18 10:48:14 UTC
Simulating a SOP in pig for optimization/debugging purpose
I am trying to figure out a way to identify the potential bottlenecks
in my pig script by putting some timestamps in the log before storing
my output. Thinking of using the following :
--debug
tmp = LIMIT <your_relation> 1;
tmp = STREAM tmp THROUGH `perl -lane ' print \"Printing MY RELATION
TEST \", scalar(localtime()), \" \", \\$F[0]'`;
DUMP tmp;
STORE <your_relation>;
Is there a better way than this ? And will it ensure that this dump
will occur just before the store ?
-Thanks,
prasen
Re: Simulating a SOP in pig for optimization/debugging purpose
Posted by Jeff Zhang <zj...@gmail.com>.
I think you can do some optimization according to the principle here:
http://hadoop.apache.org/pig/docs/r0.5.0/cookbook.html
And if you have some background of hadoop and mapreduce, you can see the
logs to find the time consumed by each job. Because one pig script will be
compiled into several map reduce jobs to hadoop.
On Tue, Jan 12, 2010 at 7:55 AM, prasenjit mukherjee <
pmukherjee@quattrowireless.com> wrote:
> So any solution available for identifying time-consuming pig-statements ?
>
> On Mon, Jan 11, 2010 at 10:44 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > The script you give below will run twice, once for the dump, and once for
> > the store. And dump is implemented as store plus cat. So I don't think
> > this will do what you want.
> >
> > Alan.
> >
> > On Dec 18, 2009, at 1:48 AM, prasenjit mukherjee wrote:
> >
> >> I am trying to figure out a way to identify the potential bottlenecks
> >> in my pig script by putting some timestamps in the log before storing
> >> my output. Thinking of using the following :
> >>
> >> --debug
> >> tmp = LIMIT <your_relation> 1;
> >> tmp = STREAM tmp THROUGH `perl -lane ' print \"Printing MY RELATION
> >> TEST \", scalar(localtime()), \" \", \\$F[0]'`;
> >> DUMP tmp;
> >>
> >> STORE <your_relation>;
> >>
> >> Is there a better way than this ? And will it ensure that this dump
> >> will occur just before the store ?
> >>
> >> -Thanks,
> >> prasen
> >
> >
>
--
Best Regards
Jeff Zhang
Re: Simulating a SOP in pig for optimization/debugging purpose
Posted by prasenjit mukherjee <pm...@quattrowireless.com>.
So any solution available for identifying time-consuming pig-statements ?
On Mon, Jan 11, 2010 at 10:44 PM, Alan Gates <ga...@yahoo-inc.com> wrote:
> The script you give below will run twice, once for the dump, and once for
> the store. And dump is implemented as store plus cat. So I don't think
> this will do what you want.
>
> Alan.
>
> On Dec 18, 2009, at 1:48 AM, prasenjit mukherjee wrote:
>
>> I am trying to figure out a way to identify the potential bottlenecks
>> in my pig script by putting some timestamps in the log before storing
>> my output. Thinking of using the following :
>>
>> --debug
>> tmp = LIMIT <your_relation> 1;
>> tmp = STREAM tmp THROUGH `perl -lane ' print \"Printing MY RELATION
>> TEST \", scalar(localtime()), \" \", \\$F[0]'`;
>> DUMP tmp;
>>
>> STORE <your_relation>;
>>
>> Is there a better way than this ? And will it ensure that this dump
>> will occur just before the store ?
>>
>> -Thanks,
>> prasen
>
>
Re: Simulating a SOP in pig for optimization/debugging purpose
Posted by Alan Gates <ga...@yahoo-inc.com>.
The script you give below will run twice, once for the dump, and once
for the store. And dump is implemented as store plus cat. So I don't
think this will do what you want.
Alan.
On Dec 18, 2009, at 1:48 AM, prasenjit mukherjee wrote:
> I am trying to figure out a way to identify the potential bottlenecks
> in my pig script by putting some timestamps in the log before storing
> my output. Thinking of using the following :
>
> --debug
> tmp = LIMIT <your_relation> 1;
> tmp = STREAM tmp THROUGH `perl -lane ' print \"Printing MY RELATION
> TEST \", scalar(localtime()), \" \", \\$F[0]'`;
> DUMP tmp;
>
> STORE <your_relation>;
>
> Is there a better way than this ? And will it ensure that this dump
> will occur just before the store ?
>
> -Thanks,
> prasen
Re: Simulating a SOP in pig for optimization/debugging purpose
Posted by Rekha Joshi <re...@yahoo-inc.com>.
I am not sure if dump/store work well together.
In anycase, if you want to debug and since the dataset is limited to 1 row, you can have one run with dump alone to see timings.
Or use store and you can always get the timings from your jobtracker/tasktracker logs for each task.
If you insist on SOP, writing your udf with SOP and passing data via it would work.
Cheers,
/R
On 12/18/09 3:18 PM, "prasenjit mukherjee" <pm...@quattrowireless.com> wrote:
I am trying to figure out a way to identify the potential bottlenecks
in my pig script by putting some timestamps in the log before storing
my output. Thinking of using the following :
--debug
tmp = LIMIT <your_relation> 1;
tmp = STREAM tmp THROUGH `perl -lane ' print \"Printing MY RELATION
TEST \", scalar(localtime()), \" \", \\$F[0]'`;
DUMP tmp;
STORE <your_relation>;
Is there a better way than this ? And will it ensure that this dump
will occur just before the store ?
-Thanks,
prasen