You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by sumit ghosh <su...@yahoo.com> on 2011/04/22 07:18:58 UTC
Getting the total Mapred time
Hi,
How to get the actual time spent in doing all the map-reduce operations while
executing a pig script. It should exclude the time wait for the scheduler - and
any other waiting time. Please help.
Another question is - how to use the function getDuration() in:
http://pig.apache.org/docs/r0.8.0/api/org/apache/pig/tools/pigstats/PigStats.html
Can you please help me with the Pig Statements.
Thanks,
Sumit
Re: Getting the total Mapred time
Posted by sumit ghosh <su...@yahoo.com>.
Ah! Thanks a lot!
I am trying to learn pig. :) So I was looking into all the different sources &
types. Kind of aimless drifting.
Sumit
________________________________
From: Dmitriy Ryaboy <dv...@gmail.com>
To: user@pig.apache.org
Sent: Mon, 25 April, 2011 9:54:25 PM
Subject: Re: Getting the total Mapred time
This is not available from inside Pig Latin -- PigStats are provided to
allow programs that embed pig to monitor and report on it.
What are you trying to do?
D
On Mon, Apr 25, 2011 at 2:03 AM, sumit ghosh <su...@yahoo.com> wrote:
> Hi,
>
> I guess this is java Syntax. I am looking to use this function in Pig
> Latin.
> What will be the syntax in Pig Latin.
>
>
> Thanks,
> Sumit
>
>
> ________________________________
> From: Dmitriy Ryaboy <dv...@gmail.com>
> To: user@pig.apache.org; gopi.daiict@gmail.com
> Sent: Sun, 24 April, 2011 5:04:20 AM
> Subject: Re: Getting the total Mapred time
>
> I am still unclear on what isn't clear :)
>
> What are you trying to do? Did you read the javadocs for PigStats?
>
> To call get duration it's just
>
> PigStats stats = PigRunner.run(myscript, mylistener);
> long duration = stats.getDuration();
>
> D
>
>
>
> On Sat, Apr 23, 2011 at 10:09 AM, Chaitanya Sharma <gopi.daiict@gmail.com
> >wrote:
>
> > Are there any example's around this which could prove useful for
> > understanding this?
> >
> > Thanks,
> > Chaitanya
> >
> > On Fri, Apr 22, 2011 at 1:52 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > wrote:
> >
> > > I may be misunderstanding what you are asking. The tricky part is
> > measuring
> > > MR time *without* wait time, which one cannot control (it depends
> mostly
> > on
> > > the size and utilization level of your cluster). This tricky bit is
> what
> > > PigStats helps you with.
> > > If you just want to measure the full time, including wait time, you can
> > > just
> > > time the process like you would time anything else. Use the "time" unix
> > > command, or if you are in Java, use System.currentTimeMillis() before
> you
> > > start executing and again when the procedure returns.
> > >
> > > On Thu, Apr 21, 2011 at 10:18 PM, sumit ghosh <su...@yahoo.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > How to get the actual time spent in doing all the map-reduce
> operations
> > > > while
> > > > executing a pig script. It should exclude the time wait for the
> > scheduler
> > > -
> > > > and
> > > > any other waiting time. Please help.
> > > >
> > > > Another question is - how to use the function getDuration() in:
> > > >
> > > >
> > >
> >
>http://pig.apache.org/docs/r0.8.0/api/org/apache/pig/tools/pigstats/PigStats.html
>l
> >l
> > > >
> > > > Can you please help me with the Pig Statements.
> > > >
> > > > Thanks,
> > > > Sumit
> > > >
> > >
> >
>
Re: Getting the total Mapred time
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
This is not available from inside Pig Latin -- PigStats are provided to
allow programs that embed pig to monitor and report on it.
What are you trying to do?
D
On Mon, Apr 25, 2011 at 2:03 AM, sumit ghosh <su...@yahoo.com> wrote:
> Hi,
>
> I guess this is java Syntax. I am looking to use this function in Pig
> Latin.
> What will be the syntax in Pig Latin.
>
>
> Thanks,
> Sumit
>
>
> ________________________________
> From: Dmitriy Ryaboy <dv...@gmail.com>
> To: user@pig.apache.org; gopi.daiict@gmail.com
> Sent: Sun, 24 April, 2011 5:04:20 AM
> Subject: Re: Getting the total Mapred time
>
> I am still unclear on what isn't clear :)
>
> What are you trying to do? Did you read the javadocs for PigStats?
>
> To call get duration it's just
>
> PigStats stats = PigRunner.run(myscript, mylistener);
> long duration = stats.getDuration();
>
> D
>
>
>
> On Sat, Apr 23, 2011 at 10:09 AM, Chaitanya Sharma <gopi.daiict@gmail.com
> >wrote:
>
> > Are there any example's around this which could prove useful for
> > understanding this?
> >
> > Thanks,
> > Chaitanya
> >
> > On Fri, Apr 22, 2011 at 1:52 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > wrote:
> >
> > > I may be misunderstanding what you are asking. The tricky part is
> > measuring
> > > MR time *without* wait time, which one cannot control (it depends
> mostly
> > on
> > > the size and utilization level of your cluster). This tricky bit is
> what
> > > PigStats helps you with.
> > > If you just want to measure the full time, including wait time, you can
> > > just
> > > time the process like you would time anything else. Use the "time" unix
> > > command, or if you are in Java, use System.currentTimeMillis() before
> you
> > > start executing and again when the procedure returns.
> > >
> > > On Thu, Apr 21, 2011 at 10:18 PM, sumit ghosh <su...@yahoo.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > How to get the actual time spent in doing all the map-reduce
> operations
> > > > while
> > > > executing a pig script. It should exclude the time wait for the
> > scheduler
> > > -
> > > > and
> > > > any other waiting time. Please help.
> > > >
> > > > Another question is - how to use the function getDuration() in:
> > > >
> > > >
> > >
> >
> http://pig.apache.org/docs/r0.8.0/api/org/apache/pig/tools/pigstats/PigStats.html
> >l
> > > >
> > > > Can you please help me with the Pig Statements.
> > > >
> > > > Thanks,
> > > > Sumit
> > > >
> > >
> >
>
Re: Getting the total Mapred time
Posted by sumit ghosh <su...@yahoo.com>.
Hi,
I guess this is java Syntax. I am looking to use this function in Pig Latin.
What will be the syntax in Pig Latin.
Thanks,
Sumit
________________________________
From: Dmitriy Ryaboy <dv...@gmail.com>
To: user@pig.apache.org; gopi.daiict@gmail.com
Sent: Sun, 24 April, 2011 5:04:20 AM
Subject: Re: Getting the total Mapred time
I am still unclear on what isn't clear :)
What are you trying to do? Did you read the javadocs for PigStats?
To call get duration it's just
PigStats stats = PigRunner.run(myscript, mylistener);
long duration = stats.getDuration();
D
On Sat, Apr 23, 2011 at 10:09 AM, Chaitanya Sharma <go...@gmail.com>wrote:
> Are there any example's around this which could prove useful for
> understanding this?
>
> Thanks,
> Chaitanya
>
> On Fri, Apr 22, 2011 at 1:52 PM, Dmitriy Ryaboy <dv...@gmail.com>
> wrote:
>
> > I may be misunderstanding what you are asking. The tricky part is
> measuring
> > MR time *without* wait time, which one cannot control (it depends mostly
> on
> > the size and utilization level of your cluster). This tricky bit is what
> > PigStats helps you with.
> > If you just want to measure the full time, including wait time, you can
> > just
> > time the process like you would time anything else. Use the "time" unix
> > command, or if you are in Java, use System.currentTimeMillis() before you
> > start executing and again when the procedure returns.
> >
> > On Thu, Apr 21, 2011 at 10:18 PM, sumit ghosh <su...@yahoo.com> wrote:
> >
> > > Hi,
> > >
> > > How to get the actual time spent in doing all the map-reduce operations
> > > while
> > > executing a pig script. It should exclude the time wait for the
> scheduler
> > -
> > > and
> > > any other waiting time. Please help.
> > >
> > > Another question is - how to use the function getDuration() in:
> > >
> > >
> >
>http://pig.apache.org/docs/r0.8.0/api/org/apache/pig/tools/pigstats/PigStats.html
>l
> > >
> > > Can you please help me with the Pig Statements.
> > >
> > > Thanks,
> > > Sumit
> > >
> >
>
Re: Getting the total Mapred time
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
I am still unclear on what isn't clear :)
What are you trying to do? Did you read the javadocs for PigStats?
To call get duration it's just
PigStats stats = PigRunner.run(myscript, mylistener);
long duration = stats.getDuration();
D
On Sat, Apr 23, 2011 at 10:09 AM, Chaitanya Sharma <go...@gmail.com>wrote:
> Are there any example's around this which could prove useful for
> understanding this?
>
> Thanks,
> Chaitanya
>
> On Fri, Apr 22, 2011 at 1:52 PM, Dmitriy Ryaboy <dv...@gmail.com>
> wrote:
>
> > I may be misunderstanding what you are asking. The tricky part is
> measuring
> > MR time *without* wait time, which one cannot control (it depends mostly
> on
> > the size and utilization level of your cluster). This tricky bit is what
> > PigStats helps you with.
> > If you just want to measure the full time, including wait time, you can
> > just
> > time the process like you would time anything else. Use the "time" unix
> > command, or if you are in Java, use System.currentTimeMillis() before you
> > start executing and again when the procedure returns.
> >
> > On Thu, Apr 21, 2011 at 10:18 PM, sumit ghosh <su...@yahoo.com> wrote:
> >
> > > Hi,
> > >
> > > How to get the actual time spent in doing all the map-reduce operations
> > > while
> > > executing a pig script. It should exclude the time wait for the
> scheduler
> > -
> > > and
> > > any other waiting time. Please help.
> > >
> > > Another question is - how to use the function getDuration() in:
> > >
> > >
> >
> http://pig.apache.org/docs/r0.8.0/api/org/apache/pig/tools/pigstats/PigStats.html
> > >
> > > Can you please help me with the Pig Statements.
> > >
> > > Thanks,
> > > Sumit
> > >
> >
>
Re: Getting the total Mapred time
Posted by Chaitanya Sharma <go...@gmail.com>.
Are there any example's around this which could prove useful for
understanding this?
Thanks,
Chaitanya
On Fri, Apr 22, 2011 at 1:52 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> I may be misunderstanding what you are asking. The tricky part is measuring
> MR time *without* wait time, which one cannot control (it depends mostly on
> the size and utilization level of your cluster). This tricky bit is what
> PigStats helps you with.
> If you just want to measure the full time, including wait time, you can
> just
> time the process like you would time anything else. Use the "time" unix
> command, or if you are in Java, use System.currentTimeMillis() before you
> start executing and again when the procedure returns.
>
> On Thu, Apr 21, 2011 at 10:18 PM, sumit ghosh <su...@yahoo.com> wrote:
>
> > Hi,
> >
> > How to get the actual time spent in doing all the map-reduce operations
> > while
> > executing a pig script. It should exclude the time wait for the scheduler
> -
> > and
> > any other waiting time. Please help.
> >
> > Another question is - how to use the function getDuration() in:
> >
> >
> http://pig.apache.org/docs/r0.8.0/api/org/apache/pig/tools/pigstats/PigStats.html
> >
> > Can you please help me with the Pig Statements.
> >
> > Thanks,
> > Sumit
> >
>
Re: Getting the total Mapred time
Posted by sumit ghosh <su...@yahoo.com>.
Thanks. I will try that out.
Any idea on the second one - What will be the pig syntax for using this function
"getDuration();
Thanks,
Sumit.
________________________________
From: Dmitriy Ryaboy <dv...@gmail.com>
To: user@pig.apache.org
Sent: Fri, 22 April, 2011 11:22:54 PM
Subject: Re: Getting the total Mapred time
I may be misunderstanding what you are asking. The tricky part is measuring
MR time *without* wait time, which one cannot control (it depends mostly on
the size and utilization level of your cluster). This tricky bit is what
PigStats helps you with.
If you just want to measure the full time, including wait time, you can just
time the process like you would time anything else. Use the "time" unix
command, or if you are in Java, use System.currentTimeMillis() before you
start executing and again when the procedure returns.
On Thu, Apr 21, 2011 at 10:18 PM, sumit ghosh <su...@yahoo.com> wrote:
> Hi,
>
> How to get the actual time spent in doing all the map-reduce operations
> while
> executing a pig script. It should exclude the time wait for the scheduler -
> and
> any other waiting time. Please help.
>
> Another question is - how to use the function getDuration() in:
>
>http://pig.apache.org/docs/r0.8.0/api/org/apache/pig/tools/pigstats/PigStats.html
>l
>
> Can you please help me with the Pig Statements.
>
> Thanks,
> Sumit
>
Re: Getting the total Mapred time
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
I may be misunderstanding what you are asking. The tricky part is measuring
MR time *without* wait time, which one cannot control (it depends mostly on
the size and utilization level of your cluster). This tricky bit is what
PigStats helps you with.
If you just want to measure the full time, including wait time, you can just
time the process like you would time anything else. Use the "time" unix
command, or if you are in Java, use System.currentTimeMillis() before you
start executing and again when the procedure returns.
On Thu, Apr 21, 2011 at 10:18 PM, sumit ghosh <su...@yahoo.com> wrote:
> Hi,
>
> How to get the actual time spent in doing all the map-reduce operations
> while
> executing a pig script. It should exclude the time wait for the scheduler -
> and
> any other waiting time. Please help.
>
> Another question is - how to use the function getDuration() in:
>
> http://pig.apache.org/docs/r0.8.0/api/org/apache/pig/tools/pigstats/PigStats.html
>
> Can you please help me with the Pig Statements.
>
> Thanks,
> Sumit
>