You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Atul Thapliyal <at...@aggiemail.usu.edu> on 2012/04/12 18:04:05 UTC
Profiling Pig Query
Hi Guys,
Has anyone used any tools to profile a pig query ? or
Can anyone guide me on 'How to profile a pig query". I am trying to figure
out the CPU, disk I/O, RAM usage.
I have tried Starfish but its current release doesn't support Pig.
Thanks for reading !!
--
Warm Regards
Atul
Re: Profiling Pig Query
Posted by Atul Thapliyal <at...@aggiemail.usu.edu>.
Hi Dmitriy / Keren -
To profile pig queries, I was looking into these two tools.
1. Yourkit (Thanks to Thejas) - You can find a document
here<https://github.com/athap/Documents/blob/master/HowTo/ProfilePigQueriesUsingYourkit/profilingPigQuery.txt>
and also here <https://cwiki.apache.org/confluence/display/PIG/HowToProfile>
2. Jinspired (Thanks to William) - A document is available
here<http://www.jinspired.com/site/profiling-apache-hadoop-child-mapreduce-tasks-with-jxinsight-opus>
Hope this saves someone's time !!
Thanks
On Sat, May 12, 2012 at 7:48 PM, Atul Thapliyal <
atul.thapliyal@aggiemail.usu.edu> wrote:
> Hi Keren,
>
> I ended up using Yourkit Java profiler with Pig. It works for me.
>
> Thanks
>
>
> On Sat, May 12, 2012 at 4:28 PM, Keren Ouaknine <ke...@gmail.com> wrote:
>
>> Hello,
>>
>> I am interested by this topic (profiling pig), any progress since last
>> suggestion?
>> Thanks,
>> Keren
>>
>> On Fri, Apr 13, 2012 at 4:47 AM, Atul Thapliyal <
>> atul.thapliyal@aggiemail.usu.edu> wrote:
>>
>> > Hi Dmitriy,
>> >
>> > Thanks for you reply. If nothing comes up then I will try to use Hprof
>> with
>> > Pig. Though I am not sure if its going to work.
>> >
>> > Thanks again !!
>> >
>> > On Thu, Apr 12, 2012 at 6:12 PM, Dmitriy Ryaboy <dv...@gmail.com>
>> > wrote:
>> >
>> > > Hi Atul,
>> > > I usually profile by running pig in local mode and connecting to it
>> > > from yourkit (or whatever profiler you like best).
>> > >
>> > > To understand what's happening on a live cluster, I don't currently
>> > > have anything fancy -- just looking at the normal system stats, doing
>> > > occasional jstacks, etc. If people have suggestions, that'd be great
>> > > to hear.
>> > >
>> > > D
>> > >
>> > > On Thu, Apr 12, 2012 at 9:04 AM, Atul Thapliyal
>> > > <at...@aggiemail.usu.edu> wrote:
>> > > > Hi Guys,
>> > > >
>> > > > Has anyone used any tools to profile a pig query ? or
>> > > >
>> > > > Can anyone guide me on 'How to profile a pig query". I am trying to
>> > > figure
>> > > > out the CPU, disk I/O, RAM usage.
>> > > >
>> > > > I have tried Starfish but its current release doesn't support Pig.
>> > > >
>> > > > Thanks for reading !!
>> > > >
>> > > > --
>> > > > Warm Regards
>> > > > Atul
>> > >
>> >
>> >
>> >
>> > --
>> > Warm Regards
>> > Atul
>> >
>>
>>
>>
>> --
>> Keren Ouaknine
>> Web: www.kereno.com
>>
>
>
>
> --
> Warm Regards
> Atul
>
>
--
Warm Regards
Atul
Re: Profiling Pig Query
Posted by Atul Thapliyal <at...@aggiemail.usu.edu>.
Hi Keren,
I ended up using Yourkit Java profiler with Pig. It works for me.
Thanks
On Sat, May 12, 2012 at 4:28 PM, Keren Ouaknine <ke...@gmail.com> wrote:
> Hello,
>
> I am interested by this topic (profiling pig), any progress since last
> suggestion?
> Thanks,
> Keren
>
> On Fri, Apr 13, 2012 at 4:47 AM, Atul Thapliyal <
> atul.thapliyal@aggiemail.usu.edu> wrote:
>
> > Hi Dmitriy,
> >
> > Thanks for you reply. If nothing comes up then I will try to use Hprof
> with
> > Pig. Though I am not sure if its going to work.
> >
> > Thanks again !!
> >
> > On Thu, Apr 12, 2012 at 6:12 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > wrote:
> >
> > > Hi Atul,
> > > I usually profile by running pig in local mode and connecting to it
> > > from yourkit (or whatever profiler you like best).
> > >
> > > To understand what's happening on a live cluster, I don't currently
> > > have anything fancy -- just looking at the normal system stats, doing
> > > occasional jstacks, etc. If people have suggestions, that'd be great
> > > to hear.
> > >
> > > D
> > >
> > > On Thu, Apr 12, 2012 at 9:04 AM, Atul Thapliyal
> > > <at...@aggiemail.usu.edu> wrote:
> > > > Hi Guys,
> > > >
> > > > Has anyone used any tools to profile a pig query ? or
> > > >
> > > > Can anyone guide me on 'How to profile a pig query". I am trying to
> > > figure
> > > > out the CPU, disk I/O, RAM usage.
> > > >
> > > > I have tried Starfish but its current release doesn't support Pig.
> > > >
> > > > Thanks for reading !!
> > > >
> > > > --
> > > > Warm Regards
> > > > Atul
> > >
> >
> >
> >
> > --
> > Warm Regards
> > Atul
> >
>
>
>
> --
> Keren Ouaknine
> Web: www.kereno.com
>
--
Warm Regards
Atul
Re: Profiling Pig Query
Posted by Keren Ouaknine <ke...@gmail.com>.
Hello,
I am interested by this topic (profiling pig), any progress since last
suggestion?
Thanks,
Keren
On Fri, Apr 13, 2012 at 4:47 AM, Atul Thapliyal <
atul.thapliyal@aggiemail.usu.edu> wrote:
> Hi Dmitriy,
>
> Thanks for you reply. If nothing comes up then I will try to use Hprof with
> Pig. Though I am not sure if its going to work.
>
> Thanks again !!
>
> On Thu, Apr 12, 2012 at 6:12 PM, Dmitriy Ryaboy <dv...@gmail.com>
> wrote:
>
> > Hi Atul,
> > I usually profile by running pig in local mode and connecting to it
> > from yourkit (or whatever profiler you like best).
> >
> > To understand what's happening on a live cluster, I don't currently
> > have anything fancy -- just looking at the normal system stats, doing
> > occasional jstacks, etc. If people have suggestions, that'd be great
> > to hear.
> >
> > D
> >
> > On Thu, Apr 12, 2012 at 9:04 AM, Atul Thapliyal
> > <at...@aggiemail.usu.edu> wrote:
> > > Hi Guys,
> > >
> > > Has anyone used any tools to profile a pig query ? or
> > >
> > > Can anyone guide me on 'How to profile a pig query". I am trying to
> > figure
> > > out the CPU, disk I/O, RAM usage.
> > >
> > > I have tried Starfish but its current release doesn't support Pig.
> > >
> > > Thanks for reading !!
> > >
> > > --
> > > Warm Regards
> > > Atul
> >
>
>
>
> --
> Warm Regards
> Atul
>
--
Keren Ouaknine
Web: www.kereno.com
Re: Profiling Pig Query
Posted by Atul Thapliyal <at...@aggiemail.usu.edu>.
Hi Dmitriy,
Thanks for you reply. If nothing comes up then I will try to use Hprof with
Pig. Though I am not sure if its going to work.
Thanks again !!
On Thu, Apr 12, 2012 at 6:12 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Hi Atul,
> I usually profile by running pig in local mode and connecting to it
> from yourkit (or whatever profiler you like best).
>
> To understand what's happening on a live cluster, I don't currently
> have anything fancy -- just looking at the normal system stats, doing
> occasional jstacks, etc. If people have suggestions, that'd be great
> to hear.
>
> D
>
> On Thu, Apr 12, 2012 at 9:04 AM, Atul Thapliyal
> <at...@aggiemail.usu.edu> wrote:
> > Hi Guys,
> >
> > Has anyone used any tools to profile a pig query ? or
> >
> > Can anyone guide me on 'How to profile a pig query". I am trying to
> figure
> > out the CPU, disk I/O, RAM usage.
> >
> > I have tried Starfish but its current release doesn't support Pig.
> >
> > Thanks for reading !!
> >
> > --
> > Warm Regards
> > Atul
>
--
Warm Regards
Atul
Re: Profiling Pig Query
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Hi Atul,
I usually profile by running pig in local mode and connecting to it
from yourkit (or whatever profiler you like best).
To understand what's happening on a live cluster, I don't currently
have anything fancy -- just looking at the normal system stats, doing
occasional jstacks, etc. If people have suggestions, that'd be great
to hear.
D
On Thu, Apr 12, 2012 at 9:04 AM, Atul Thapliyal
<at...@aggiemail.usu.edu> wrote:
> Hi Guys,
>
> Has anyone used any tools to profile a pig query ? or
>
> Can anyone guide me on 'How to profile a pig query". I am trying to figure
> out the CPU, disk I/O, RAM usage.
>
> I have tried Starfish but its current release doesn't support Pig.
>
> Thanks for reading !!
>
> --
> Warm Regards
> Atul