You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by bikash sharma <sh...@gmail.com> on 2011/02/22 16:35:55 UTC
measure the resource usage of each map/reduce task
Hi,
Is there any way in which we can measure the resource usage of each
map/reduce task running?
I was trying to use sar utility to track each process resource usage,
however it seems these individual map/reduce tasks are not listed as
processes when I do ps -ex.
Thanks,
Bikash
Re: measure the resource usage of each map/reduce task
Posted by Christopher Smith <cb...@gmail.com>.
Actually, that is configurable. Multiple tasks can be run from the
same JVM, and you can also use the MultithreadedMapper
(http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapreduce/lib/map/MultithreadedMapper.html)
to at least make map tasks run in the same thread.
On Tue, Mar 1, 2011 at 9:09 AM, Harsh J <qw...@gmail.com> wrote:
> Hello,
>
> On Tue, Mar 1, 2011 at 7:29 PM, bikash sharma <sh...@gmail.com> wrote:
>> Hi,
>> As a follow-up question, do map/reduce tasks run as threads or processes?
>
> Every launched Task runs as an independent process, communicating over
> a network interface (lo) with the TaskTracker for reporting/etc.
> purposes.
>
> --
> Harsh J
> www.harshj.com
>
--
Chris
Re: measure the resource usage of each map/reduce task
Posted by Harsh J <qw...@gmail.com>.
Hello,
On Tue, Mar 1, 2011 at 7:29 PM, bikash sharma <sh...@gmail.com> wrote:
> Hi,
> As a follow-up question, do map/reduce tasks run as threads or processes?
Every launched Task runs as an independent process, communicating over
a network interface (lo) with the TaskTracker for reporting/etc.
purposes.
--
Harsh J
www.harshj.com
Re: measure the resource usage of each map/reduce task
Posted by icebergs <hk...@gmail.com>.
You can try the command as follow:
pstree
2011/3/1 bikash sharma <sh...@gmail.com>
> Hi,
> As a follow-up question, do map/reduce tasks run as threads or processes?
>
> On Tue, Feb 22, 2011 at 10:35 AM, bikash sharma <sharmabiks.07@gmail.com
> >wrote:
>
> > Hi,
> > Is there any way in which we can measure the resource usage of each
> > map/reduce task running?
> > I was trying to use sar utility to track each process resource usage,
> > however it seems these individual map/reduce tasks are not listed as
> > processes when I do ps -ex.
> >
> > Thanks,
> > Bikash
> >
>
Re: measure the resource usage of each map/reduce task
Posted by bikash sharma <sh...@gmail.com>.
Hi,
As a follow-up question, do map/reduce tasks run as threads or processes?
On Tue, Feb 22, 2011 at 10:35 AM, bikash sharma <sh...@gmail.com>wrote:
> Hi,
> Is there any way in which we can measure the resource usage of each
> map/reduce task running?
> I was trying to use sar utility to track each process resource usage,
> however it seems these individual map/reduce tasks are not listed as
> processes when I do ps -ex.
>
> Thanks,
> Bikash
>
Re: measure the resource usage of each map/reduce task
Posted by bikash sharma <sh...@gmail.com>.
Hi,
As a follow-up question, do map/reduce tasks run as threads or processes?
On Tue, Feb 22, 2011 at 10:35 AM, bikash sharma <sh...@gmail.com>wrote:
> Hi,
> Is there any way in which we can measure the resource usage of each
> map/reduce task running?
> I was trying to use sar utility to track each process resource usage,
> however it seems these individual map/reduce tasks are not listed as
> processes when I do ps -ex.
>
> Thanks,
> Bikash
>