You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Naveen Kumar Prasad <na...@huawei.com> on 2010/01/25 07:33:32 UTC
set how much CPU to be utilised by a MapReduce job
Hi All,
If many jobs are running concurrently in Hadoop, how can we set CPU usage
for individual tasks.
The purpose here is to avoid one job affecting the main job's execution.
Regards,
Naveen Kumar
HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo
<outbind://24-000000004ED456449CB8724C859F40771D413A7584132000/cid:919325906
@21052009-015A>
Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
-------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
Re: RE: set how much CPU to be utilised by a MapReduce job
Posted by Gautam Singaraju <ga...@gmail.com>.
One thing that you might want to consider is to increase the
replication factor. This might take a lot of disk space, but also
might increase the performance.
You might also want to check out: Sun Grid Engine Hadoop Integration
http://blogs.sun.com/templedf/entry/beta_testing_the_sun_grid
---
Gautam
On Mon, Jan 25, 2010 at 11:38 AM, Todd Lipcon <to...@cloudera.com> wrote:
> If you can require a recent kernel, you could use cgroups:
>
> http://broadcast.oreilly.com/2009/06/manage-your-performance-with-cgroups-and-projects.html
>
> No one has integrated this with hadoop yet as it's still pretty new, and
> Hadoop clusters are meant to be run on unshared hardware.
>
> -Todd
>
> On Mon, Jan 25, 2010 at 1:13 AM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> The only thing you could do is to have the tasktracker nice the child when
>> its exceeding its reservation. Aside from that, its hard to limit without
>> killing a process.
>>
>> On Jan 25, 2010 12:23 AM, "Naveen Kumar Prasad" <na...@huawei.com>
>> wrote:
>>
>>
>> This functionality may not be readily available with Hadoop.
>>
>> But it would be appreciable if anyone can help me in understanding how to
>> go
>> about developing this feature.
>>
>> Regards, Naveen Kumar HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo
>>
>> Address: Huawei Industrial Base Bantian Longgang Shenzhen 518129, P.R.China
>> www.huawei.com ---------...
>>
>> -----Original Message----- From: Allen Wittenauer [mailto:
>> awittenauer@linkedin.com] Sent: Monday, J...
>>
>
Re: RE: set how much CPU to be utilised by a MapReduce job
Posted by Todd Lipcon <to...@cloudera.com>.
If you can require a recent kernel, you could use cgroups:
http://broadcast.oreilly.com/2009/06/manage-your-performance-with-cgroups-and-projects.html
No one has integrated this with hadoop yet as it's still pretty new, and
Hadoop clusters are meant to be run on unshared hardware.
-Todd
On Mon, Jan 25, 2010 at 1:13 AM, Ryan Rawson <ry...@gmail.com> wrote:
> The only thing you could do is to have the tasktracker nice the child when
> its exceeding its reservation. Aside from that, its hard to limit without
> killing a process.
>
> On Jan 25, 2010 12:23 AM, "Naveen Kumar Prasad" <na...@huawei.com>
> wrote:
>
>
> This functionality may not be readily available with Hadoop.
>
> But it would be appreciable if anyone can help me in understanding how to
> go
> about developing this feature.
>
> Regards, Naveen Kumar HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo
>
> Address: Huawei Industrial Base Bantian Longgang Shenzhen 518129, P.R.China
> www.huawei.com ---------...
>
> -----Original Message----- From: Allen Wittenauer [mailto:
> awittenauer@linkedin.com] Sent: Monday, J...
>
Re: RE: set how much CPU to be utilised by a MapReduce job
Posted by Ryan Rawson <ry...@gmail.com>.
The only thing you could do is to have the tasktracker nice the child when
its exceeding its reservation. Aside from that, its hard to limit without
killing a process.
On Jan 25, 2010 12:23 AM, "Naveen Kumar Prasad" <na...@huawei.com>
wrote:
This functionality may not be readily available with Hadoop.
But it would be appreciable if anyone can help me in understanding how to go
about developing this feature.
Regards, Naveen Kumar HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo
Address: Huawei Industrial Base Bantian Longgang Shenzhen 518129, P.R.China
www.huawei.com ---------...
-----Original Message----- From: Allen Wittenauer [mailto:
awittenauer@linkedin.com] Sent: Monday, J...
RE: set how much CPU to be utilised by a MapReduce job
Posted by Naveen Kumar Prasad <na...@huawei.com>.
This functionality may not be readily available with Hadoop.
But it would be appreciable if anyone can help me in understanding how to go
about developing this feature.
Regards,
Naveen Kumar
HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo
Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
-------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
-----Original Message-----
From: Allen Wittenauer [mailto:awittenauer@linkedin.com]
Sent: Monday, January 25, 2010 12:44 PM
To: general@hadoop.apache.org
Subject: Re: set how much CPU to be utilised by a MapReduce job
On 1/24/10 10:33 PM, "Naveen Kumar Prasad" <na...@huawei.com> wrote:
> If many jobs are running concurrently in Hadoop, how can we set CPU
> usage for individual tasks.
That functionality does not exist.
Re: set how much CPU to be utilised by a MapReduce job
Posted by Allen Wittenauer <aw...@linkedin.com>.
On 1/24/10 10:33 PM, "Naveen Kumar Prasad" <na...@huawei.com> wrote:
> If many jobs are running concurrently in Hadoop, how can we set CPU usage
> for individual tasks.
That functionality does not exist.