You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by pr...@nokia.com on 2011/02/15 22:33:12 UTC

Selecting only few slaves in the cluster

Hello all,
We have a 100 node hadoop cluster that is used for multiple purposes. I want to run few mapred jobs and I know 4 to 5 slaves should be enough. Is there anyway to restrict my jobs to use only 4 slaves instead of all 100. I noticed that more the number of slaves more overhead there is.

Also can I pass in hadoop parameters like mapred.child.java.opts so that the actual child processes gets the specified value for max heap size. I want to set the heap size to 2G instead of going with the default..

Thanks
Praveen

Re: Selecting only few slaves in the cluster

Posted by Jun Young Kim <ju...@gmail.com>.
you can use a fair-scheduler library to use only some parts of nodes you 
have to run a job.

by using max/min map/reduce job counts.

here is the documentation you can reference.

http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html

Junyoung Kim (juneng603@gmail.com)
On 02/16/2011 06:33 AM, praveen.peddi@nokia.com wrote:
> Hello all,
> We have a 100 node hadoop cluster that is used for multiple purposes. I want to run few mapred jobs and I know 4 to 5 slaves should be enough. Is there anyway to restrict my jobs to use only 4 slaves instead of all 100. I noticed that more the number of slaves more overhead there is.
>
> Also can I pass in hadoop parameters like mapred.child.java.opts so that the actual child processes gets the specified value for max heap size. I want to set the heap size to 2G instead of going with the default..
>
> Thanks
> Praveen
>