You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by pr...@nokia.com on 2011/02/15 22:33:12 UTC
Selecting only few slaves in the cluster
Hello all,
We have a 100 node hadoop cluster that is used for multiple purposes. I want to run few mapred jobs and I know 4 to 5 slaves should be enough. Is there anyway to restrict my jobs to use only 4 slaves instead of all 100. I noticed that more the number of slaves more overhead there is.
Also can I pass in hadoop parameters like mapred.child.java.opts so that the actual child processes gets the specified value for max heap size. I want to set the heap size to 2G instead of going with the default..
Thanks
Praveen
Re: Selecting only few slaves in the cluster
Posted by Jun Young Kim <ju...@gmail.com>.
you can use a fair-scheduler library to use only some parts of nodes you
have to run a job.
by using max/min map/reduce job counts.
here is the documentation you can reference.
http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html
Junyoung Kim (juneng603@gmail.com)
On 02/16/2011 06:33 AM, praveen.peddi@nokia.com wrote:
> Hello all,
> We have a 100 node hadoop cluster that is used for multiple purposes. I want to run few mapred jobs and I know 4 to 5 slaves should be enough. Is there anyway to restrict my jobs to use only 4 slaves instead of all 100. I noticed that more the number of slaves more overhead there is.
>
> Also can I pass in hadoop parameters like mapred.child.java.opts so that the actual child processes gets the specified value for max heap size. I want to set the heap size to 2G instead of going with the default..
>
> Thanks
> Praveen
>