You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by lohit <lo...@gmail.com> on 2013/02/25 22:03:44 UTC
Max mappers per cluster
Hi Devs,
With YARN, is there a way to limit max number of map tasks per cluster. For
example, if we have thousands of jobs running we want to make sure there
are room for reducers and not all mappers from all job occupy resources.
Is there some config people can use to restrict or divide map/reduce. At
first I thought it could be scheduler config, but I could not find any
information about this.
--
Have a Nice Day!
Lohit
Re: Max mappers per cluster
Posted by lohit <lo...@gmail.com>.
Thanks Arun.
I am thinking of scenario where there is way of mappers which occupy all
resources (containers) in YARN and there is potential starvation of
reducers.
When you say reduce request have higher priority, will app master send
preempt signal to capacity scheduler to free up some containers for
reducers?
2013/2/25 Arun C Murthy <ac...@hortonworks.com>
> YARN doesn't know about maps or reduces...
>
> With CapacityScheduler, this situation you described gets averted since
> resources are not 'peanut buttered' among too many jobs or users, and
> within each job reduces resource-requests have higher priority anyway.
>
> hth,
> Arun
>
> On Feb 25, 2013, at 1:03 PM, lohit wrote:
>
> > Hi Devs,
> >
> > With YARN, is there a way to limit max number of map tasks per cluster.
> For
> > example, if we have thousands of jobs running we want to make sure there
> > are room for reducers and not all mappers from all job occupy resources.
> > Is there some config people can use to restrict or divide map/reduce. At
> > first I thought it could be scheduler config, but I could not find any
> > information about this.
> >
> > --
> > Have a Nice Day!
> > Lohit
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
--
Have a Nice Day!
Lohit
Re: Max mappers per cluster
Posted by Arun C Murthy <ac...@hortonworks.com>.
YARN doesn't know about maps or reduces...
With CapacityScheduler, this situation you described gets averted since resources are not 'peanut buttered' among too many jobs or users, and within each job reduces resource-requests have higher priority anyway.
hth,
Arun
On Feb 25, 2013, at 1:03 PM, lohit wrote:
> Hi Devs,
>
> With YARN, is there a way to limit max number of map tasks per cluster. For
> example, if we have thousands of jobs running we want to make sure there
> are room for reducers and not all mappers from all job occupy resources.
> Is there some config people can use to restrict or divide map/reduce. At
> first I thought it could be scheduler config, but I could not find any
> information about this.
>
> --
> Have a Nice Day!
> Lohit
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/