conceptual question regarding slots

[bikash sharma <sh...@gmail.com>]

Hi,
Could someone throw some light as to how intuitively fixed-type slots in
Hadoop have a negative impact of cluster utilization as mentioned in Arun's
blog?
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/

Thanks,
Bikash

Re: conceptual question regarding slots

[bikash sharma <sh...@gmail.com>]

Thanks Greg!

On Wed, Mar 2, 2011 at 5:07 PM, Greg Roelofs <ro...@yahoo-inc.com> wrote:

> > Could someone throw some light as to how intuitively fixed-type slots in
> > Hadoop have a negative impact of cluster utilization as mentioned in
> Arun's
> > blog?
>
> It's pretty simple:  in the course of time, a real cluster with a mix of
> job types and submission times will have periods of high map-slot usage
> (with reduce-slots available), and vice versa.  Being able to assign
> machine
> resources to whatever kind of task needs them at any given time means
> you're
> not wasting those resources.
>
> In short:  map- vs. reduce-slots is an artificial and unnecessary
> restriction.
>
> Greg
>

Re: conceptual question regarding slots

[Greg Roelofs <ro...@yahoo-inc.com>]

> Could someone throw some light as to how intuitively fixed-type slots in
> Hadoop have a negative impact of cluster utilization as mentioned in Arun's
> blog?

It's pretty simple:  in the course of time, a real cluster with a mix of
job types and submission times will have periods of high map-slot usage
(with reduce-slots available), and vice versa.  Being able to assign machine
resources to whatever kind of task needs them at any given time means you're
not wasting those resources.

In short:  map- vs. reduce-slots is an artificial and unnecessary restriction.

Greg