You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Todd Lipcon <to...@cloudera.com> on 2011/10/19 02:59:13 UTC
Delay scheduling in MR2?
Hey folks,
Looking at ContainerLeafQueue in MR2, it seems that delay scheduling
is only used for rack locality and not for node locality. This is
resulting in jobs on a single-rack cluster having much less locality
than MR1 and impacting performance.
Was this an intentional decision? Or am I misunderstanding the code here?
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Delay scheduling in MR2?
Posted by Todd Lipcon <to...@cloudera.com>.
I filed MR-3210. Thanks
On Tue, Oct 18, 2011 at 10:30 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On Tue, Oct 18, 2011 at 6:31 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> The CS has always treated rack-local same as node-local... so yes, the current behaviour is expected.
>
> Hrm - never knew that -- I'm usually a fairscheduler user, and the FS
> does do delay scheduling for both types of locality.
>
> What was the logic that lead to this decision? A small amount of delay
> for node locality seems worth it, especially on small clusters (<40
> nodes) with 1gbit networking.
>
> If there's no particular reason against it, I'll file a JIRA to add it
> as an option.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Delay scheduling in MR2?
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, Oct 18, 2011 at 6:31 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> The CS has always treated rack-local same as node-local... so yes, the current behaviour is expected.
Hrm - never knew that -- I'm usually a fairscheduler user, and the FS
does do delay scheduling for both types of locality.
What was the logic that lead to this decision? A small amount of delay
for node locality seems worth it, especially on small clusters (<40
nodes) with 1gbit networking.
If there's no particular reason against it, I'll file a JIRA to add it
as an option.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Delay scheduling in MR2?
Posted by Arun C Murthy <ac...@hortonworks.com>.
The CS has always treated rack-local same as node-local... so yes, the current behaviour is expected.
Arun
On Oct 18, 2011, at 5:59 PM, Todd Lipcon wrote:
> Hey folks,
>
> Looking at ContainerLeafQueue in MR2, it seems that delay scheduling
> is only used for rack locality and not for node locality. This is
> resulting in jobs on a single-rack cluster having much less locality
> than MR1 and impacting performance.
>
> Was this an intentional decision? Or am I misunderstanding the code here?
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera