You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by devdoer bird <de...@gmail.com> on 2013/08/08 13:25:26 UTC
why FairScheduler prefer to schedule MR jobs into the same node?
HI:
I configure the FairScheduler with default settings and my job has 19
reduce tasks. I found that all the reduce tasks are schedule to run in one
node.
While with default FIFO schedule, the 19 reduce tasks are scheduled into
diffrent nodes.
How can I configure FairSchedule to load more balance?
--
devdoer
devdoer@gmail.com
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by devdoer bird <de...@gmail.com>.
Thanks. Let me try this configuration. I use the default setting. Is it
set to true in default settings?
2013/8/10 Karthik Kambatla <ka...@cloudera.com>
> It is possible that you have assignMultiple set to true in your fair
> scheduler configuration - that leads to assigning as many tasks on a single
> node heartbeat as the node can accommodate. Setting it to false, would
> assign a single task on each heartbeat and can help in spreading out the
> tasks.
>
>
> On Fri, Aug 9, 2013 at 4:44 AM, devdoer bird <de...@gmail.com> wrote:
>
>> The hadoop version is 1.0.3
>>
>>
>> 2013/8/9 Sandy Ryza <sa...@cloudera.com>
>>
>>> Hi devdoer,
>>>
>>> What version are you using?
>>>
>>> -Sandy
>>>
>>>
>>> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>>>
>>>> HI:
>>>>
>>>> I configure the FairScheduler with default settings and my job has 19
>>>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>>>> node.
>>>>
>>>> While with default FIFO schedule, the 19 reduce tasks are scheduled
>>>> into diffrent nodes.
>>>>
>>>> How can I configure FairSchedule to load more balance?
>>>>
>>>>
>>>> --
>>>> devdoer
>>>> devdoer@gmail.com
>>>>
>>>
>>>
>>
>>
>> --
>> devdoer
>> devdoer@gmail.com
>>
>
>
--
devdoer
devdoer@gmail.com
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by devdoer bird <de...@gmail.com>.
Thanks. Let me try this configuration. I use the default setting. Is it
set to true in default settings?
2013/8/10 Karthik Kambatla <ka...@cloudera.com>
> It is possible that you have assignMultiple set to true in your fair
> scheduler configuration - that leads to assigning as many tasks on a single
> node heartbeat as the node can accommodate. Setting it to false, would
> assign a single task on each heartbeat and can help in spreading out the
> tasks.
>
>
> On Fri, Aug 9, 2013 at 4:44 AM, devdoer bird <de...@gmail.com> wrote:
>
>> The hadoop version is 1.0.3
>>
>>
>> 2013/8/9 Sandy Ryza <sa...@cloudera.com>
>>
>>> Hi devdoer,
>>>
>>> What version are you using?
>>>
>>> -Sandy
>>>
>>>
>>> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>>>
>>>> HI:
>>>>
>>>> I configure the FairScheduler with default settings and my job has 19
>>>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>>>> node.
>>>>
>>>> While with default FIFO schedule, the 19 reduce tasks are scheduled
>>>> into diffrent nodes.
>>>>
>>>> How can I configure FairSchedule to load more balance?
>>>>
>>>>
>>>> --
>>>> devdoer
>>>> devdoer@gmail.com
>>>>
>>>
>>>
>>
>>
>> --
>> devdoer
>> devdoer@gmail.com
>>
>
>
--
devdoer
devdoer@gmail.com
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by devdoer bird <de...@gmail.com>.
Thanks. Let me try this configuration. I use the default setting. Is it
set to true in default settings?
2013/8/10 Karthik Kambatla <ka...@cloudera.com>
> It is possible that you have assignMultiple set to true in your fair
> scheduler configuration - that leads to assigning as many tasks on a single
> node heartbeat as the node can accommodate. Setting it to false, would
> assign a single task on each heartbeat and can help in spreading out the
> tasks.
>
>
> On Fri, Aug 9, 2013 at 4:44 AM, devdoer bird <de...@gmail.com> wrote:
>
>> The hadoop version is 1.0.3
>>
>>
>> 2013/8/9 Sandy Ryza <sa...@cloudera.com>
>>
>>> Hi devdoer,
>>>
>>> What version are you using?
>>>
>>> -Sandy
>>>
>>>
>>> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>>>
>>>> HI:
>>>>
>>>> I configure the FairScheduler with default settings and my job has 19
>>>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>>>> node.
>>>>
>>>> While with default FIFO schedule, the 19 reduce tasks are scheduled
>>>> into diffrent nodes.
>>>>
>>>> How can I configure FairSchedule to load more balance?
>>>>
>>>>
>>>> --
>>>> devdoer
>>>> devdoer@gmail.com
>>>>
>>>
>>>
>>
>>
>> --
>> devdoer
>> devdoer@gmail.com
>>
>
>
--
devdoer
devdoer@gmail.com
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by devdoer bird <de...@gmail.com>.
Thanks. Let me try this configuration. I use the default setting. Is it
set to true in default settings?
2013/8/10 Karthik Kambatla <ka...@cloudera.com>
> It is possible that you have assignMultiple set to true in your fair
> scheduler configuration - that leads to assigning as many tasks on a single
> node heartbeat as the node can accommodate. Setting it to false, would
> assign a single task on each heartbeat and can help in spreading out the
> tasks.
>
>
> On Fri, Aug 9, 2013 at 4:44 AM, devdoer bird <de...@gmail.com> wrote:
>
>> The hadoop version is 1.0.3
>>
>>
>> 2013/8/9 Sandy Ryza <sa...@cloudera.com>
>>
>>> Hi devdoer,
>>>
>>> What version are you using?
>>>
>>> -Sandy
>>>
>>>
>>> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>>>
>>>> HI:
>>>>
>>>> I configure the FairScheduler with default settings and my job has 19
>>>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>>>> node.
>>>>
>>>> While with default FIFO schedule, the 19 reduce tasks are scheduled
>>>> into diffrent nodes.
>>>>
>>>> How can I configure FairSchedule to load more balance?
>>>>
>>>>
>>>> --
>>>> devdoer
>>>> devdoer@gmail.com
>>>>
>>>
>>>
>>
>>
>> --
>> devdoer
>> devdoer@gmail.com
>>
>
>
--
devdoer
devdoer@gmail.com
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by Karthik Kambatla <ka...@cloudera.com>.
It is possible that you have assignMultiple set to true in your fair
scheduler configuration - that leads to assigning as many tasks on a single
node heartbeat as the node can accommodate. Setting it to false, would
assign a single task on each heartbeat and can help in spreading out the
tasks.
On Fri, Aug 9, 2013 at 4:44 AM, devdoer bird <de...@gmail.com> wrote:
> The hadoop version is 1.0.3
>
>
> 2013/8/9 Sandy Ryza <sa...@cloudera.com>
>
>> Hi devdoer,
>>
>> What version are you using?
>>
>> -Sandy
>>
>>
>> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>>
>>> HI:
>>>
>>> I configure the FairScheduler with default settings and my job has 19
>>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>>> node.
>>>
>>> While with default FIFO schedule, the 19 reduce tasks are scheduled into
>>> diffrent nodes.
>>>
>>> How can I configure FairSchedule to load more balance?
>>>
>>>
>>> --
>>> devdoer
>>> devdoer@gmail.com
>>>
>>
>>
>
>
> --
> devdoer
> devdoer@gmail.com
>
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by Karthik Kambatla <ka...@cloudera.com>.
It is possible that you have assignMultiple set to true in your fair
scheduler configuration - that leads to assigning as many tasks on a single
node heartbeat as the node can accommodate. Setting it to false, would
assign a single task on each heartbeat and can help in spreading out the
tasks.
On Fri, Aug 9, 2013 at 4:44 AM, devdoer bird <de...@gmail.com> wrote:
> The hadoop version is 1.0.3
>
>
> 2013/8/9 Sandy Ryza <sa...@cloudera.com>
>
>> Hi devdoer,
>>
>> What version are you using?
>>
>> -Sandy
>>
>>
>> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>>
>>> HI:
>>>
>>> I configure the FairScheduler with default settings and my job has 19
>>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>>> node.
>>>
>>> While with default FIFO schedule, the 19 reduce tasks are scheduled into
>>> diffrent nodes.
>>>
>>> How can I configure FairSchedule to load more balance?
>>>
>>>
>>> --
>>> devdoer
>>> devdoer@gmail.com
>>>
>>
>>
>
>
> --
> devdoer
> devdoer@gmail.com
>
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by Karthik Kambatla <ka...@cloudera.com>.
It is possible that you have assignMultiple set to true in your fair
scheduler configuration - that leads to assigning as many tasks on a single
node heartbeat as the node can accommodate. Setting it to false, would
assign a single task on each heartbeat and can help in spreading out the
tasks.
On Fri, Aug 9, 2013 at 4:44 AM, devdoer bird <de...@gmail.com> wrote:
> The hadoop version is 1.0.3
>
>
> 2013/8/9 Sandy Ryza <sa...@cloudera.com>
>
>> Hi devdoer,
>>
>> What version are you using?
>>
>> -Sandy
>>
>>
>> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>>
>>> HI:
>>>
>>> I configure the FairScheduler with default settings and my job has 19
>>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>>> node.
>>>
>>> While with default FIFO schedule, the 19 reduce tasks are scheduled into
>>> diffrent nodes.
>>>
>>> How can I configure FairSchedule to load more balance?
>>>
>>>
>>> --
>>> devdoer
>>> devdoer@gmail.com
>>>
>>
>>
>
>
> --
> devdoer
> devdoer@gmail.com
>
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by Karthik Kambatla <ka...@cloudera.com>.
It is possible that you have assignMultiple set to true in your fair
scheduler configuration - that leads to assigning as many tasks on a single
node heartbeat as the node can accommodate. Setting it to false, would
assign a single task on each heartbeat and can help in spreading out the
tasks.
On Fri, Aug 9, 2013 at 4:44 AM, devdoer bird <de...@gmail.com> wrote:
> The hadoop version is 1.0.3
>
>
> 2013/8/9 Sandy Ryza <sa...@cloudera.com>
>
>> Hi devdoer,
>>
>> What version are you using?
>>
>> -Sandy
>>
>>
>> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>>
>>> HI:
>>>
>>> I configure the FairScheduler with default settings and my job has 19
>>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>>> node.
>>>
>>> While with default FIFO schedule, the 19 reduce tasks are scheduled into
>>> diffrent nodes.
>>>
>>> How can I configure FairSchedule to load more balance?
>>>
>>>
>>> --
>>> devdoer
>>> devdoer@gmail.com
>>>
>>
>>
>
>
> --
> devdoer
> devdoer@gmail.com
>
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by devdoer bird <de...@gmail.com>.
The hadoop version is 1.0.3
2013/8/9 Sandy Ryza <sa...@cloudera.com>
> Hi devdoer,
>
> What version are you using?
>
> -Sandy
>
>
> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>
>> HI:
>>
>> I configure the FairScheduler with default settings and my job has 19
>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>> node.
>>
>> While with default FIFO schedule, the 19 reduce tasks are scheduled into
>> diffrent nodes.
>>
>> How can I configure FairSchedule to load more balance?
>>
>>
>> --
>> devdoer
>> devdoer@gmail.com
>>
>
>
--
devdoer
devdoer@gmail.com
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by devdoer bird <de...@gmail.com>.
The hadoop version is 1.0.3
2013/8/9 Sandy Ryza <sa...@cloudera.com>
> Hi devdoer,
>
> What version are you using?
>
> -Sandy
>
>
> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>
>> HI:
>>
>> I configure the FairScheduler with default settings and my job has 19
>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>> node.
>>
>> While with default FIFO schedule, the 19 reduce tasks are scheduled into
>> diffrent nodes.
>>
>> How can I configure FairSchedule to load more balance?
>>
>>
>> --
>> devdoer
>> devdoer@gmail.com
>>
>
>
--
devdoer
devdoer@gmail.com
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by devdoer bird <de...@gmail.com>.
The hadoop version is 1.0.3
2013/8/9 Sandy Ryza <sa...@cloudera.com>
> Hi devdoer,
>
> What version are you using?
>
> -Sandy
>
>
> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>
>> HI:
>>
>> I configure the FairScheduler with default settings and my job has 19
>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>> node.
>>
>> While with default FIFO schedule, the 19 reduce tasks are scheduled into
>> diffrent nodes.
>>
>> How can I configure FairSchedule to load more balance?
>>
>>
>> --
>> devdoer
>> devdoer@gmail.com
>>
>
>
--
devdoer
devdoer@gmail.com
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by devdoer bird <de...@gmail.com>.
The hadoop version is 1.0.3
2013/8/9 Sandy Ryza <sa...@cloudera.com>
> Hi devdoer,
>
> What version are you using?
>
> -Sandy
>
>
> On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
>
>> HI:
>>
>> I configure the FairScheduler with default settings and my job has 19
>> reduce tasks. I found that all the reduce tasks are schedule to run in one
>> node.
>>
>> While with default FIFO schedule, the 19 reduce tasks are scheduled into
>> diffrent nodes.
>>
>> How can I configure FairSchedule to load more balance?
>>
>>
>> --
>> devdoer
>> devdoer@gmail.com
>>
>
>
--
devdoer
devdoer@gmail.com
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi devdoer,
What version are you using?
-Sandy
On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
> HI:
>
> I configure the FairScheduler with default settings and my job has 19
> reduce tasks. I found that all the reduce tasks are schedule to run in one
> node.
>
> While with default FIFO schedule, the 19 reduce tasks are scheduled into
> diffrent nodes.
>
> How can I configure FairSchedule to load more balance?
>
>
> --
> devdoer
> devdoer@gmail.com
>
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi devdoer,
What version are you using?
-Sandy
On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
> HI:
>
> I configure the FairScheduler with default settings and my job has 19
> reduce tasks. I found that all the reduce tasks are schedule to run in one
> node.
>
> While with default FIFO schedule, the 19 reduce tasks are scheduled into
> diffrent nodes.
>
> How can I configure FairSchedule to load more balance?
>
>
> --
> devdoer
> devdoer@gmail.com
>
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi devdoer,
What version are you using?
-Sandy
On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
> HI:
>
> I configure the FairScheduler with default settings and my job has 19
> reduce tasks. I found that all the reduce tasks are schedule to run in one
> node.
>
> While with default FIFO schedule, the 19 reduce tasks are scheduled into
> diffrent nodes.
>
> How can I configure FairSchedule to load more balance?
>
>
> --
> devdoer
> devdoer@gmail.com
>
Re: why FairScheduler prefer to schedule MR jobs into the same node?
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi devdoer,
What version are you using?
-Sandy
On Thu, Aug 8, 2013 at 4:25 AM, devdoer bird <de...@gmail.com> wrote:
> HI:
>
> I configure the FairScheduler with default settings and my job has 19
> reduce tasks. I found that all the reduce tasks are schedule to run in one
> node.
>
> While with default FIFO schedule, the 19 reduce tasks are scheduled into
> diffrent nodes.
>
> How can I configure FairSchedule to load more balance?
>
>
> --
> devdoer
> devdoer@gmail.com
>