You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by "Felix.徐" <yg...@gmail.com> on 2013/07/16 05:55:24 UTC

Policies for placing a reducer

Hi all,

What is the policy of choosing a node for a reducer in mapreduce (Hadoop
v1.2.0)?
For example,
If a cluster has 5 slaves, each slave can serve 2 maps and 2 reduces ,
there is a job who occupies 5 mappers and 3 reducers , how does the
jobtracker assign reducers to these nodes (choosing free slaves or placing
reducers close to mappers)?

Thanks.

RE: Policies for placing a reducer

Posted by Devaraj k <de...@huawei.com>.

Hi,

It doesn’t consider where the map’s ran to schedule the reducers because reducers need to contact all the mappers for the map o/p’s.  It schedules reducers wherever the slots available.

Thanks
Devaraj k

From: Felix.徐 [mailto:ygnhzeus@gmail.com]
Sent: 16 July 2013 09:25
To: user@hadoop.apache.org
Subject: Policies for placing a reducer

Hi all,

What is the policy of choosing a node for a reducer in mapreduce (Hadoop v1.2.0)?
For example,
If a cluster has 5 slaves, each slave can serve 2 maps and 2 reduces , there is a job who occupies 5 mappers and 3 reducers , how does the jobtracker assign reducers to these nodes (choosing free slaves or placing reducers close to mappers)?

Thanks.

RE: Policies for placing a reducer

Posted by Devaraj k <de...@huawei.com>.

Hi,

It doesn’t consider where the map’s ran to schedule the reducers because reducers need to contact all the mappers for the map o/p’s.  It schedules reducers wherever the slots available.

Thanks
Devaraj k

From: Felix.徐 [mailto:ygnhzeus@gmail.com]
Sent: 16 July 2013 09:25
To: user@hadoop.apache.org
Subject: Policies for placing a reducer

Hi all,

What is the policy of choosing a node for a reducer in mapreduce (Hadoop v1.2.0)?
For example,
If a cluster has 5 slaves, each slave can serve 2 maps and 2 reduces , there is a job who occupies 5 mappers and 3 reducers , how does the jobtracker assign reducers to these nodes (choosing free slaves or placing reducers close to mappers)?

Thanks.

RE: Policies for placing a reducer

Posted by Devaraj k <de...@huawei.com>.

Hi,

It doesn’t consider where the map’s ran to schedule the reducers because reducers need to contact all the mappers for the map o/p’s.  It schedules reducers wherever the slots available.

Thanks
Devaraj k

From: Felix.徐 [mailto:ygnhzeus@gmail.com]
Sent: 16 July 2013 09:25
To: user@hadoop.apache.org
Subject: Policies for placing a reducer

Hi all,

What is the policy of choosing a node for a reducer in mapreduce (Hadoop v1.2.0)?
For example,
If a cluster has 5 slaves, each slave can serve 2 maps and 2 reduces , there is a job who occupies 5 mappers and 3 reducers , how does the jobtracker assign reducers to these nodes (choosing free slaves or placing reducers close to mappers)?

Thanks.

RE: Policies for placing a reducer

Posted by Devaraj k <de...@huawei.com>.

Hi,

It doesn’t consider where the map’s ran to schedule the reducers because reducers need to contact all the mappers for the map o/p’s.  It schedules reducers wherever the slots available.

Thanks
Devaraj k

From: Felix.徐 [mailto:ygnhzeus@gmail.com]
Sent: 16 July 2013 09:25
To: user@hadoop.apache.org
Subject: Policies for placing a reducer

Hi all,

What is the policy of choosing a node for a reducer in mapreduce (Hadoop v1.2.0)?
For example,
If a cluster has 5 slaves, each slave can serve 2 maps and 2 reduces , there is a job who occupies 5 mappers and 3 reducers , how does the jobtracker assign reducers to these nodes (choosing free slaves or placing reducers close to mappers)?

Thanks.