You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Nate Carlson <ha...@natecarlson.com> on 2008/04/16 20:41:42 UTC
How to set up rack awareness?
I'm setting up a hadoop cluster across two data centers (with gig
bandwidth between them).. I'd like to use the rack awareness features to
help Hadoop know which nodes are "local".. I see that it's possible, but
haven't found any guides on how to set it up. If anyone's got a quick
primer I'd appreciate it!
-nc
Re: How to set up rack awareness?
Posted by Amar Kamat <am...@yahoo-inc.com>.
Nate Carlson wrote:
> On Thu, 17 Apr 2008, Rong-en Fan wrote:
>> I think you have to prepare a 'network script' in hadoop-site.xml.
>> This script tells which rack this host belongs to.
>
> Awesome - I did that. ;)
>
> Is there any way to verify that the config is being used?
>
To check whether the rack locality stuff worked or not see the job
counters on the jobs web ui. There you will see how many rack local maps
are there.
Amar
> ------------------------------------------------------------------------
> | nate carlson | natecars@natecarlson.com | http://www.natecarlson.com |
> | depriving some poor village of its idiot since 1981 |
> ------------------------------------------------------------------------
Re: How to set up rack awareness?
Posted by Nate Carlson <na...@natecarlson.com>.
On Thu, 17 Apr 2008, Rong-en Fan wrote:
> I think you have to prepare a 'network script' in hadoop-site.xml. This
> script tells which rack this host belongs to.
Awesome - I did that. ;)
Is there any way to verify that the config is being used?
------------------------------------------------------------------------
| nate carlson | natecars@natecarlson.com | http://www.natecarlson.com |
| depriving some poor village of its idiot since 1981 |
------------------------------------------------------------------------
Re: How to set up rack awareness?
Posted by Rong-en Fan <gr...@gmail.com>.
On Thu, Apr 17, 2008 at 2:41 AM, Nate Carlson <ha...@natecarlson.com> wrote:
> I'm setting up a hadoop cluster across two data centers (with gig bandwidth
> between them).. I'd like to use the rack awareness features to help Hadoop
> know which nodes are "local".. I see that it's possible, but haven't found
> any guides on how to set it up. If anyone's got a quick primer I'd
> appreciate it!
I think you have to prepare a 'network script' in hadoop-site.xml. This script
tells which rack this host belongs to.
Regards,
Rong-En Fan
>
> -nc
>