You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-dev@hadoop.apache.org by Mohammad Mustaqeem <3m...@gmail.com> on 2013/05/22 15:29:17 UTC

Rack-awareness in Hadoop-2.0.3-alpha

Is Hadoop-2.0.3-alpha does not support Rack-awarness?
I am trying to make Hadoop cluster Rack-Aware for a week but I
haven't succeed.

What I am doing.
I am adding following property in etc/hadoop/core-site.xml :
<property>
<name>net.topology.script.file.name</name>
<value>/home/hadoop/hadoop-2.0.3-alpha/etc/hadoop/topology.sh</value>
</property>
The topology.sh and topology.data is attached with this mail.

I want to tell you that once I used this property(topology.script.file.name)
and files to make Hadoop-0.22.0 rack-aware but I am unable to do the same
for Hadoop-2.0.3-alpha.
What I am doing wrong?
Please, somebody help me.


-- 
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad

Re: Rack-awareness in Hadoop-2.0.3-alpha

Posted by Patai Sangbutsarakum <si...@gmail.com>.

I believe that his topology.sh and .data files are already correct.
bash topology.sh 172.31.13.133 mustaqeem-1 mustaqeem-4
/default/rack /rack2 /rack3

output looks exactly the same as mine.


Mohammad,
1. did you restart namenode after you modified the configuration? in 0.20,
restart namenode is required.
2. Try to revert the scheduler to the default i believe it is FIFO, and
play with Rack-awareness again.
3. Check if you need to chown topology.sh to the user that run namenode
4. Check if you need to chmod u+x topology.sh
5. Check if topology.data can be read by the user.


hope this help



On Wed, May 22, 2013 at 9:49 AM, Chris Nauroth <cn...@hortonworks.com>wrote:

> common-dev and hdfs-dev removed/bcc'd
>
> Hi Mohammad,
>
> Rack awareness is supported in 2.0.3-alpha.  The only potential problem I
> see in your configuration is that topology.sh contains a definition for
> HADOOP_CONF that points back at your hadoop-0.22.0/conf directory.  If that
> directory doesn't contain the right topology.data file, then resolution to
> a rack might not work as you expected.
>
> If that doesn't fix the problem, then I have another suggestion: change
> topology.sh to add some debugging statements that echo to a temp file the
> arguments that the script has received, and the results it decides to
> print.  This would show you exactly how Hadoop called your script and
> exactly what rack your script replied back to Hadoop.  It might give you an
> idea for what to investigate next.
>
> Hope this helps,
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Wed, May 22, 2013 at 6:29 AM, Mohammad Mustaqeem <
> 3m.mustaqeem@gmail.com> wrote:
>
>> Is Hadoop-2.0.3-alpha does not support Rack-awarness?
>> I am trying to make Hadoop cluster Rack-Aware for a week but I
>> haven't succeed.
>>
>> What I am doing.
>> I am adding following property in etc/hadoop/core-site.xml :
>> <property>
>> <name>net.topology.script.file.name</name>
>> <value>/home/hadoop/hadoop-2.0.3-alpha/etc/hadoop/topology.sh</value>
>> </property>
>> The topology.sh and topology.data is attached with this mail.
>>
>> I want to tell you that once I used this property(
>> topology.script.file.name) and files to make Hadoop-0.22.0 rack-aware
>> but I am unable to do the same for Hadoop-2.0.3-alpha.
>> What I am doing wrong?
>> Please, somebody help me.
>>
>>
>> --
>> *With regards ---*
>> *Mohammad Mustaqeem*,
>> M.Tech (CSE)
>> MNNIT Allahabad
>>
>
>

Re: Rack-awareness in Hadoop-2.0.3-alpha

Posted by Patai Sangbutsarakum <si...@gmail.com>.

I believe that his topology.sh and .data files are already correct.
bash topology.sh 172.31.13.133 mustaqeem-1 mustaqeem-4
/default/rack /rack2 /rack3

output looks exactly the same as mine.


Mohammad,
1. did you restart namenode after you modified the configuration? in 0.20,
restart namenode is required.
2. Try to revert the scheduler to the default i believe it is FIFO, and
play with Rack-awareness again.
3. Check if you need to chown topology.sh to the user that run namenode
4. Check if you need to chmod u+x topology.sh
5. Check if topology.data can be read by the user.


hope this help



On Wed, May 22, 2013 at 9:49 AM, Chris Nauroth <cn...@hortonworks.com>wrote:

> common-dev and hdfs-dev removed/bcc'd
>
> Hi Mohammad,
>
> Rack awareness is supported in 2.0.3-alpha.  The only potential problem I
> see in your configuration is that topology.sh contains a definition for
> HADOOP_CONF that points back at your hadoop-0.22.0/conf directory.  If that
> directory doesn't contain the right topology.data file, then resolution to
> a rack might not work as you expected.
>
> If that doesn't fix the problem, then I have another suggestion: change
> topology.sh to add some debugging statements that echo to a temp file the
> arguments that the script has received, and the results it decides to
> print.  This would show you exactly how Hadoop called your script and
> exactly what rack your script replied back to Hadoop.  It might give you an
> idea for what to investigate next.
>
> Hope this helps,
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Wed, May 22, 2013 at 6:29 AM, Mohammad Mustaqeem <
> 3m.mustaqeem@gmail.com> wrote:
>
>> Is Hadoop-2.0.3-alpha does not support Rack-awarness?
>> I am trying to make Hadoop cluster Rack-Aware for a week but I
>> haven't succeed.
>>
>> What I am doing.
>> I am adding following property in etc/hadoop/core-site.xml :
>> <property>
>> <name>net.topology.script.file.name</name>
>> <value>/home/hadoop/hadoop-2.0.3-alpha/etc/hadoop/topology.sh</value>
>> </property>
>> The topology.sh and topology.data is attached with this mail.
>>
>> I want to tell you that once I used this property(
>> topology.script.file.name) and files to make Hadoop-0.22.0 rack-aware
>> but I am unable to do the same for Hadoop-2.0.3-alpha.
>> What I am doing wrong?
>> Please, somebody help me.
>>
>>
>> --
>> *With regards ---*
>> *Mohammad Mustaqeem*,
>> M.Tech (CSE)
>> MNNIT Allahabad
>>
>
>

Re: Rack-awareness in Hadoop-2.0.3-alpha

Posted by Patai Sangbutsarakum <si...@gmail.com>.

I believe that his topology.sh and .data files are already correct.
bash topology.sh 172.31.13.133 mustaqeem-1 mustaqeem-4
/default/rack /rack2 /rack3

output looks exactly the same as mine.


Mohammad,
1. did you restart namenode after you modified the configuration? in 0.20,
restart namenode is required.
2. Try to revert the scheduler to the default i believe it is FIFO, and
play with Rack-awareness again.
3. Check if you need to chown topology.sh to the user that run namenode
4. Check if you need to chmod u+x topology.sh
5. Check if topology.data can be read by the user.


hope this help



On Wed, May 22, 2013 at 9:49 AM, Chris Nauroth <cn...@hortonworks.com>wrote:

> common-dev and hdfs-dev removed/bcc'd
>
> Hi Mohammad,
>
> Rack awareness is supported in 2.0.3-alpha.  The only potential problem I
> see in your configuration is that topology.sh contains a definition for
> HADOOP_CONF that points back at your hadoop-0.22.0/conf directory.  If that
> directory doesn't contain the right topology.data file, then resolution to
> a rack might not work as you expected.
>
> If that doesn't fix the problem, then I have another suggestion: change
> topology.sh to add some debugging statements that echo to a temp file the
> arguments that the script has received, and the results it decides to
> print.  This would show you exactly how Hadoop called your script and
> exactly what rack your script replied back to Hadoop.  It might give you an
> idea for what to investigate next.
>
> Hope this helps,
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Wed, May 22, 2013 at 6:29 AM, Mohammad Mustaqeem <
> 3m.mustaqeem@gmail.com> wrote:
>
>> Is Hadoop-2.0.3-alpha does not support Rack-awarness?
>> I am trying to make Hadoop cluster Rack-Aware for a week but I
>> haven't succeed.
>>
>> What I am doing.
>> I am adding following property in etc/hadoop/core-site.xml :
>> <property>
>> <name>net.topology.script.file.name</name>
>> <value>/home/hadoop/hadoop-2.0.3-alpha/etc/hadoop/topology.sh</value>
>> </property>
>> The topology.sh and topology.data is attached with this mail.
>>
>> I want to tell you that once I used this property(
>> topology.script.file.name) and files to make Hadoop-0.22.0 rack-aware
>> but I am unable to do the same for Hadoop-2.0.3-alpha.
>> What I am doing wrong?
>> Please, somebody help me.
>>
>>
>> --
>> *With regards ---*
>> *Mohammad Mustaqeem*,
>> M.Tech (CSE)
>> MNNIT Allahabad
>>
>
>