You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Chandrasekar <ch...@gmail.com> on 2012/02/08 01:51:31 UTC
Setting up Federated HDFS
Hi,
I'm trying to setup federated HDFS (0.23) in a single node. I'm
having a bit of trouble configuring it. I'm using this
http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html#Configuration:as
reference.
In which file should i specify all this information about
nameservices and the list of namenodes?
If there are multiple namenodes, then which one should i specify in
core-site.xml as fs.defaultFS?
Can someone please provide detailed instructions on how to configure
it?
Re: Setting up Federated HDFS
Posted by Praveen Sripati <pr...@gmail.com>.
Chandra,
In the namenode hdfs*xml, dfs.federation.nameservice.id is set to ns1, but
ns1 is not being used in the xml for defining the name node properties..
Here are the instructions to getting started with HDFS federation and mount
tables.
http://www.thecloudavenue.com/2012/01/getting-started-with-hdfs-federation.html
http://www.thecloudavenue.com/2012/01/hdfs-client-side-mount-table.html
Praveen
On Thu, Feb 9, 2012 at 2:12 PM, Chandrasekar <ch...@gmail.com>wrote:
> Hello,
> I'm still having problems configuring a cluster running
> federated HDFS, in a single node. Here's what i've done so far:
>
> 1. Extracted hdfs 0.23 tarball at 4 different locations, one for
> each of the 3 NameNode daemons and 1 for DataNode and client
>
> 2. I've added hdfs-site.xml and core-site.xml having the following
> information to each of the NameNode:
>
> core-site.xml ----
> http://pastebin.com/nJaTq00s
> hdfs-site.xml ----
> http://pastebin.com/DkueDHDf
>
> I've specified different port numbers, http port numbers and
> nameservice ids for each NameNode
>
> 3. Finally, to the fourth copy (DataNode) i've added the following
> configuration information:
>
> core-site.xml -----
> http://pastebin.com/DYfwfi44 <http://pastebin.com/58FJfTnn>
> hdfs-site.xml -----
> http://pastebin.com/xnmNnCKq
>
> I've specified fs.defaultFs here as I'll be using this as my
> client too.
>
>
> 4. I then go to each bin folder and run the "hadoop namenode
> -format" and "hadoop namenode"
> I haven't set the HADOOP_HOME env variable yet. (all namenodes
> run without any exception)
>
> 5. I then go to the DataNode bin folder and run "hadoop datanode".
> (no exception here)
>
> 6. Then I set the HADOOP_HOME env variable to the hadoop
> distribution from which the ran the DataNode.
>
> 7. Finally, I try to list out the contents of the viewfs file
> system by running "*hadoop fs -ls /*"
>
> The output i'm getting is only this : "*ls:
> viewfs://localhost:33332/*"
>
> Is this the right way to configure federated HDFS?
>
>
>
> On Wed, Feb 8, 2012 at 12:28, Suresh Srinivas <su...@hortonworks.com>wrote:
>
>>
>>
>> On Tue, Feb 7, 2012 at 4:51 PM, Chandrasekar <ch...@gmail.com>wrote:
>>
>>> In which file should i specify all this information about
>>> nameservices and the list of namenodes?
>>>
>>
>> hdfs-site.xml is the appropriate place, since it is hdfs-specific
>> configuration.
>>
>> If there are multiple namenodes, then which one should i specify
>>> in core-site.xml as fs.defaultFS?
>>>
>>
>> core-site.xml is the right place for fs.defaultFS.
>>
>> Given you have multiple namespaces from in federation setup, fs.defaultFS
>> should point to ViewFileSystem for a unified view of the namespaces to the
>> clients. There is an open bug HDFS-2558 to track this. I will get to this
>> as soon as I can.
>>
>> Regards,
>> Suresh
>>
>
>
Re: Setting up Federated HDFS
Posted by Chandrasekar <ch...@gmail.com>.
Hello,
I'm still having problems configuring a cluster running federated
HDFS, in a single node. Here's what i've done so far:
1. Extracted hdfs 0.23 tarball at 4 different locations, one for
each of the 3 NameNode daemons and 1 for DataNode and client
2. I've added hdfs-site.xml and core-site.xml having the following
information to each of the NameNode:
core-site.xml ---- http://pastebin.com/nJaTq00s
hdfs-site.xml ---- http://pastebin.com/DkueDHDf
I've specified different port numbers, http port numbers and
nameservice ids for each NameNode
3. Finally, to the fourth copy (DataNode) i've added the following
configuration information:
core-site.xml -----
http://pastebin.com/DYfwfi44<http://pastebin.com/58FJfTnn>
hdfs-site.xml ----- http://pastebin.com/xnmNnCKq
I've specified fs.defaultFs here as I'll be using this as my
client too.
4. I then go to each bin folder and run the "hadoop namenode
-format" and "hadoop namenode"
I haven't set the HADOOP_HOME env variable yet. (all namenodes
run without any exception)
5. I then go to the DataNode bin folder and run "hadoop datanode".
(no exception here)
6. Then I set the HADOOP_HOME env variable to the hadoop
distribution from which the ran the DataNode.
7. Finally, I try to list out the contents of the viewfs file
system by running "*hadoop fs -ls /*"
The output i'm getting is only this : "*ls:
viewfs://localhost:33332/*"
Is this the right way to configure federated HDFS?
On Wed, Feb 8, 2012 at 12:28, Suresh Srinivas <su...@hortonworks.com>wrote:
>
>
> On Tue, Feb 7, 2012 at 4:51 PM, Chandrasekar <ch...@gmail.com>wrote:
>
>> In which file should i specify all this information about
>> nameservices and the list of namenodes?
>>
>
> hdfs-site.xml is the appropriate place, since it is hdfs-specific
> configuration.
>
> If there are multiple namenodes, then which one should i specify
>> in core-site.xml as fs.defaultFS?
>>
>
> core-site.xml is the right place for fs.defaultFS.
>
> Given you have multiple namespaces from in federation setup, fs.defaultFS
> should point to ViewFileSystem for a unified view of the namespaces to the
> clients. There is an open bug HDFS-2558 to track this. I will get to this
> as soon as I can.
>
> Regards,
> Suresh
>
Re: Setting up Federated HDFS
Posted by Suresh Srinivas <su...@hortonworks.com>.
On Tue, Feb 7, 2012 at 4:51 PM, Chandrasekar <ch...@gmail.com>wrote:
> In which file should i specify all this information about
> nameservices and the list of namenodes?
>
hdfs-site.xml is the appropriate place, since it is hdfs-specific
configuration.
If there are multiple namenodes, then which one should i specify in
> core-site.xml as fs.defaultFS?
>
core-site.xml is the right place for fs.defaultFS.
Given you have multiple namespaces from in federation setup, fs.defaultFS
should point to ViewFileSystem for a unified view of the namespaces to the
clients. There is an open bug HDFS-2558 to track this. I will get to this
as soon as I can.
Regards,
Suresh