You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by souravm <SO...@infosys.com> on 2008/09/17 01:43:46 UTC
Remote datanode (fully distributed way) is not starting in Mac
OSX...
Hi.
Any pointer on what could be the problem ?
Regards,
Sourav
________________________________________
From: souravm
Sent: Tuesday, September 16, 2008 1:07 AM
To: 'core-user@hadoop.apache.org'
Subject: Re: Need help in hdfs configuration fully distributed way in Mac OSX...
Hi,
I tried the way u suggested. I setup ssh without password. So now namenode can connect to datanode without password - the start-dfs.sh script does not ask for any password. However, even with this fix I still face the same problem.
Regards,
Sourav
----- Original Message -----
From: Mafish Liu <ma...@gmail.com>
To: core-user@hadoop.apache.org <co...@hadoop.apache.org>
Sent: Mon Sep 15 23:26:10 2008
Subject: Re: Need help in hdfs configuration fully distributed way in Mac OSX...
Hi:
You need to configure your nodes to ensure that node 1 can connect to node
2 without password.
On Tue, Sep 16, 2008 at 2:04 PM, souravm <SO...@infosys.com> wrote:
> Hi All,
>
> I'm facing a problem in configuring hdfs in a fully distributed way in Mac
> OSX.
>
> Here is the topology -
>
> 1. The namenode is in machine 1
> 2. There is 1 datanode in machine 2
>
> Now when I execute start-dfs.sh from machine 1, it connects to machine 2
> (after it asks for password for connecting to machine 2) and starts datanode
> in machine 2 (as the console message says).
>
> However -
> 1. When I go to http://machine1:50070 - it does not show the data node at
> all. It says 0 data node configured
> 2. In the log file in machine 2 what I see is -
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = rc0902b-dhcp169.apple.com/17.229.22.169
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.17.2.1
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.17 -r
> 684969; compiled by 'oom' on Wed Aug 20 22:29:32 UTC 2008
> ************************************************************/
> 2008-09-15 18:54:44,626 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /17.229.23.77:9000. Already tried 1 time(s).
> 2008-09-15 18:54:45,627 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /17.229.23.77:9000. Already tried 2 time(s).
> 2008-09-15 18:54:46,628 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /17.229.23.77:9000. Already tried 3 time(s).
> 2008-09-15 18:54:47,629 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /17.229.23.77:9000. Already tried 4 time(s).
> 2008-09-15 18:54:48,630 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /17.229.23.77:9000. Already tried 5 time(s).
> 2008-09-15 18:54:49,631 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /17.229.23.77:9000. Already tried 6 time(s).
> 2008-09-15 18:54:50,632 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /17.229.23.77:9000. Already tried 7 time(s).
> 2008-09-15 18:54:51,633 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /17.229.23.77:9000. Already tried 8 time(s).
> 2008-09-15 18:54:52,635 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /17.229.23.77:9000. Already tried 9 time(s).
> 2008-09-15 18:54:53,640 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: /17.229.23.77:9000. Already tried 10 time(s).
> 2008-09-15 18:54:54,641 INFO org.apache.hadoop.ipc.RPC: Server at /
> 17.229.23.77:9000 not available yet, Zzzzz...
>
> ....... and this retyring gets on repeating
>
>
> The hadoop-site.xmls are like this -
>
> 1. In machine 1
> -
> <configuration>
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://localhost:9000</value>
> </property>
>
> <property>
> <name>dfs.name.dir</name>
> <value>/Users/souravm/hdpn</value>
> </property>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>localhost:9001</value>
> </property>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> </configuration>
>
>
> 2. In machine 2
>
> <configuration>
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://<machine1 ip>:9000</value>
> </property>
> <property>
> <name>dfs.data.dir</name>
> <value>/Users/nirdosh/hdfsd1</value>
> </property>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> </configuration>
>
> The slaves file in machine 1 has single entry - <user name>@<ip of
> machine2>
>
> The exact steps I did -
>
> 1. Reformat the namenode in machine 1
> 2. execute start-dfs.sh in machine 1
> 3. Then I try to see whether the datanode is created through http://<machine
> 1 ip>:50070
>
> Any pointer to resolve this issue would be appreciated.
>
> Regards,
> Sourav
>
>
>
> **************** CAUTION - Disclaimer *****************
> This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended
> solely
> for the use of the addressee(s). If you are not the intended recipient,
> please
> notify the sender by e-mail and delete the original message. Further, you
> are not
> to copy, disclose, or distribute this e-mail or its contents to any other
> person and
> any such actions are unlawful. This e-mail may contain viruses. Infosys has
> taken
> every reasonable precaution to minimize this risk, but is not liable for
> any damage
> you may sustain as a result of any virus in this e-mail. You should carry
> out your
> own virus checks before opening the e-mail or attachment. Infosys reserves
> the
> right to monitor and review the content of all messages sent to or from
> this e-mail
> address. Messages sent to or from this e-mail address may be stored on the
> Infosys e-mail system.
> ***INFOSYS******** End of Disclaimer ********INFOSYS***
>
--
Mafish@gmail.com
Institute of Computing Technology, Chinese Academy of Sciences, Beijing.
Re: Remote datanode (fully distributed way) is not starting in Mac
OSX...
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Did you try using same config file (used on machine 2) on all the nodes?
You can make the configs you have to work, with more effort, I don't
think that is necessary.
Raghu.
souravm wrote:
> Hi.
>
> Any pointer on what could be the problem ?
>
> Regards,
> Sourav
> ________________________________________
> From: souravm
> Sent: Tuesday, September 16, 2008 1:07 AM
> To: 'core-user@hadoop.apache.org'
> Subject: Re: Need help in hdfs configuration fully distributed way in Mac OSX...
>
> Hi,
>
> I tried the way u suggested. I setup ssh without password. So now namenode can connect to datanode without password - the start-dfs.sh script does not ask for any password. However, even with this fix I still face the same problem.
>
> Regards,
> Sourav
>
> ----- Original Message -----
> From: Mafish Liu <ma...@gmail.com>
> To: core-user@hadoop.apache.org <co...@hadoop.apache.org>
> Sent: Mon Sep 15 23:26:10 2008
> Subject: Re: Need help in hdfs configuration fully distributed way in Mac OSX...
>
> Hi:
> You need to configure your nodes to ensure that node 1 can connect to node
> 2 without password.
>
> On Tue, Sep 16, 2008 at 2:04 PM, souravm <SO...@infosys.com> wrote:
>
>> Hi All,
>>
>> I'm facing a problem in configuring hdfs in a fully distributed way in Mac
>> OSX.
>>
>> Here is the topology -
>>
>> 1. The namenode is in machine 1
>> 2. There is 1 datanode in machine 2
>>
>> Now when I execute start-dfs.sh from machine 1, it connects to machine 2
>> (after it asks for password for connecting to machine 2) and starts datanode
>> in machine 2 (as the console message says).
>>
>> However -
>> 1. When I go to http://machine1:50070 - it does not show the data node at
>> all. It says 0 data node configured
>> 2. In the log file in machine 2 what I see is -
>> /************************************************************
>> STARTUP_MSG: Starting DataNode
>> STARTUP_MSG: host = rc0902b-dhcp169.apple.com/17.229.22.169
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 0.17.2.1
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.17 -r
>> 684969; compiled by 'oom' on Wed Aug 20 22:29:32 UTC 2008
>> ************************************************************/
>> 2008-09-15 18:54:44,626 INFO org.apache.hadoop.ipc.Client: Retrying connect
>> to server: /17.229.23.77:9000. Already tried 1 time(s).
>> 2008-09-15 18:54:45,627 INFO org.apache.hadoop.ipc.Client: Retrying connect
>> to server: /17.229.23.77:9000. Already tried 2 time(s).
>> 2008-09-15 18:54:46,628 INFO org.apache.hadoop.ipc.Client: Retrying connect
>> to server: /17.229.23.77:9000. Already tried 3 time(s).
>> 2008-09-15 18:54:47,629 INFO org.apache.hadoop.ipc.Client: Retrying connect
>> to server: /17.229.23.77:9000. Already tried 4 time(s).
>> 2008-09-15 18:54:48,630 INFO org.apache.hadoop.ipc.Client: Retrying connect
>> to server: /17.229.23.77:9000. Already tried 5 time(s).
>> 2008-09-15 18:54:49,631 INFO org.apache.hadoop.ipc.Client: Retrying connect
>> to server: /17.229.23.77:9000. Already tried 6 time(s).
>> 2008-09-15 18:54:50,632 INFO org.apache.hadoop.ipc.Client: Retrying connect
>> to server: /17.229.23.77:9000. Already tried 7 time(s).
>> 2008-09-15 18:54:51,633 INFO org.apache.hadoop.ipc.Client: Retrying connect
>> to server: /17.229.23.77:9000. Already tried 8 time(s).
>> 2008-09-15 18:54:52,635 INFO org.apache.hadoop.ipc.Client: Retrying connect
>> to server: /17.229.23.77:9000. Already tried 9 time(s).
>> 2008-09-15 18:54:53,640 INFO org.apache.hadoop.ipc.Client: Retrying connect
>> to server: /17.229.23.77:9000. Already tried 10 time(s).
>> 2008-09-15 18:54:54,641 INFO org.apache.hadoop.ipc.RPC: Server at /
>> 17.229.23.77:9000 not available yet, Zzzzz...
>>
>> ....... and this retyring gets on repeating
>>
>>
>> The hadoop-site.xmls are like this -
>>
>> 1. In machine 1
>> -
>> <configuration>
>>
>> <property>
>> <name>fs.default.name</name>
>> <value>hdfs://localhost:9000</value>
>> </property>
>>
>> <property>
>> <name>dfs.name.dir</name>
>> <value>/Users/souravm/hdpn</value>
>> </property>
>>
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>localhost:9001</value>
>> </property>
>> <property>
>> <name>dfs.replication</name>
>> <value>1</value>
>> </property>
>> </configuration>
>>
>>
>> 2. In machine 2
>>
>> <configuration>
>>
>> <property>
>> <name>fs.default.name</name>
>> <value>hdfs://<machine1 ip>:9000</value>
>> </property>
>> <property>
>> <name>dfs.data.dir</name>
>> <value>/Users/nirdosh/hdfsd1</value>
>> </property>
>> <property>
>> <name>dfs.replication</name>
>> <value>1</value>
>> </property>
>> </configuration>
>>
>> The slaves file in machine 1 has single entry - <user name>@<ip of
>> machine2>
>>
>> The exact steps I did -
>>
>> 1. Reformat the namenode in machine 1
>> 2. execute start-dfs.sh in machine 1
>> 3. Then I try to see whether the datanode is created through http://<machine
>> 1 ip>:50070
>>
>> Any pointer to resolve this issue would be appreciated.
>>
>> Regards,
>> Sourav
>>
>>
>>
>> **************** CAUTION - Disclaimer *****************
>> This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended
>> solely
>> for the use of the addressee(s). If you are not the intended recipient,
>> please
>> notify the sender by e-mail and delete the original message. Further, you
>> are not
>> to copy, disclose, or distribute this e-mail or its contents to any other
>> person and
>> any such actions are unlawful. This e-mail may contain viruses. Infosys has
>> taken
>> every reasonable precaution to minimize this risk, but is not liable for
>> any damage
>> you may sustain as a result of any virus in this e-mail. You should carry
>> out your
>> own virus checks before opening the e-mail or attachment. Infosys reserves
>> the
>> right to monitor and review the content of all messages sent to or from
>> this e-mail
>> address. Messages sent to or from this e-mail address may be stored on the
>> Infosys e-mail system.
>> ***INFOSYS******** End of Disclaimer ********INFOSYS***
>>
>
>
>
> --
> Mafish@gmail.com
> Institute of Computing Technology, Chinese Academy of Sciences, Beijing.