You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by asif md <as...@gmail.com> on 2009/06/04 21:39:08 UTC
Cluster Setup Issues : Datanode not being initialized.
Hello all,
I'm trying to setup a two node cluster < remote > using the following
tutorials
{ NOTE : i'm ignoring the tmp directory property in hadoop-site.xml
suggested by Michael }
Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G.
Noll<http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29>
Running Hadoop On Ubuntu Linux (Multi-Node Cluster) - Michael G.
Noll<http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29>
I get the following logs when try to run $HADOOP_HOME/bin/start-dfs.sh at
master.
***************************************************************************************************
AT MASTER :
-----------------
2009-06-04 12:16:30,864 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = *******
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.18.3
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250;
compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
************************************************************/
2009-06-04 12:16:31,071 ERROR org.apache.hadoop.dfs.DataNode:
java.io.IOException: Incompatible namespaceIDs in /tmp/*****/dfs/data:
namenode namespaceID = 34351921; datanode namespaceID = 539590337
at
org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:226)
at
org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:141)
at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:306)
at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
at
org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
2009-06-04 12:16:31,071 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
/************************************************************
*******************************************************************************************************
AT SLAVE :
----------------
2009-06-04 12:16:28,203 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = ****************
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.18.3
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250;
compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
************************************************************/
2009-06-04 12:16:32,175 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 0 time(s).
2009-06-04 12:16:33,178 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 1 time(s).
2009-06-04 12:16:34,181 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 2 time(s).
2009-06-04 12:16:35,184 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 3 time(s).
2009-06-04 12:16:36,187 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 4 time(s).
2009-06-04 12:16:37,190 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 5 time(s).
2009-06-04 12:16:38,193 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 6 time(s).
2009-06-04 12:16:39,196 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 7 time(s).
2009-06-04 12:16:40,198 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 8 time(s).
2009-06-04 12:16:41,200 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 9 time(s).
2009-06-04 12:16:41,222 ERROR org.apache.hadoop.dfs.DataNode:
java.io.IOException: Call to master/198.55.35.229:54310 failed on local
exception: java.net.NoRouteToHostException: No route to host
at org.apache.hadoop.ipc.Client.wrapException(Client.java:751)
at org.apache.hadoop.ipc.Client.call(Client.java:719)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:309)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:286)
at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:277)
at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
at
org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)
at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)
at org.apache.hadoop.ipc.Client.call(Client.java:705)
... 13 more
2009-06-04 12:16:41,222 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at
opencirrus-1262.hpl.hp.com/198.55.36.243
************************************************************/
PLEASE COMMENT.
Thanks.
Asif.
Re: Cluster Setup Issues : Datanode not being initialized.
Posted by asif md <as...@gmail.com>.
@ Ravi.
Not able to do that.
On Thu, Jun 4, 2009 at 5:38 PM, Raghu Angadi <ra...@yahoo-inc.com> wrote:
>
> Did you try 'telnet 198.55.35.229 54310' from this datanode? The log show
> that it is not able to connect to "master:54310". ssh from datanode does not
> matter.
>
> Raghu.
>
> asif md wrote:
>
>> I can SSH both ways .i.e. From master to slave and slave to master.
>>
>> the datanode is getting intialized at master but the log at slave looks
>> like
>> this
>>
>> ************************************************************/
>> 2009-06-04 15:20:06,066 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting DataNode
>> STARTUP_MSG: host = ************
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 0.18.3
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r
>> 736250;
>> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
>> ************************************************************/
>> 2009-06-04 15:20:08,826 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 0 time(s).
>> 2009-06-04 15:20:09,829 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 1 time(s).
>> 2009-06-04 15:20:10,831 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 2 time(s).
>> 2009-06-04 15:20:11,832 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 3 time(s).
>> 2009-06-04 15:20:12,834 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 4 time(s).
>> 2009-06-04 15:20:13,837 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 5 time(s).
>> 2009-06-04 15:20:14,840 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 6 time(s).
>> 2009-06-04 15:20:15,841 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 7 time(s).
>> 2009-06-04 15:20:16,844 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 8 time(s).
>> 2009-06-04 15:20:17,847 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 9 time(s).
>> 2009-06-04 15:20:17,873 ERROR org.apache.hadoop.dfs.DataNode:
>> java.io.IOException: Call to master/198.55.35.229:54310 failed on local
>> exception: java.net.NoRouteToHostException: No route to host
>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:751)
>> at org.apache.hadoop.ipc.Client.call(Client.java:719)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>> at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source)
>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335)
>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372)
>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:309)
>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:286)
>> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:277)
>> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
>> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
>> at
>> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
>> at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
>> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
>> Caused by: java.net.NoRouteToHostException: No route to host
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
>> at
>> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)
>> at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)
>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)
>> at org.apache.hadoop.ipc.Client.call(Client.java:705)
>> ... 13 more
>>
>> 2009-06-04 15:20:17,874 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down DataNode at *******
>>
>> **********************************************************************************88
>>
>> Please suggest.
>>
>> Asif.
>>
>>
>> On Thu, Jun 4, 2009 at 4:15 PM, asif md <as...@gmail.com> wrote:
>>
>> @Ravi
>>>
>>> thanx ravi .. i'm now using my a definded tmp dir so the second issue is
>>> resolved.
>>>
>>> But i have ssh keys tht have passwords. But i am able to ssh to the slave
>>> and master from the master.
>>>
>>> should i be able to do tht from the slave as well.
>>>
>>> @ALL
>>>
>>> Any suggestions.
>>>
>>> Thanx
>>>
>>> Asif.
>>>
>>>
>>> On Thu, Jun 4, 2009 at 3:17 PM, Ravi Phulari <rphulari@yahoo-inc.com
>>> >wrote:
>>>
>>> From logs looks like your Hadoop cluster is facing two different issues
>>>> .
>>>>
>>>> At Slave
>>>>
>>>> 1. exception: java.net.NoRouteToHostException: No route to host in
>>>>
>>>> your logs
>>>>
>>>> Diagnosis - One of your nodes cannot be reached correctly. Make sure you
>>>> can ssh to your master and slave and passwordless ssh keys are set .
>>>>
>>>> At master
>>>> 2. java.io.IOException: Incompatible namespaceIDs in
>>>> Diagnosis - Your Hadoop namespaceID became corrupted. Unfortunately the
>>>> easiest thing to do reformat the HDFS
>>>>
>>>> As you have not configured hadoop.tmp.dir or data.dir by default Hadoop
>>>> will you /tmp as directory for temporary files , log files and for data
>>>> dir
>>>> which is not good practice .
>>>> I would suggest you using some tmpForHadoop dir somewhere else than
>>>> /tmp
>>>> .
>>>>
>>>> -
>>>> Ravi
>>>>
>>>> On 6/4/09 12:39 PM, "asif md" <as...@gmail.com> wrote:
>>>>
>>>> Hello all,
>>>>
>>>> I'm trying to setup a two node cluster < remote > using the following
>>>> tutorials
>>>> { NOTE : i'm ignoring the tmp directory property in hadoop-site.xml
>>>> suggested by Michael }
>>>>
>>>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G.
>>>> Noll<
>>>>
>>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
>>>> Running Hadoop On Ubuntu Linux (Multi-Node Cluster) - Michael G.
>>>> Noll<
>>>>
>>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
>>>>
>>>> I get the following logs when try to run $HADOOP_HOME/bin/start-dfs.sh
>>>> at
>>>> master.
>>>>
>>>>
>>>>
>>>> ***************************************************************************************************
>>>> AT MASTER :
>>>> -----------------
>>>> 2009-06-04 12:16:30,864 INFO org.apache.hadoop.dfs.DataNode:
>>>> STARTUP_MSG:
>>>> /************************************************************
>>>> STARTUP_MSG: Starting DataNode
>>>> STARTUP_MSG: host = *******
>>>> STARTUP_MSG: args = []
>>>> STARTUP_MSG: version = 0.18.3
>>>> STARTUP_MSG: build =
>>>> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r
>>>> 736250;
>>>> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
>>>> ************************************************************/
>>>> 2009-06-04 12:16:31,071 ERROR org.apache.hadoop.dfs.DataNode:
>>>> java.io.IOException: Incompatible namespaceIDs in /tmp/*****/dfs/data:
>>>> namenode namespaceID = 34351921; datanode namespaceID = 539590337
>>>> at
>>>> org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:226)
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:141)
>>>> at
>>>> org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:306)
>>>> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
>>>> at
>>>> org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
>>>> at
>>>> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
>>>> at
>>>> org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
>>>> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
>>>>
>>>> 2009-06-04 12:16:31,071 INFO org.apache.hadoop.dfs.DataNode:
>>>> SHUTDOWN_MSG:
>>>> /************************************************************
>>>>
>>>>
>>>> *******************************************************************************************************
>>>> AT SLAVE :
>>>> ----------------
>>>> 2009-06-04 12:16:28,203 INFO org.apache.hadoop.dfs.DataNode:
>>>> STARTUP_MSG:
>>>> /************************************************************
>>>> STARTUP_MSG: Starting DataNode
>>>> STARTUP_MSG: host = ****************
>>>> STARTUP_MSG: args = []
>>>> STARTUP_MSG: version = 0.18.3
>>>> STARTUP_MSG: build =
>>>> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r
>>>> 736250;
>>>> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
>>>> ************************************************************/
>>>> 2009-06-04 12:16:32,175 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/198.55.35.229:54310. Already tried 0 time(s).
>>>> 2009-06-04 12:16:33,178 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/198.55.35.229:54310. Already tried 1 time(s).
>>>> 2009-06-04 12:16:34,181 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/198.55.35.229:54310. Already tried 2 time(s).
>>>> 2009-06-04 12:16:35,184 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/198.55.35.229:54310. Already tried 3 time(s).
>>>> 2009-06-04 12:16:36,187 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/198.55.35.229:54310. Already tried 4 time(s).
>>>> 2009-06-04 12:16:37,190 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/198.55.35.229:54310. Already tried 5 time(s).
>>>> 2009-06-04 12:16:38,193 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/198.55.35.229:54310. Already tried 6 time(s).
>>>> 2009-06-04 12:16:39,196 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/198.55.35.229:54310. Already tried 7 time(s).
>>>> 2009-06-04 12:16:40,198 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/198.55.35.229:54310. Already tried 8 time(s).
>>>> 2009-06-04 12:16:41,200 INFO org.apache.hadoop.ipc.Client: Retrying
>>>> connect
>>>> to server: master/198.55.35.229:54310. Already tried 9 time(s).
>>>> 2009-06-04 12:16:41,222 ERROR org.apache.hadoop.dfs.DataNode:
>>>> java.io.IOException: Call to master/198.55.35.229:54310 failed on local
>>>> exception: java.net.NoRouteToHostException: No route to host
>>>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:751)
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:719)
>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>>>> at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source)
>>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
>>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335)
>>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372)
>>>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:309)
>>>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:286)
>>>> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:277)
>>>> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
>>>> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
>>>> at
>>>> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
>>>> at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
>>>> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
>>>> Caused by: java.net.NoRouteToHostException: No route to host
>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>>> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
>>>> at
>>>> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)
>>>> at
>>>> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)
>>>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:705)
>>>> ... 13 more
>>>>
>>>> 2009-06-04 12:16:41,222 INFO org.apache.hadoop.dfs.DataNode:
>>>> SHUTDOWN_MSG:
>>>> /************************************************************
>>>> SHUTDOWN_MSG: Shutting down DataNode at
>>>> opencirrus-1262.hpl.hp.com/198.55.36.243
>>>> ************************************************************/
>>>>
>>>>
>>>>
>>>> PLEASE COMMENT.
>>>>
>>>> Thanks.
>>>>
>>>> Asif.
>>>>
>>>>
>>>> Ravi
>>>> --
>>>>
>>>>
>>>>
>>
>
Re: Cluster Setup Issues : Datanode not being initialized.
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Did you try 'telnet 198.55.35.229 54310' from this datanode? The log
show that it is not able to connect to "master:54310". ssh from datanode
does not matter.
Raghu.
asif md wrote:
> I can SSH both ways .i.e. From master to slave and slave to master.
>
> the datanode is getting intialized at master but the log at slave looks like
> this
>
> ************************************************************/
> 2009-06-04 15:20:06,066 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = ************
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.18.3
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250;
> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
> ************************************************************/
> 2009-06-04 15:20:08,826 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 0 time(s).
> 2009-06-04 15:20:09,829 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 1 time(s).
> 2009-06-04 15:20:10,831 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 2 time(s).
> 2009-06-04 15:20:11,832 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 3 time(s).
> 2009-06-04 15:20:12,834 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 4 time(s).
> 2009-06-04 15:20:13,837 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 5 time(s).
> 2009-06-04 15:20:14,840 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 6 time(s).
> 2009-06-04 15:20:15,841 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 7 time(s).
> 2009-06-04 15:20:16,844 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 8 time(s).
> 2009-06-04 15:20:17,847 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 9 time(s).
> 2009-06-04 15:20:17,873 ERROR org.apache.hadoop.dfs.DataNode:
> java.io.IOException: Call to master/198.55.35.229:54310 failed on local
> exception: java.net.NoRouteToHostException: No route to host
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:751)
> at org.apache.hadoop.ipc.Client.call(Client.java:719)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
> at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:309)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:286)
> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:277)
> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
> at
> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
> at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
> Caused by: java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)
> at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)
> at org.apache.hadoop.ipc.Client.call(Client.java:705)
> ... 13 more
>
> 2009-06-04 15:20:17,874 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at *******
> **********************************************************************************88
>
> Please suggest.
>
> Asif.
>
>
> On Thu, Jun 4, 2009 at 4:15 PM, asif md <as...@gmail.com> wrote:
>
>> @Ravi
>>
>> thanx ravi .. i'm now using my a definded tmp dir so the second issue is
>> resolved.
>>
>> But i have ssh keys tht have passwords. But i am able to ssh to the slave
>> and master from the master.
>>
>> should i be able to do tht from the slave as well.
>>
>> @ALL
>>
>> Any suggestions.
>>
>> Thanx
>>
>> Asif.
>>
>>
>> On Thu, Jun 4, 2009 at 3:17 PM, Ravi Phulari <rp...@yahoo-inc.com>wrote:
>>
>>> From logs looks like your Hadoop cluster is facing two different issues
>>> .
>>>
>>> At Slave
>>>
>>> 1. exception: java.net.NoRouteToHostException: No route to host in
>>> your logs
>>>
>>> Diagnosis - One of your nodes cannot be reached correctly. Make sure you
>>> can ssh to your master and slave and passwordless ssh keys are set .
>>>
>>> At master
>>> 2. java.io.IOException: Incompatible namespaceIDs in
>>> Diagnosis - Your Hadoop namespaceID became corrupted. Unfortunately the
>>> easiest thing to do reformat the HDFS
>>>
>>> As you have not configured hadoop.tmp.dir or data.dir by default Hadoop
>>> will you /tmp as directory for temporary files , log files and for data dir
>>> which is not good practice .
>>> I would suggest you using some tmpForHadoop dir somewhere else than /tmp
>>> .
>>>
>>> -
>>> Ravi
>>>
>>> On 6/4/09 12:39 PM, "asif md" <as...@gmail.com> wrote:
>>>
>>> Hello all,
>>>
>>> I'm trying to setup a two node cluster < remote > using the following
>>> tutorials
>>> { NOTE : i'm ignoring the tmp directory property in hadoop-site.xml
>>> suggested by Michael }
>>>
>>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G.
>>> Noll<
>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
>>> Running Hadoop On Ubuntu Linux (Multi-Node Cluster) - Michael G.
>>> Noll<
>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
>>>
>>> I get the following logs when try to run $HADOOP_HOME/bin/start-dfs.sh at
>>> master.
>>>
>>>
>>> ***************************************************************************************************
>>> AT MASTER :
>>> -----------------
>>> 2009-06-04 12:16:30,864 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = *******
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 0.18.3
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r
>>> 736250;
>>> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
>>> ************************************************************/
>>> 2009-06-04 12:16:31,071 ERROR org.apache.hadoop.dfs.DataNode:
>>> java.io.IOException: Incompatible namespaceIDs in /tmp/*****/dfs/data:
>>> namenode namespaceID = 34351921; datanode namespaceID = 539590337
>>> at
>>> org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:226)
>>> at
>>>
>>> org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:141)
>>> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:306)
>>> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
>>> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
>>> at
>>> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
>>> at
>>> org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
>>> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
>>>
>>> 2009-06-04 12:16:31,071 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>>
>>> *******************************************************************************************************
>>> AT SLAVE :
>>> ----------------
>>> 2009-06-04 12:16:28,203 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = ****************
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 0.18.3
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r
>>> 736250;
>>> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
>>> ************************************************************/
>>> 2009-06-04 12:16:32,175 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect
>>> to server: master/198.55.35.229:54310. Already tried 0 time(s).
>>> 2009-06-04 12:16:33,178 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect
>>> to server: master/198.55.35.229:54310. Already tried 1 time(s).
>>> 2009-06-04 12:16:34,181 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect
>>> to server: master/198.55.35.229:54310. Already tried 2 time(s).
>>> 2009-06-04 12:16:35,184 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect
>>> to server: master/198.55.35.229:54310. Already tried 3 time(s).
>>> 2009-06-04 12:16:36,187 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect
>>> to server: master/198.55.35.229:54310. Already tried 4 time(s).
>>> 2009-06-04 12:16:37,190 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect
>>> to server: master/198.55.35.229:54310. Already tried 5 time(s).
>>> 2009-06-04 12:16:38,193 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect
>>> to server: master/198.55.35.229:54310. Already tried 6 time(s).
>>> 2009-06-04 12:16:39,196 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect
>>> to server: master/198.55.35.229:54310. Already tried 7 time(s).
>>> 2009-06-04 12:16:40,198 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect
>>> to server: master/198.55.35.229:54310. Already tried 8 time(s).
>>> 2009-06-04 12:16:41,200 INFO org.apache.hadoop.ipc.Client: Retrying
>>> connect
>>> to server: master/198.55.35.229:54310. Already tried 9 time(s).
>>> 2009-06-04 12:16:41,222 ERROR org.apache.hadoop.dfs.DataNode:
>>> java.io.IOException: Call to master/198.55.35.229:54310 failed on local
>>> exception: java.net.NoRouteToHostException: No route to host
>>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:751)
>>> at org.apache.hadoop.ipc.Client.call(Client.java:719)
>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>>> at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source)
>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335)
>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372)
>>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:309)
>>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:286)
>>> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:277)
>>> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
>>> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
>>> at
>>> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
>>> at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
>>> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
>>> Caused by: java.net.NoRouteToHostException: No route to host
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> at
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
>>> at
>>> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)
>>> at
>>> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)
>>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)
>>> at org.apache.hadoop.ipc.Client.call(Client.java:705)
>>> ... 13 more
>>>
>>> 2009-06-04 12:16:41,222 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at
>>> opencirrus-1262.hpl.hp.com/198.55.36.243
>>> ************************************************************/
>>>
>>>
>>>
>>> PLEASE COMMENT.
>>>
>>> Thanks.
>>>
>>> Asif.
>>>
>>>
>>> Ravi
>>> --
>>>
>>>
>
Re: Cluster Setup Issues : Datanode not being initialized.
Posted by asif md <as...@gmail.com>.
I can SSH both ways .i.e. From master to slave and slave to master.
the datanode is getting intialized at master but the log at slave looks like
this
************************************************************/
2009-06-04 15:20:06,066 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = ************
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.18.3
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250;
compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
************************************************************/
2009-06-04 15:20:08,826 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 0 time(s).
2009-06-04 15:20:09,829 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 1 time(s).
2009-06-04 15:20:10,831 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 2 time(s).
2009-06-04 15:20:11,832 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 3 time(s).
2009-06-04 15:20:12,834 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 4 time(s).
2009-06-04 15:20:13,837 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 5 time(s).
2009-06-04 15:20:14,840 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 6 time(s).
2009-06-04 15:20:15,841 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 7 time(s).
2009-06-04 15:20:16,844 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 8 time(s).
2009-06-04 15:20:17,847 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 9 time(s).
2009-06-04 15:20:17,873 ERROR org.apache.hadoop.dfs.DataNode:
java.io.IOException: Call to master/198.55.35.229:54310 failed on local
exception: java.net.NoRouteToHostException: No route to host
at org.apache.hadoop.ipc.Client.wrapException(Client.java:751)
at org.apache.hadoop.ipc.Client.call(Client.java:719)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:309)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:286)
at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:277)
at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
at
org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)
at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)
at org.apache.hadoop.ipc.Client.call(Client.java:705)
... 13 more
2009-06-04 15:20:17,874 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at *******
**********************************************************************************88
Please suggest.
Asif.
On Thu, Jun 4, 2009 at 4:15 PM, asif md <as...@gmail.com> wrote:
> @Ravi
>
> thanx ravi .. i'm now using my a definded tmp dir so the second issue is
> resolved.
>
> But i have ssh keys tht have passwords. But i am able to ssh to the slave
> and master from the master.
>
> should i be able to do tht from the slave as well.
>
> @ALL
>
> Any suggestions.
>
> Thanx
>
> Asif.
>
>
> On Thu, Jun 4, 2009 at 3:17 PM, Ravi Phulari <rp...@yahoo-inc.com>wrote:
>
>> From logs looks like your Hadoop cluster is facing two different issues
>> .
>>
>> At Slave
>>
>> 1. exception: java.net.NoRouteToHostException: No route to host in
>> your logs
>>
>> Diagnosis - One of your nodes cannot be reached correctly. Make sure you
>> can ssh to your master and slave and passwordless ssh keys are set .
>>
>> At master
>> 2. java.io.IOException: Incompatible namespaceIDs in
>> Diagnosis - Your Hadoop namespaceID became corrupted. Unfortunately the
>> easiest thing to do reformat the HDFS
>>
>> As you have not configured hadoop.tmp.dir or data.dir by default Hadoop
>> will you /tmp as directory for temporary files , log files and for data dir
>> which is not good practice .
>> I would suggest you using some tmpForHadoop dir somewhere else than /tmp
>> .
>>
>> -
>> Ravi
>>
>> On 6/4/09 12:39 PM, "asif md" <as...@gmail.com> wrote:
>>
>> Hello all,
>>
>> I'm trying to setup a two node cluster < remote > using the following
>> tutorials
>> { NOTE : i'm ignoring the tmp directory property in hadoop-site.xml
>> suggested by Michael }
>>
>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G.
>> Noll<
>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
>> >
>> Running Hadoop On Ubuntu Linux (Multi-Node Cluster) - Michael G.
>> Noll<
>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
>> >
>>
>>
>> I get the following logs when try to run $HADOOP_HOME/bin/start-dfs.sh at
>> master.
>>
>>
>> ***************************************************************************************************
>> AT MASTER :
>> -----------------
>> 2009-06-04 12:16:30,864 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting DataNode
>> STARTUP_MSG: host = *******
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 0.18.3
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r
>> 736250;
>> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
>> ************************************************************/
>> 2009-06-04 12:16:31,071 ERROR org.apache.hadoop.dfs.DataNode:
>> java.io.IOException: Incompatible namespaceIDs in /tmp/*****/dfs/data:
>> namenode namespaceID = 34351921; datanode namespaceID = 539590337
>> at
>> org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:226)
>> at
>>
>> org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:141)
>> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:306)
>> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
>> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
>> at
>> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
>> at
>> org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
>> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
>>
>> 2009-06-04 12:16:31,071 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
>> /************************************************************
>>
>> *******************************************************************************************************
>> AT SLAVE :
>> ----------------
>> 2009-06-04 12:16:28,203 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting DataNode
>> STARTUP_MSG: host = ****************
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 0.18.3
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r
>> 736250;
>> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
>> ************************************************************/
>> 2009-06-04 12:16:32,175 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 0 time(s).
>> 2009-06-04 12:16:33,178 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 1 time(s).
>> 2009-06-04 12:16:34,181 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 2 time(s).
>> 2009-06-04 12:16:35,184 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 3 time(s).
>> 2009-06-04 12:16:36,187 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 4 time(s).
>> 2009-06-04 12:16:37,190 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 5 time(s).
>> 2009-06-04 12:16:38,193 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 6 time(s).
>> 2009-06-04 12:16:39,196 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 7 time(s).
>> 2009-06-04 12:16:40,198 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 8 time(s).
>> 2009-06-04 12:16:41,200 INFO org.apache.hadoop.ipc.Client: Retrying
>> connect
>> to server: master/198.55.35.229:54310. Already tried 9 time(s).
>> 2009-06-04 12:16:41,222 ERROR org.apache.hadoop.dfs.DataNode:
>> java.io.IOException: Call to master/198.55.35.229:54310 failed on local
>> exception: java.net.NoRouteToHostException: No route to host
>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:751)
>> at org.apache.hadoop.ipc.Client.call(Client.java:719)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>> at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source)
>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335)
>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372)
>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:309)
>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:286)
>> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:277)
>> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
>> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
>> at
>> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
>> at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
>> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
>> Caused by: java.net.NoRouteToHostException: No route to host
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
>> at
>> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)
>> at
>> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)
>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)
>> at org.apache.hadoop.ipc.Client.call(Client.java:705)
>> ... 13 more
>>
>> 2009-06-04 12:16:41,222 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down DataNode at
>> opencirrus-1262.hpl.hp.com/198.55.36.243
>> ************************************************************/
>>
>>
>>
>> PLEASE COMMENT.
>>
>> Thanks.
>>
>> Asif.
>>
>>
>> Ravi
>> --
>>
>>
>
Re: Cluster Setup Issues : Datanode not being initialized.
Posted by asif md <as...@gmail.com>.
@Ravi
thanx ravi .. i'm now using my a definded tmp dir so the second issue is
resolved.
But i have ssh keys tht have passwords. But i am able to ssh to the slave
and master from the master.
should i be able to do tht from the slave as well.
@ALL
Any suggestions.
Thanx
Asif.
On Thu, Jun 4, 2009 at 3:17 PM, Ravi Phulari <rp...@yahoo-inc.com> wrote:
> From logs looks like your Hadoop cluster is facing two different issues .
>
>
> At Slave
>
> 1. exception: java.net.NoRouteToHostException: No route to host in your
> logs
>
> Diagnosis - One of your nodes cannot be reached correctly. Make sure you
> can ssh to your master and slave and passwordless ssh keys are set .
>
> At master
> 2. java.io.IOException: Incompatible namespaceIDs in
> Diagnosis - Your Hadoop namespaceID became corrupted. Unfortunately the
> easiest thing to do reformat the HDFS
>
> As you have not configured hadoop.tmp.dir or data.dir by default Hadoop
> will you /tmp as directory for temporary files , log files and for data dir
> which is not good practice .
> I would suggest you using some tmpForHadoop dir somewhere else than /tmp .
>
> -
> Ravi
>
> On 6/4/09 12:39 PM, "asif md" <as...@gmail.com> wrote:
>
> Hello all,
>
> I'm trying to setup a two node cluster < remote > using the following
> tutorials
> { NOTE : i'm ignoring the tmp directory property in hadoop-site.xml
> suggested by Michael }
>
> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G.
> Noll<
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
> >
> Running Hadoop On Ubuntu Linux (Multi-Node Cluster) - Michael G.
> Noll<
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
> >
>
>
> I get the following logs when try to run $HADOOP_HOME/bin/start-dfs.sh at
> master.
>
>
> ***************************************************************************************************
> AT MASTER :
> -----------------
> 2009-06-04 12:16:30,864 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = *******
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.18.3
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r
> 736250;
> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
> ************************************************************/
> 2009-06-04 12:16:31,071 ERROR org.apache.hadoop.dfs.DataNode:
> java.io.IOException: Incompatible namespaceIDs in /tmp/*****/dfs/data:
> namenode namespaceID = 34351921; datanode namespaceID = 539590337
> at
> org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:226)
> at
>
> org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:141)
> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:306)
> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
> at
> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
> at
> org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
>
> 2009-06-04 12:16:31,071 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
> /************************************************************
>
> *******************************************************************************************************
> AT SLAVE :
> ----------------
> 2009-06-04 12:16:28,203 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = ****************
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 0.18.3
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r
> 736250;
> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
> ************************************************************/
> 2009-06-04 12:16:32,175 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 0 time(s).
> 2009-06-04 12:16:33,178 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 1 time(s).
> 2009-06-04 12:16:34,181 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 2 time(s).
> 2009-06-04 12:16:35,184 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 3 time(s).
> 2009-06-04 12:16:36,187 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 4 time(s).
> 2009-06-04 12:16:37,190 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 5 time(s).
> 2009-06-04 12:16:38,193 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 6 time(s).
> 2009-06-04 12:16:39,196 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 7 time(s).
> 2009-06-04 12:16:40,198 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 8 time(s).
> 2009-06-04 12:16:41,200 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/198.55.35.229:54310. Already tried 9 time(s).
> 2009-06-04 12:16:41,222 ERROR org.apache.hadoop.dfs.DataNode:
> java.io.IOException: Call to master/198.55.35.229:54310 failed on local
> exception: java.net.NoRouteToHostException: No route to host
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:751)
> at org.apache.hadoop.ipc.Client.call(Client.java:719)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
> at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:309)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:286)
> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:277)
> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
> at
> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
> at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
> Caused by: java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)
> at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)
> at org.apache.hadoop.ipc.Client.call(Client.java:705)
> ... 13 more
>
> 2009-06-04 12:16:41,222 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at
> opencirrus-1262.hpl.hp.com/198.55.36.243
> ************************************************************/
>
>
>
> PLEASE COMMENT.
>
> Thanks.
>
> Asif.
>
>
> Ravi
> --
>
>
Re: Cluster Setup Issues : Datanode not being initialized.
Posted by Ravi Phulari <rp...@yahoo-inc.com>.
>From logs looks like your Hadoop cluster is facing two different issues .
At Slave
1. exception: java.net.NoRouteToHostException: No route to host in your logs
Diagnosis - One of your nodes cannot be reached correctly. Make sure you can ssh to your master and slave and passwordless ssh keys are set .
At master
2. java.io.IOException: Incompatible namespaceIDs in
Diagnosis - Your Hadoop namespaceID became corrupted. Unfortunately the easiest thing to do reformat the HDFS
As you have not configured hadoop.tmp.dir or data.dir by default Hadoop will you /tmp as directory for temporary files , log files and for data dir which is not good practice .
I would suggest you using some tmpForHadoop dir somewhere else than /tmp .
-
Ravi
On 6/4/09 12:39 PM, "asif md" <as...@gmail.com> wrote:
Hello all,
I'm trying to setup a two node cluster < remote > using the following
tutorials
{ NOTE : i'm ignoring the tmp directory property in hadoop-site.xml
suggested by Michael }
Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G.
Noll<http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29>
Running Hadoop On Ubuntu Linux (Multi-Node Cluster) - Michael G.
Noll<http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29>
I get the following logs when try to run $HADOOP_HOME/bin/start-dfs.sh at
master.
***************************************************************************************************
AT MASTER :
-----------------
2009-06-04 12:16:30,864 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = *******
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.18.3
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250;
compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
************************************************************/
2009-06-04 12:16:31,071 ERROR org.apache.hadoop.dfs.DataNode:
java.io.IOException: Incompatible namespaceIDs in /tmp/*****/dfs/data:
namenode namespaceID = 34351921; datanode namespaceID = 539590337
at
org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:226)
at
org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:141)
at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:306)
at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
at
org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
2009-06-04 12:16:31,071 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
/************************************************************
*******************************************************************************************************
AT SLAVE :
----------------
2009-06-04 12:16:28,203 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = ****************
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.18.3
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250;
compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
************************************************************/
2009-06-04 12:16:32,175 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 0 time(s).
2009-06-04 12:16:33,178 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 1 time(s).
2009-06-04 12:16:34,181 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 2 time(s).
2009-06-04 12:16:35,184 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 3 time(s).
2009-06-04 12:16:36,187 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 4 time(s).
2009-06-04 12:16:37,190 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 5 time(s).
2009-06-04 12:16:38,193 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 6 time(s).
2009-06-04 12:16:39,196 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 7 time(s).
2009-06-04 12:16:40,198 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 8 time(s).
2009-06-04 12:16:41,200 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/198.55.35.229:54310. Already tried 9 time(s).
2009-06-04 12:16:41,222 ERROR org.apache.hadoop.dfs.DataNode:
java.io.IOException: Call to master/198.55.35.229:54310 failed on local
exception: java.net.NoRouteToHostException: No route to host
at org.apache.hadoop.ipc.Client.wrapException(Client.java:751)
at org.apache.hadoop.ipc.Client.call(Client.java:719)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:309)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:286)
at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:277)
at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071)
at
org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026)
at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034)
at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156)
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)
at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)
at org.apache.hadoop.ipc.Client.call(Client.java:705)
... 13 more
2009-06-04 12:16:41,222 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at
opencirrus-1262.hpl.hp.com/198.55.36.243
************************************************************/
PLEASE COMMENT.
Thanks.
Asif.
Ravi
--