You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Lipets Vladimir <li...@cs.bgu.ac.il> on 2008/09/11 14:32:18 UTC
Hadoop IPC problem
I discovered the following problem trying setting up Hadoop cluster.
DataNode fails to connect with a NameNode, but
* It happened only with a DataNode connects from remote host, however local DataNode connects properly
-------------------------------------------------------
2008-09-09 02:58:20,075 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: kariba.haifa.ibm.com/9.148.41.240:50000. Already tried 9 time(s).
2008-09-09 02:58:20,077 ERROR org.apache.hadoop.dfs.DataNode: java.io.IOException: Call failed on local exception
at org.apache.hadoop.ipc.Client.call(Client.java:718)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:319)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:306)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:343)
at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:288)
at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:244)
at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:190)
at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:2987)
at org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:2942)
at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2950)
at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3072)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:300)
at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:177)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:789)
at org.apache.hadoop.ipc.Client.call(Client.java:704)
... 12 more
-----------------------------------------
* There is no firewall installed, for. example simple echo server works fine on any ports
* Moreover, even telnet from remote computer to the port listened by the NameNode fails, when local connection causes warning (in the log)
How this may happen? How to fix it?
Thank you in advance,
Re: Hadoop IPC problem
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
You should check the value of the configuration field
"fs.default.name"
on the remote data-node.
It should be a valid address and port # of the name-node.
Lipets Vladimir wrote:
> I discovered the following problem trying setting up Hadoop cluster.
>
> DataNode fails to connect with a NameNode, but
>
> * It happened only with a DataNode connects from remote host, however local DataNode connects properly
>
> -------------------------------------------------------
> 2008-09-09 02:58:20,075 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: kariba.haifa.ibm.com/9.148.41.240:50000. Already tried 9 time(s).
> 2008-09-09 02:58:20,077 ERROR org.apache.hadoop.dfs.DataNode: java.io.IOException: Call failed on local exception
> at org.apache.hadoop.ipc.Client.call(Client.java:718)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
> at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:319)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:306)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:343)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:288)
> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:244)
> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:190)
> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:2987)
> at org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:2942)
> at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2950)
> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3072)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
> at sun.nio.ch.SocketAdaptor.connect(Unknown Source)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:300)
> at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:177)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:789)
> at org.apache.hadoop.ipc.Client.call(Client.java:704)
> ... 12 more
> -----------------------------------------
>
> * There is no firewall installed, for. example simple echo server works fine on any ports
> * Moreover, even telnet from remote computer to the port listened by the NameNode fails, when local connection causes warning (in the log)
>
> How this may happen? How to fix it?
>
> Thank you in advance,
>
>
>