You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by xeon Mailinglist <xe...@gmail.com> on 2014/02/18 13:02:00 UTC

Set Hadoop MRv2 behind a NAT

I am trying to set Hadoop MapReduce (MRv2) behind the NAT, but when I try
to connect the Datanode, I get the error below.

The hosts have 2 interfaces, one with a private address and another with
the NAT address. To access the host with SSH, I must use an external IP,
that NAT server will redirect.

I want to access MRv2 from outside,  and for that I tried to set an NAT'd
IP, but the Datanado don't starts. How I set the MRv2 address in the
configuration files, so that I can put Hadoop running?

2014-02-18 11:55:43,105 FATAL
org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
block pool Block pool
 BP-1302615141-172.16.100.1-1392724171451 (storage id
DS-1964144366-172.16.100.2-50010-1392724135477) service to /10.103.0.11:
9000
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException):
Datanode denied com
munication with namenode: DatanodeRegistration(0.0.0.0,
storageID=DS-1964144366-172.16.100.2-50010-1392724135477, infoPort=500
75, ipcPort=50020,
storageInfo=lv=-40;cid=CID-f45cf960-4e55-420b-a20c-43f6edf1a847;nsid=1117538035;c=0)
        at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:631)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3398)
        at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:881)
        at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:90)
        at
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:18295)
        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
        at java.security.AccessController.doPrivileged(Native Method)
:

RE: Set Hadoop MRv2 behind a NAT

Posted by Brahma Reddy Battula <br...@huawei.com>.
Seems to be /etc/hosts mapping is missed b/w Namenode and DataNode machine..Please check hostmapping is done with NAT'd IP or not...





DataNode IP : 172.16.100.2

Namenode IP : 10.103.0.11<http://10.103.0.11>



Need to configure with above IP's







Thanks & Regards



Brahma Reddy Battula



________________________________

From: xeon Mailinglist [xeonmailinglist@gmail.com]
Sent: Tuesday, February 18, 2014 5:32 PM
To: user@hadoop.apache.org
Subject: Set Hadoop MRv2 behind a NAT

I am trying to set Hadoop MapReduce (MRv2) behind the NAT, but when I try to connect the Datanode, I get the error below.

The hosts have 2 interfaces, one with a private address and another with the NAT address. To access the host with SSH, I must use an external IP, that NAT server will redirect.

I want to access MRv2 from outside,  and for that I tried to set an NAT'd IP, but the Datanado don't starts. How I set the MRv2 address in the configuration files, so that I can put Hadoop running?

2014-02-18 11:55:43,105 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool
 BP-1302615141-172.16.100.1-1392724171451 (storage id DS-1964144366-172.16.100.2-50010-1392724135477) service to /10.103.0.11<http://10.103.0.11>:
9000
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied com
munication with namenode: DatanodeRegistration(0.0.0.0, storageID=DS-1964144366-172.16.100.2-50010-1392724135477, infoPort=500
75, ipcPort=50020, storageInfo=lv=-40;cid=CID-f45cf960-4e55-420b-a20c-43f6edf1a847;nsid=1117538035;c=0)
        at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:631)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3398)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:881)
        at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:90)
        at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:18295)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
        at java.security.AccessController.doPrivileged(Native Method)
:

RE: Set Hadoop MRv2 behind a NAT

Posted by Brahma Reddy Battula <br...@huawei.com>.
Seems to be /etc/hosts mapping is missed b/w Namenode and DataNode machine..Please check hostmapping is done with NAT'd IP or not...





DataNode IP : 172.16.100.2

Namenode IP : 10.103.0.11<http://10.103.0.11>



Need to configure with above IP's







Thanks & Regards



Brahma Reddy Battula



________________________________

From: xeon Mailinglist [xeonmailinglist@gmail.com]
Sent: Tuesday, February 18, 2014 5:32 PM
To: user@hadoop.apache.org
Subject: Set Hadoop MRv2 behind a NAT

I am trying to set Hadoop MapReduce (MRv2) behind the NAT, but when I try to connect the Datanode, I get the error below.

The hosts have 2 interfaces, one with a private address and another with the NAT address. To access the host with SSH, I must use an external IP, that NAT server will redirect.

I want to access MRv2 from outside,  and for that I tried to set an NAT'd IP, but the Datanado don't starts. How I set the MRv2 address in the configuration files, so that I can put Hadoop running?

2014-02-18 11:55:43,105 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool
 BP-1302615141-172.16.100.1-1392724171451 (storage id DS-1964144366-172.16.100.2-50010-1392724135477) service to /10.103.0.11<http://10.103.0.11>:
9000
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied com
munication with namenode: DatanodeRegistration(0.0.0.0, storageID=DS-1964144366-172.16.100.2-50010-1392724135477, infoPort=500
75, ipcPort=50020, storageInfo=lv=-40;cid=CID-f45cf960-4e55-420b-a20c-43f6edf1a847;nsid=1117538035;c=0)
        at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:631)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3398)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:881)
        at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:90)
        at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:18295)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
        at java.security.AccessController.doPrivileged(Native Method)
:

RE: Set Hadoop MRv2 behind a NAT

Posted by Brahma Reddy Battula <br...@huawei.com>.
Seems to be /etc/hosts mapping is missed b/w Namenode and DataNode machine..Please check hostmapping is done with NAT'd IP or not...





DataNode IP : 172.16.100.2

Namenode IP : 10.103.0.11<http://10.103.0.11>



Need to configure with above IP's







Thanks & Regards



Brahma Reddy Battula



________________________________

From: xeon Mailinglist [xeonmailinglist@gmail.com]
Sent: Tuesday, February 18, 2014 5:32 PM
To: user@hadoop.apache.org
Subject: Set Hadoop MRv2 behind a NAT

I am trying to set Hadoop MapReduce (MRv2) behind the NAT, but when I try to connect the Datanode, I get the error below.

The hosts have 2 interfaces, one with a private address and another with the NAT address. To access the host with SSH, I must use an external IP, that NAT server will redirect.

I want to access MRv2 from outside,  and for that I tried to set an NAT'd IP, but the Datanado don't starts. How I set the MRv2 address in the configuration files, so that I can put Hadoop running?

2014-02-18 11:55:43,105 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool
 BP-1302615141-172.16.100.1-1392724171451 (storage id DS-1964144366-172.16.100.2-50010-1392724135477) service to /10.103.0.11<http://10.103.0.11>:
9000
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied com
munication with namenode: DatanodeRegistration(0.0.0.0, storageID=DS-1964144366-172.16.100.2-50010-1392724135477, infoPort=500
75, ipcPort=50020, storageInfo=lv=-40;cid=CID-f45cf960-4e55-420b-a20c-43f6edf1a847;nsid=1117538035;c=0)
        at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:631)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3398)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:881)
        at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:90)
        at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:18295)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
        at java.security.AccessController.doPrivileged(Native Method)
:

RE: Set Hadoop MRv2 behind a NAT

Posted by Brahma Reddy Battula <br...@huawei.com>.
Seems to be /etc/hosts mapping is missed b/w Namenode and DataNode machine..Please check hostmapping is done with NAT'd IP or not...





DataNode IP : 172.16.100.2

Namenode IP : 10.103.0.11<http://10.103.0.11>



Need to configure with above IP's







Thanks & Regards



Brahma Reddy Battula



________________________________

From: xeon Mailinglist [xeonmailinglist@gmail.com]
Sent: Tuesday, February 18, 2014 5:32 PM
To: user@hadoop.apache.org
Subject: Set Hadoop MRv2 behind a NAT

I am trying to set Hadoop MapReduce (MRv2) behind the NAT, but when I try to connect the Datanode, I get the error below.

The hosts have 2 interfaces, one with a private address and another with the NAT address. To access the host with SSH, I must use an external IP, that NAT server will redirect.

I want to access MRv2 from outside,  and for that I tried to set an NAT'd IP, but the Datanado don't starts. How I set the MRv2 address in the configuration files, so that I can put Hadoop running?

2014-02-18 11:55:43,105 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool
 BP-1302615141-172.16.100.1-1392724171451 (storage id DS-1964144366-172.16.100.2-50010-1392724135477) service to /10.103.0.11<http://10.103.0.11>:
9000
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied com
munication with namenode: DatanodeRegistration(0.0.0.0, storageID=DS-1964144366-172.16.100.2-50010-1392724135477, infoPort=500
75, ipcPort=50020, storageInfo=lv=-40;cid=CID-f45cf960-4e55-420b-a20c-43f6edf1a847;nsid=1117538035;c=0)
        at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:631)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3398)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:881)
        at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:90)
        at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:18295)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
        at java.security.AccessController.doPrivileged(Native Method)
: