You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Michael Lynch <ml...@spacialaudio.com> on 2009/02/13 13:29:09 UTC
Namenode not listening for remote connections to port 9000
Hi,
As far as I can tell I've followed the setup instructions for a hadoop cluster to the letter,
but I find that the datanodes can't connect to the namenode on port 9000 because it is only
listening for connections from localhost.
In my case, the namenode is called centos1, and the datanode is called centos2. They are
centos 5.1 servers with an unmodified sun java 6 runtime.
The log file (logs/hadoop-hadoop-datanode-centos2.log) on the datanode has an infinite loop of the following output:
2009-02-13 13:39:54,528 INFO org.apache.hadoop.ipc.RPC: Server at centos1/192.168.1.42:9000 not available yet, Zzzzz...
2009-02-13 13:39:55,534 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos1/192.168.1.42:9000. Already tried 1 time(s).
2009-02-13 13:39:56,539 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: centos1/192.168.1.42:9000. Already tried 2 time(s).
etcetera...
The log file on (logs/hadoop-hadoop-namenode-centos1.log) seems to show no errors except for:
2009-02-13 13:57:44,258 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9000, call rollEditLog() from 127.0.0.1:1292: error: org.apache.hadoop.dfs.SafeModeException: Checkpoint not created. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically.
But it's my understanding that this safe mode occurs because the datanodes are failing
to connect. So I figure this error is a result and not a cause of my problem (please
correct me if I'm wrong).
Looking at the output of "netstat -nlpt" I get the following:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1626/portmap
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1907/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1947/sendmail: acce
tcp 0 0 0.0.0.0:985 0.0.0.0:* LISTEN 1651/rpc.statd
tcp 0 0 ::ffff:127.0.0.1:9000 :::* LISTEN 5353/java
tcp 0 0 :::1448 :::* LISTEN 5353/java
tcp 0 0 :::50090 :::* LISTEN 5511/java
tcp 0 0 :::50070 :::* LISTEN 5353/java
tcp 0 0 :::22 :::* LISTEN 1892/sshd
tcp 0 0 :::1947 :::* LISTEN 5511/java
My understanding of that output is that the namenode is only listening for connections from
localhost. Obviously this shouldn't be the case. I couldn't find a config option which governs
this, so that leaves me wondering: What am I missing?
Michael
Re: Namenode not listening for remote connections to port 9000
Posted by Michael Lynch <ml...@spacialaudio.com>.
Hmmm - I checked all the /etc/hosts files, and they're all fine. Then I switched the
conf/hadoop-site.xml to specify ip addresses instead of host names. Then oddly enough
it starts working...
Now the funny thing is this: It's fine ssh-ing to the correct machines to start up
datanodes, but when the datanode thread tries to make the connection back to the
namenode (from within a java app I assume) it doesn't resolve the names correctly.
Just looking at this makes me want to think that the namenode does its ssh work
in some non-java way, actually checking the hosts file, while the datanode does
its thing in Java, which doesn't seem to check the hosts file. Could this be some
Java funnyness that it's not checking the hosts file?
Now there's just one sucky thing about my setup: If I change my file with a list
of datanodes (dfs.hosts property) to also have IP addresses instead of hostnames,
then it fails. So parts of my config is specifying hostnames, and other parts are
specifying IP addresses.
Oh well - for development purposes this is good enough, 'cause out in the real world
I won't be using the hosts file to string it all together.
Thanks for the responses.
Mark Kerzner wrote:
> I had a problem that it listened only on 8020, even though I told it to use
> 9000
>
> On Fri, Feb 13, 2009 at 7:50 AM, Norbert Burger <no...@gmail.com>wrote:
>
>> On Fri, Feb 13, 2009 at 8:37 AM, Steve Loughran <st...@apache.org> wrote:
>>
>>> Michael Lynch wrote:
>>>
>>>> Hi,
>>>>
>>>> As far as I can tell I've followed the setup instructions for a hadoop
>>>> cluster to the letter,
>>>> but I find that the datanodes can't connect to the namenode on port 9000
>>>> because it is only
>>>> listening for connections from localhost.
>>>>
>>>> In my case, the namenode is called centos1, and the datanode is called
>>>> centos2. They are
>>>> centos 5.1 servers with an unmodified sun java 6 runtime.
>>>>
>>> fs.default.name takes a URL to the filesystem. such as
>>> hdfs://centos1:9000/
>>>
>>> If the machine is only binding to localhost, that may mean DNS fun. Try a
>>> fully qualified name instead
>>
>> (fs.default.name is defined in conf/hadoop-site.xml, overriding entries
>> from
>> conf/hadoop-default.xml).
>>
>> Also, check your /etc/hosts file on both machines. Could be that you have
>> a
>> incorrect setup where both localhost and the namenode hostname (centos1)
>> are
>> aliased to 127.0.0.1.
>>
>> Norbert
>>
>
Re: Namenode not listening for remote connections to port 9000
Posted by Mark Kerzner <ma...@gmail.com>.
I had a problem that it listened only on 8020, even though I told it to use
9000
On Fri, Feb 13, 2009 at 7:50 AM, Norbert Burger <no...@gmail.com>wrote:
> On Fri, Feb 13, 2009 at 8:37 AM, Steve Loughran <st...@apache.org> wrote:
>
> > Michael Lynch wrote:
> >
> >> Hi,
> >>
> >> As far as I can tell I've followed the setup instructions for a hadoop
> >> cluster to the letter,
> >> but I find that the datanodes can't connect to the namenode on port 9000
> >> because it is only
> >> listening for connections from localhost.
> >>
> >> In my case, the namenode is called centos1, and the datanode is called
> >> centos2. They are
> >> centos 5.1 servers with an unmodified sun java 6 runtime.
> >>
> >
> > fs.default.name takes a URL to the filesystem. such as
> > hdfs://centos1:9000/
> >
> > If the machine is only binding to localhost, that may mean DNS fun. Try a
> > fully qualified name instead
>
>
> (fs.default.name is defined in conf/hadoop-site.xml, overriding entries
> from
> conf/hadoop-default.xml).
>
> Also, check your /etc/hosts file on both machines. Could be that you have
> a
> incorrect setup where both localhost and the namenode hostname (centos1)
> are
> aliased to 127.0.0.1.
>
> Norbert
>
Re: Namenode not listening for remote connections to port 9000
Posted by Norbert Burger <no...@gmail.com>.
On Fri, Feb 13, 2009 at 8:37 AM, Steve Loughran <st...@apache.org> wrote:
> Michael Lynch wrote:
>
>> Hi,
>>
>> As far as I can tell I've followed the setup instructions for a hadoop
>> cluster to the letter,
>> but I find that the datanodes can't connect to the namenode on port 9000
>> because it is only
>> listening for connections from localhost.
>>
>> In my case, the namenode is called centos1, and the datanode is called
>> centos2. They are
>> centos 5.1 servers with an unmodified sun java 6 runtime.
>>
>
> fs.default.name takes a URL to the filesystem. such as
> hdfs://centos1:9000/
>
> If the machine is only binding to localhost, that may mean DNS fun. Try a
> fully qualified name instead
(fs.default.name is defined in conf/hadoop-site.xml, overriding entries from
conf/hadoop-default.xml).
Also, check your /etc/hosts file on both machines. Could be that you have a
incorrect setup where both localhost and the namenode hostname (centos1) are
aliased to 127.0.0.1.
Norbert
Re: Namenode not listening for remote connections to port 9000
Posted by Steve Loughran <st...@apache.org>.
Michael Lynch wrote:
> Hi,
>
> As far as I can tell I've followed the setup instructions for a hadoop
> cluster to the letter,
> but I find that the datanodes can't connect to the namenode on port 9000
> because it is only
> listening for connections from localhost.
>
> In my case, the namenode is called centos1, and the datanode is called
> centos2. They are
> centos 5.1 servers with an unmodified sun java 6 runtime.
fs.default.name takes a URL to the filesystem. such as hdfs://centos1:9000/
If the machine is only binding to localhost, that may mean DNS fun. Try
a fully qualified name instead