You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Fredrik Möllerstrand <fr...@mollerstrand.se> on 2009/06/15 19:00:07 UTC
HMaster and /etc/hosts
Hello list!
I've spent the better part of the afternoon upgrading from 0.19.3 to
trunk, and I did fall into a hole or two. Specifically, it turns out
that we rely on DNS lookups to find out what address HMaster binds to,
which caused me some grief. The documentation is also weak on what
part Zookeeper plays in the process; a stronger write-up of how HBase
utilizes Zookeeper would be a great help in troubleshooting issues
like this.
Basically, I saw the region servers trying to connect to
127.0.0.1:6000 without any hint as to why, neither in the (default)
configuration nor the logs.
HMaster happily reported this:
INFO org.apache.hadoop.hbase.master.HMaster: My address is
master2.internal.net:6000
quickly followed by:
INFO org.apache.hadoop.hbase.master.HMaster: HMaster initialized on
127.0.0.1:6000
The cause was simple and my face-palm genuine as the realization
struck me: there was a record for master2 in the hosts file, and it
was pointing to 127.0.0.1.
I suggest that an hbase.master.address option (that overrides any
resolved address) would make it easier to get the installation up and
running.
Cheers,
Fredrik
Re: HMaster and /etc/hosts
Posted by Fredrik Möllerstrand <fr...@mollerstrand.se>.
> wrt your problem, have you tried setting the following configs?
> hbase.master.dns.interface
> hbase.master.dns.nameserver
>
Indeed I did. Any such lookup was overriden by /etc/hosts as per
/etc/nsswitch.conf. Now, if I only could get a hold of the person who
put that hosts entry there in the first place.. :-)
>
> The reason we removed the master address is that the master can now
> failover to any other waiting master. You see how this could confuse
> everything if it moves 2-3 times?
Oh yes, I do realize how useful that is. In the end I suppose it's the
usual trade-off between straightforwardness – which I was very much
looking for – and useful features (and the complexity that comes with
those).
Cheers,
Fredrik
Re: HMaster and /etc/hosts
Posted by Fredrik Möllerstrand <fr...@mollerstrand.se>.
On Mon, Jun 15, 2009 at 6:18 PM, Lars George<la...@worldlingo.com> wrote:
> Hi Fredrik,
>
> Stack suggested it could be that your servers have in the nsswitch.conf to
> use files before dns? Could you try for us and switch that, revert the entry
> in the /etc/hosts and then try if the options J-D suggest to see if they
> work for this problem?
>
> Then we can document this properly in the overview.html or up on the wiki.
>
> Thanks,
> Lars
I tried to verify our assumptions but ran into fresh trouble:
After setting hbase.master.dns.interface and
hbase.master.dns.nameserver to appropriate values, I switched
/etc/hosts back as it were (with master2 pointing to 127.0.0.1) and
switched the order of 'files' and 'dns' in /etc/nsswitch.conf to read
like so:
hosts: dns files
The expected outcome of this is to force a successful DNS resolution.
Instead the master fails to start as per the exception dump below.
I suppose this is because the name server does not have an answer to
the query for "master2" - not so strange after all. But surely the
lookup mechanism should fall back on /etc/hosts? Weirdness.
2009-06-15 17:42:02,080 ERROR org.apache.hadoop.hbase.master.HMaster:
Can not start master
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1113)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1154)
Caused by: java.net.UnknownHostException: master2: master2
at java.net.InetAddress.getLocalHost(InetAddress.java:1353)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:185)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:166)
... 6 more
Cheers,
Fredrik
Re: HMaster and /etc/hosts
Posted by Lars George <la...@worldlingo.com>.
Hi Fredrik,
Stack suggested it could be that your servers have in the nsswitch.conf
to use files before dns? Could you try for us and switch that, revert
the entry in the /etc/hosts and then try if the options J-D suggest to
see if they work for this problem?
Then we can document this properly in the overview.html or up on the wiki.
Thanks,
Lars
Jean-Daniel Cryans wrote:
> Fredrik,
>
> First, thanks for trying out trunk.
>
> wrt your problem, have you tried setting the following configs?
> hbase.master.dns.interface
> hbase.master.dns.nameserver
>
> This works just like in Hadoop.
>
> The reason we removed the master address is that the master can now
> failover to any other waiting master. You see how this could confuse
> everything if it moves 2-3 times?
>
> Thx,
>
> J-D
>
> On Mon, Jun 15, 2009 at 1:00 PM, Fredrik
> Möllerstrand<fr...@mollerstrand.se> wrote:
>
>> Hello list!
>>
>> I've spent the better part of the afternoon upgrading from 0.19.3 to
>> trunk, and I did fall into a hole or two. Specifically, it turns out
>> that we rely on DNS lookups to find out what address HMaster binds to,
>> which caused me some grief. The documentation is also weak on what
>> part Zookeeper plays in the process; a stronger write-up of how HBase
>> utilizes Zookeeper would be a great help in troubleshooting issues
>> like this.
>>
>> Basically, I saw the region servers trying to connect to
>> 127.0.0.1:6000 without any hint as to why, neither in the (default)
>> configuration nor the logs.
>>
>> HMaster happily reported this:
>> INFO org.apache.hadoop.hbase.master.HMaster: My address is
>> master2.internal.net:6000
>> quickly followed by:
>> INFO org.apache.hadoop.hbase.master.HMaster: HMaster initialized on
>> 127.0.0.1:6000
>>
>> The cause was simple and my face-palm genuine as the realization
>> struck me: there was a record for master2 in the hosts file, and it
>> was pointing to 127.0.0.1.
>>
>> I suggest that an hbase.master.address option (that overrides any
>> resolved address) would make it easier to get the installation up and
>> running.
>>
>>
>> Cheers,
>> Fredrik
>>
>>
>
>
Re: HMaster and /etc/hosts
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Fredrik,
First, thanks for trying out trunk.
wrt your problem, have you tried setting the following configs?
hbase.master.dns.interface
hbase.master.dns.nameserver
This works just like in Hadoop.
The reason we removed the master address is that the master can now
failover to any other waiting master. You see how this could confuse
everything if it moves 2-3 times?
Thx,
J-D
On Mon, Jun 15, 2009 at 1:00 PM, Fredrik
Möllerstrand<fr...@mollerstrand.se> wrote:
> Hello list!
>
> I've spent the better part of the afternoon upgrading from 0.19.3 to
> trunk, and I did fall into a hole or two. Specifically, it turns out
> that we rely on DNS lookups to find out what address HMaster binds to,
> which caused me some grief. The documentation is also weak on what
> part Zookeeper plays in the process; a stronger write-up of how HBase
> utilizes Zookeeper would be a great help in troubleshooting issues
> like this.
>
> Basically, I saw the region servers trying to connect to
> 127.0.0.1:6000 without any hint as to why, neither in the (default)
> configuration nor the logs.
>
> HMaster happily reported this:
> INFO org.apache.hadoop.hbase.master.HMaster: My address is
> master2.internal.net:6000
> quickly followed by:
> INFO org.apache.hadoop.hbase.master.HMaster: HMaster initialized on
> 127.0.0.1:6000
>
> The cause was simple and my face-palm genuine as the realization
> struck me: there was a record for master2 in the hosts file, and it
> was pointing to 127.0.0.1.
>
> I suggest that an hbase.master.address option (that overrides any
> resolved address) would make it easier to get the installation up and
> running.
>
>
> Cheers,
> Fredrik
>