You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Daniel Leffel <da...@gmail.com> on 2008/05/01 18:32:13 UTC
HBase Master only listening to Local Connections on 60000?
Not sure what's going on with this and thought I'd ask. I can't find any
rationale for this on the wiki or in the logs.
I've begun to set up our production environment which is a multi-node setup.
(To-date our development environment was a single node in psudeo-distributed
mode).
The HBase master is listening on port 60000 only for connections that are
local and is refusing remote connections. All the other services on the box
(60010,600020 and 600030) listen both locally and remotely. There's no
firewall on the box.
In the hbase-site.xml, I've specificed hbase.master to be [hostname]:60000
Any ideas?
Re: HBase Master only listening to Local Connections on 60000?
Posted by Daniel Leffel <da...@gmail.com>.
Hi All,
Wanted to put the answer out there in case someone else was having the same
problem.
In looking at the /etc/hosts file, it's important that the hostname NOT map
to 127.0.0.1 there. Instead, map it to the real ip address that you want
regions servers to connect to the master on.
Cheers!
Danny
On Thu, May 1, 2008 at 12:51 PM, Daniel Leffel <da...@gmail.com>
wrote:
> Also, in looking at the master, it clearly thinks it's ip address is
> 127.0.0.1, although it is accepting remote connections on port 60000 now.
>
>
> On Thu, May 1, 2008 at 12:47 PM, Daniel Leffel <da...@gmail.com>
> wrote:
>
>> Switching to all IPs (both in hbase-site.xml and regionservers file)
>> causes the local region server and master to start up fine. The remote ones
>> start throwing the following exception (debug enabled):
>>
>> 2008-05-01 14:45:32,284 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>> Initializing RPC Metrics with hostName=60020, port=60020
>> 2008-05-01 14:45:32,326 DEBUG org.apache.hadoop.hbase.HRegionServer:
>> Telling master at 10.10.1.114:60000 that we are up
>> 2008-05-01 14:45:32,376 WARN org.apache.hadoop.hbase.HRegionServer: error
>> telling master we are up
>> org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>> at org.apache.hadoop.hbase.Leases.createLease(Leases.java:147)
>> at
>> org.apache.hadoop.hbase.HMaster.regionServerStartup(HMaster.java:1318)
>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:512)
>> at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:210)
>> at $Proxy0.regionServerStartup(Unknown Source)
>> at
>> org.apache.hadoop.hbase.HRegionServer.reportForDuty(HRegionServer.java:1112)
>> at
>> org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:712)
>> at java.lang.Thread.run(Thread.java:595)
>>
>>
>>
>>
>> On Thu, May 1, 2008 at 10:13 AM, stack <st...@duboce.net> wrote:
>>
>>> Daniel:
>>>
>>> The line out of RpcMetrics reporting port for hostname looks like a
>>> little bug we should fix up in hadoop core (I took a quick look; looks like
>>> the right args are being passed -- problem must be further up the stack or
>>> perhaps your hadoop is older?).
>>>
>>> Regards the next line reporting localhost, thats odd. Is your networking
>>> setup properly? What does the host report for IP when you query using the
>>> hostname you've inserted as the value for hbase.master? What if you use IP
>>> instead?
>>>
>>> Here is what my logs looks like:
>>>
>>> 2008-05-01 04:15:34,234 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>>> Initializing RPC Metrics with hostName=60000, port=60000
>>> 2008-05-01 04:15:34,361 INFO org.apache.hadoop.hbase.HMaster: HMaster
>>> initialized on XXX.XXX.44.139:60000
>>>
>>> Here is my hbase.master:
>>>
>>> aa0-XXX-XXX.at.our.domain:60000
>>>
>>> St.Ack
>>>
>>> P.S. And on offlining of tables, we're still working on it..... should
>>> have a candidate up in the next day or so w/ a fix included.
>>>
>>>
>>>
>>> Daniel Leffel wrote:
>>>
>>>> Actually, I did find something curious in the master log with debug
>>>> turned
>>>> on:
>>>>
>>>> 2008-05-01 11:05:56,407 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>>>> Initializing RPC Metrics with hostName=60000, port=60000
>>>> 2008-05-01 11:05:56,482 INFO org.apache.hadoop.hbase.HMaster: HMaster
>>>> initialized on 127.0.0.1:60000
>>>>
>>>> hostname 60000? but then the next line shows localhost?
>>>>
>>>> On Thu, May 1, 2008 at 9:32 AM, Daniel Leffel <da...@gmail.com>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>> Not sure what's going on with this and thought I'd ask. I can't find
>>>>> any
>>>>> rationale for this on the wiki or in the logs.
>>>>>
>>>>> I've begun to set up our production environment which is a multi-node
>>>>> setup. (To-date our development environment was a single node in
>>>>> psudeo-distributed mode).
>>>>>
>>>>> The HBase master is listening on port 60000 only for connections that
>>>>> are
>>>>> local and is refusing remote connections. All the other services on the
>>>>> box
>>>>> (60010,600020 and 600030) listen both locally and remotely. There's no
>>>>> firewall on the box.
>>>>>
>>>>> In the hbase-site.xml, I've specificed hbase.master to be
>>>>> [hostname]:60000
>>>>>
>>>>> Any ideas?
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: HBase Master only listening to Local Connections on 60000?
Posted by Daniel Leffel <da...@gmail.com>.
Also, in looking at the master, it clearly thinks it's ip address is
127.0.0.1, although it is accepting remote connections on port 60000 now.
On Thu, May 1, 2008 at 12:47 PM, Daniel Leffel <da...@gmail.com>
wrote:
> Switching to all IPs (both in hbase-site.xml and regionservers file) causes
> the local region server and master to start up fine. The remote ones start
> throwing the following exception (debug enabled):
>
> 2008-05-01 14:45:32,284 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=60020, port=60020
> 2008-05-01 14:45:32,326 DEBUG org.apache.hadoop.hbase.HRegionServer:
> Telling master at 10.10.1.114:60000 that we are up
> 2008-05-01 14:45:32,376 WARN org.apache.hadoop.hbase.HRegionServer: error
> telling master we are up
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
> at org.apache.hadoop.hbase.Leases.createLease(Leases.java:147)
> at
> org.apache.hadoop.hbase.HMaster.regionServerStartup(HMaster.java:1318)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:512)
> at
> org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:210)
> at $Proxy0.regionServerStartup(Unknown Source)
> at
> org.apache.hadoop.hbase.HRegionServer.reportForDuty(HRegionServer.java:1112)
> at
> org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:712)
> at java.lang.Thread.run(Thread.java:595)
>
>
>
>
> On Thu, May 1, 2008 at 10:13 AM, stack <st...@duboce.net> wrote:
>
>> Daniel:
>>
>> The line out of RpcMetrics reporting port for hostname looks like a little
>> bug we should fix up in hadoop core (I took a quick look; looks like the
>> right args are being passed -- problem must be further up the stack or
>> perhaps your hadoop is older?).
>>
>> Regards the next line reporting localhost, thats odd. Is your networking
>> setup properly? What does the host report for IP when you query using the
>> hostname you've inserted as the value for hbase.master? What if you use IP
>> instead?
>>
>> Here is what my logs looks like:
>>
>> 2008-05-01 04:15:34,234 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>> Initializing RPC Metrics with hostName=60000, port=60000
>> 2008-05-01 04:15:34,361 INFO org.apache.hadoop.hbase.HMaster: HMaster
>> initialized on XXX.XXX.44.139:60000
>>
>> Here is my hbase.master:
>>
>> aa0-XXX-XXX.at.our.domain:60000
>>
>> St.Ack
>>
>> P.S. And on offlining of tables, we're still working on it..... should
>> have a candidate up in the next day or so w/ a fix included.
>>
>>
>>
>> Daniel Leffel wrote:
>>
>>> Actually, I did find something curious in the master log with debug
>>> turned
>>> on:
>>>
>>> 2008-05-01 11:05:56,407 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>>> Initializing RPC Metrics with hostName=60000, port=60000
>>> 2008-05-01 11:05:56,482 INFO org.apache.hadoop.hbase.HMaster: HMaster
>>> initialized on 127.0.0.1:60000
>>>
>>> hostname 60000? but then the next line shows localhost?
>>>
>>> On Thu, May 1, 2008 at 9:32 AM, Daniel Leffel <da...@gmail.com>
>>> wrote:
>>>
>>>
>>>
>>>> Not sure what's going on with this and thought I'd ask. I can't find any
>>>> rationale for this on the wiki or in the logs.
>>>>
>>>> I've begun to set up our production environment which is a multi-node
>>>> setup. (To-date our development environment was a single node in
>>>> psudeo-distributed mode).
>>>>
>>>> The HBase master is listening on port 60000 only for connections that
>>>> are
>>>> local and is refusing remote connections. All the other services on the
>>>> box
>>>> (60010,600020 and 600030) listen both locally and remotely. There's no
>>>> firewall on the box.
>>>>
>>>> In the hbase-site.xml, I've specificed hbase.master to be
>>>> [hostname]:60000
>>>>
>>>> Any ideas?
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
Re: HBase Master only listening to Local Connections on 60000?
Posted by Daniel Leffel <da...@gmail.com>.
Switching to all IPs (both in hbase-site.xml and regionservers file) causes
the local region server and master to start up fine. The remote ones start
throwing the following exception (debug enabled):
2008-05-01 14:45:32,284 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=60020, port=60020
2008-05-01 14:45:32,326 DEBUG org.apache.hadoop.hbase.HRegionServer: Telling
master at 10.10.1.114:60000 that we are up
2008-05-01 14:45:32,376 WARN org.apache.hadoop.hbase.HRegionServer: error
telling master we are up
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.Leases$LeaseStillHeldException
at org.apache.hadoop.hbase.Leases.createLease(Leases.java:147)
at
org.apache.hadoop.hbase.HMaster.regionServerStartup(HMaster.java:1318)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
at org.apache.hadoop.ipc.Client.call(Client.java:512)
at
org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:210)
at $Proxy0.regionServerStartup(Unknown Source)
at
org.apache.hadoop.hbase.HRegionServer.reportForDuty(HRegionServer.java:1112)
at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:712)
at java.lang.Thread.run(Thread.java:595)
On Thu, May 1, 2008 at 10:13 AM, stack <st...@duboce.net> wrote:
> Daniel:
>
> The line out of RpcMetrics reporting port for hostname looks like a little
> bug we should fix up in hadoop core (I took a quick look; looks like the
> right args are being passed -- problem must be further up the stack or
> perhaps your hadoop is older?).
>
> Regards the next line reporting localhost, thats odd. Is your networking
> setup properly? What does the host report for IP when you query using the
> hostname you've inserted as the value for hbase.master? What if you use IP
> instead?
>
> Here is what my logs looks like:
>
> 2008-05-01 04:15:34,234 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=60000, port=60000
> 2008-05-01 04:15:34,361 INFO org.apache.hadoop.hbase.HMaster: HMaster
> initialized on XXX.XXX.44.139:60000
>
> Here is my hbase.master:
>
> aa0-XXX-XXX.at.our.domain:60000
>
> St.Ack
>
> P.S. And on offlining of tables, we're still working on it..... should have
> a candidate up in the next day or so w/ a fix included.
>
>
>
> Daniel Leffel wrote:
>
>> Actually, I did find something curious in the master log with debug turned
>> on:
>>
>> 2008-05-01 11:05:56,407 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>> Initializing RPC Metrics with hostName=60000, port=60000
>> 2008-05-01 11:05:56,482 INFO org.apache.hadoop.hbase.HMaster: HMaster
>> initialized on 127.0.0.1:60000
>>
>> hostname 60000? but then the next line shows localhost?
>>
>> On Thu, May 1, 2008 at 9:32 AM, Daniel Leffel <da...@gmail.com>
>> wrote:
>>
>>
>>
>>> Not sure what's going on with this and thought I'd ask. I can't find any
>>> rationale for this on the wiki or in the logs.
>>>
>>> I've begun to set up our production environment which is a multi-node
>>> setup. (To-date our development environment was a single node in
>>> psudeo-distributed mode).
>>>
>>> The HBase master is listening on port 60000 only for connections that are
>>> local and is refusing remote connections. All the other services on the
>>> box
>>> (60010,600020 and 600030) listen both locally and remotely. There's no
>>> firewall on the box.
>>>
>>> In the hbase-site.xml, I've specificed hbase.master to be
>>> [hostname]:60000
>>>
>>> Any ideas?
>>>
>>>
>>>
>>
>>
>>
>
>
Re: HBase Master only listening to Local Connections on 60000?
Posted by stack <st...@duboce.net>.
Daniel:
The line out of RpcMetrics reporting port for hostname looks like a
little bug we should fix up in hadoop core (I took a quick look; looks
like the right args are being passed -- problem must be further up the
stack or perhaps your hadoop is older?).
Regards the next line reporting localhost, thats odd. Is your
networking setup properly? What does the host report for IP when you
query using the hostname you've inserted as the value for hbase.master?
What if you use IP instead?
Here is what my logs looks like:
2008-05-01 04:15:34,234 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=60000, port=60000
2008-05-01 04:15:34,361 INFO org.apache.hadoop.hbase.HMaster: HMaster
initialized on XXX.XXX.44.139:60000
Here is my hbase.master:
aa0-XXX-XXX.at.our.domain:60000
St.Ack
P.S. And on offlining of tables, we're still working on it..... should
have a candidate up in the next day or so w/ a fix included.
Daniel Leffel wrote:
> Actually, I did find something curious in the master log with debug turned
> on:
>
> 2008-05-01 11:05:56,407 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=60000, port=60000
> 2008-05-01 11:05:56,482 INFO org.apache.hadoop.hbase.HMaster: HMaster
> initialized on 127.0.0.1:60000
>
> hostname 60000? but then the next line shows localhost?
>
> On Thu, May 1, 2008 at 9:32 AM, Daniel Leffel <da...@gmail.com>
> wrote:
>
>
>> Not sure what's going on with this and thought I'd ask. I can't find any
>> rationale for this on the wiki or in the logs.
>>
>> I've begun to set up our production environment which is a multi-node
>> setup. (To-date our development environment was a single node in
>> psudeo-distributed mode).
>>
>> The HBase master is listening on port 60000 only for connections that are
>> local and is refusing remote connections. All the other services on the box
>> (60010,600020 and 600030) listen both locally and remotely. There's no
>> firewall on the box.
>>
>> In the hbase-site.xml, I've specificed hbase.master to be [hostname]:60000
>>
>> Any ideas?
>>
>>
>
>
Re: HBase Master only listening to Local Connections on 60000?
Posted by Daniel Leffel <da...@gmail.com>.
Actually, I did find something curious in the master log with debug turned
on:
2008-05-01 11:05:56,407 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=60000, port=60000
2008-05-01 11:05:56,482 INFO org.apache.hadoop.hbase.HMaster: HMaster
initialized on 127.0.0.1:60000
hostname 60000? but then the next line shows localhost?
On Thu, May 1, 2008 at 9:32 AM, Daniel Leffel <da...@gmail.com>
wrote:
> Not sure what's going on with this and thought I'd ask. I can't find any
> rationale for this on the wiki or in the logs.
>
> I've begun to set up our production environment which is a multi-node
> setup. (To-date our development environment was a single node in
> psudeo-distributed mode).
>
> The HBase master is listening on port 60000 only for connections that are
> local and is refusing remote connections. All the other services on the box
> (60010,600020 and 600030) listen both locally and remotely. There's no
> firewall on the box.
>
> In the hbase-site.xml, I've specificed hbase.master to be [hostname]:60000
>
> Any ideas?
>