You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Daniel Iancu <da...@1and1.ro> on 2011/03/04 15:39:21 UTC
Hbase 0.90.1 master crashes regularly
Hi
I've updated our dev environment from Hbase 0.90.0 (ASF+CDH3b3) which
behaved very stable to Hbase 0.90.1 (CDH3B4) and since then the HMaster
dies regularly. Issue seems to be regarded to the connection to
Zookeeper. Even if I use a standby HMaster, this one also dies from same
cause:
2011-03-04 15:05:54,699 FATAL
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Unexpected exception handling nodeDeleted event
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:165)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:261)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
2011-03-04 15:05:54,704 INFO org.apache.zookeeper.ZooKeeper: Session:
0x22e80dcc2350001 closed
2011-03-04 15:05:54,704 INFO org.apache.zookeeper.ClientCnxn:
EventThread shut down
2011-03-04 15:05:54,718 INFO org.apache.zookeeper.ZooKeeper: Session:
0x22e80dcc2350000 closed
2011-03-04 15:05:54,718 INFO org.apache.zookeeper.ClientCnxn:
EventThread shut down
2011-03-04 15:05:54,718 INFO org.apache.hadoop.hbase.master.HMaster:
HMaster main thread exiting
just before this one there is an other exception
2011-03-04 15:07:00,611 FATAL org.apache.hadoop.hbase.master.HMaster:
Failed assignment of regions to serverName=desktop,60020,1299242075991,
load=(requests=0, regions=0, usedHeap=34, maxHeap=996)
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting
up proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to
/172.28.124.148:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:355)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:954)
at
org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:606)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:560)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:776)
at
org.apache.hadoop.hbase.master.AssignmentManager$SingleServerBulkAssigner.run(AssignmentManager.java:1310)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
... 8 more
2011-03-04 15:07:00,615 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
Any hint for me what could be wrong there?
Thanks
Daniel
Re: Hbase 0.90.1 master crashes regularly
Posted by Daniel Iancu <da...@1and1.ro>.
Thanks Stack, that was indeed the cause, the machines were not properly
registered in DNS.
I've corrected that and now is fine.
Regards
Daniel
On 03/04/2011 08:15 PM, Stack wrote:
> The first exception is about lost connection to the zk ensemble. The
> second is a failure establishing connection to 'desktop,60020'. Later
> we refer to an IP for desktop. Is your dns set so resolve and reverse
> lookup give same answer? HBase 0.90.x is finicky in this regard (To
> be fixed).
> St.Ack
>
> On Fri, Mar 4, 2011 at 6:39 AM, Daniel Iancu<da...@1and1.ro> wrote:
>> Hi
>> I've updated our dev environment from Hbase 0.90.0 (ASF+CDH3b3) which
>> behaved very stable to Hbase 0.90.1 (CDH3B4) and since then the HMaster dies
>> regularly. Issue seems to be regarded to the connection to Zookeeper. Even
>> if I use a standby HMaster, this one also dies from same cause:
>>
>>
>> 2011-03-04 15:05:54,699 FATAL
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> Unexpected exception handling nodeDeleted event
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase/master
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:165)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:261)
>> at
>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
>> 2011-03-04 15:05:54,704 INFO org.apache.zookeeper.ZooKeeper: Session:
>> 0x22e80dcc2350001 closed
>> 2011-03-04 15:05:54,704 INFO org.apache.zookeeper.ClientCnxn: EventThread
>> shut down
>> 2011-03-04 15:05:54,718 INFO org.apache.zookeeper.ZooKeeper: Session:
>> 0x22e80dcc2350000 closed
>> 2011-03-04 15:05:54,718 INFO org.apache.zookeeper.ClientCnxn: EventThread
>> shut down
>> 2011-03-04 15:05:54,718 INFO org.apache.hadoop.hbase.master.HMaster: HMaster
>> main thread exiting
>>
>>
>> just before this one there is an other exception
>>
>> 2011-03-04 15:07:00,611 FATAL org.apache.hadoop.hbase.master.HMaster: Failed
>> assignment of regions to serverName=desktop,60020,1299242075991,
>> load=(requests=0, regions=0, usedHeap=34, maxHeap=996)
>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up
>> proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to
>> /172.28.124.148:60020 after attempts=1
>> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:355)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:954)
>> at
>> org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:606)
>> at
>> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:560)
>> at
>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:776)
>> at
>> org.apache.hadoop.hbase.master.AssignmentManager$SingleServerBulkAssigner.run(AssignmentManager.java:1310)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>> at $Proxy6.getProtocolVersion(Unknown Source)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>> ... 8 more
>> 2011-03-04 15:07:00,615 INFO org.apache.hadoop.hbase.master.HMaster:
>> Aborting
>>
>> Any hint for me what could be wrong there?
>>
>> Thanks
>> Daniel
>>
--
Daniel Iancu
Java Developer,Web Components Romania
1&1 Internet Development srl.
18 Mircea Eliade St
Sect 1, Bucharest
RO Bucharest, 012015
www.1and1.ro
Phone:+40-031-223-9081
Email:daniel.iancu@1and1.ro
IM:diancu@united.domain
Re: Hbase 0.90.1 master crashes regularly
Posted by Stack <st...@duboce.net>.
The first exception is about lost connection to the zk ensemble. The
second is a failure establishing connection to 'desktop,60020'. Later
we refer to an IP for desktop. Is your dns set so resolve and reverse
lookup give same answer? HBase 0.90.x is finicky in this regard (To
be fixed).
St.Ack
On Fri, Mar 4, 2011 at 6:39 AM, Daniel Iancu <da...@1and1.ro> wrote:
> Hi
> I've updated our dev environment from Hbase 0.90.0 (ASF+CDH3b3) which
> behaved very stable to Hbase 0.90.1 (CDH3B4) and since then the HMaster dies
> regularly. Issue seems to be regarded to the connection to Zookeeper. Even
> if I use a standby HMaster, this one also dies from same cause:
>
>
> 2011-03-04 15:05:54,699 FATAL
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Unexpected exception handling nodeDeleted event
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:165)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:261)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
> 2011-03-04 15:05:54,704 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x22e80dcc2350001 closed
> 2011-03-04 15:05:54,704 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2011-03-04 15:05:54,718 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x22e80dcc2350000 closed
> 2011-03-04 15:05:54,718 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2011-03-04 15:05:54,718 INFO org.apache.hadoop.hbase.master.HMaster: HMaster
> main thread exiting
>
>
> just before this one there is an other exception
>
> 2011-03-04 15:07:00,611 FATAL org.apache.hadoop.hbase.master.HMaster: Failed
> assignment of regions to serverName=desktop,60020,1299242075991,
> load=(requests=0, regions=0, usedHeap=34, maxHeap=996)
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up
> proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to
> /172.28.124.148:60020 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:355)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:954)
> at
> org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:606)
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:560)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:776)
> at
> org.apache.hadoop.hbase.master.AssignmentManager$SingleServerBulkAssigner.run(AssignmentManager.java:1310)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy6.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> ... 8 more
> 2011-03-04 15:07:00,615 INFO org.apache.hadoop.hbase.master.HMaster:
> Aborting
>
> Any hint for me what could be wrong there?
>
> Thanks
> Daniel
>