You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Gaojinchao <ga...@huawei.com> on 2011/02/26 02:27:14 UTC
答复: the region server can't join in the new hmaster while the state become active from standy
Hbase version ：0.90.0
Zookeeper version ：3.3.2
Hadoop version : cloudera 0.20.2+320
Is there any bug in function getMaster ?
While the active hmaster is killed, this function get old hmaster address.
So , it will continue till the old hmaster recover.
In hbase version 0.20.6, This function is called while the Master znode is created.
while (!stopped && master == null) {
try {
// Do initial RPC setup. The final argument indicates that the RPC
// should retry indefinitely.
master = (HMasterRegionInterface) HBaseRPC.waitForProxy(
HMasterRegionInterface.class, HBaseRPCProtocolVersion.versionID,
masterAddress.getInetSocketAddress(), this.conf, -1,
this.rpcTimeout, this.rpcTimeout);
} catch (IOException e) {
e = e instanceof RemoteException ?
((RemoteException)e).unwrapRemoteException() : e;
if (e instanceof ServerNotRunningException) {
LOG.info("Master isn't available yet, retrying");
} else {
LOG.warn("Unable to connect to master. Retrying. Error was:", e);
}
sleeper.sleep();
}
Hmaster Logs:
2011-02-26 09:18:12,714 DEBUG org.apache.hadoop.hbase.master.HMaster: Started service threads
2011-02-26 09:18:14,214 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:15,715 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:17,215 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:18,715 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:20,215 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:21,715 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:23,216 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:24,716 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:26,216 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:27,716 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:29,217 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:30,717 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:32,217 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:33,717 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-02-26 09:18:35,218 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
Region server logs :
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy2.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1443)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
at java.lang.Thread.run(Thread.java:662)
2011-02-26 09:17:42,182 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy2.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1443)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
at java.lang.Thread.run(Thread.java:662)
2011-02-26 09:18:45,223 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy2.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1443)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
at java.lang.Thread.run(Thread.java:662)
2011-02-26 09:19:05,983 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=13.41 MB, free=1.58 GB, max=1.6 GB, blocks=0, accesses=0, hits=0, hitRatio=锟?, cachingAccesses=0, cachingHits=0, cachingHitsRatio=锟?, evictions=0, evicted=0, evictedPerRun=NaN
2011-02-26 09:19:48,263 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy2.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1443)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
at java.lang.Thread.run(Thread.java:662)
2011-02-26 09:20:51,304 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy2.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1443)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:737)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:586)
-----邮件原件-----
发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
发送时间: 2011年2月26日 2:33
收件人: user@hbase.apache.org
主题: Re: the region server can't join in the new hmaster while the state become active from standy
I see the master is waiting and I see the exceptions but there's no
context and the timestamps don't match (what happened in the region
server at the time the second master took over), can you explain
exactly what was done to get in that state? Also could you please tell
use which hbase version you're using?
J-D
On Thu, Feb 24, 2011 at 11:04 PM, Gaojinchao <ga...@huawei.com> wrote:
> The zookeeper'sZnode had update to the new hmaster node, but the region server also connect to the old one.
> Do I need change any configure for hbase?
>
> Hmaster'log:
> 2011-02-25 14:14:42,513 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 60010
> 2011-02-25 14:14:42,513 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 60010 webServer.getConnectors()[0].getLocalPort() returned 60010
> 2011-02-25 14:14:42,513 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 60010
> 2011-02-25 14:14:42,514 INFO org.mortbay.log: jetty-6.1.26
> 2011-02-25 14:14:42,743 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:60010
> 2011-02-25 14:14:42,744 DEBUG org.apache.hadoop.hbase.master.HMaster: Started service threads
> 2011-02-25 14:14:44,244 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
> 2011-02-25 14:14:45,744 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
> 2011-02-25 14:14:47,245 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
> 2011-02-25 14:14:48,745 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
>
>
> Region server's log:
>
> 2011-02-25 13:22:38,136 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://C4C1:9000/hbase/ufdr5/109de857846354947e0664b2edd5f544/value/6813060688172507178, isReference=false, isBulkLoadResult=false, seqid=60129691, majorCompaction=true
> 2011-02-25 13:22:38,138 WARN org.apache.hadoop.hbase.regionserver.Store: Failed open of hdfs://C4C1:9000/hbase/ufdr5/1a0f0d2085977a1dbdf9e9899013deef/value/5446694695136514012.6231cf1c5312c87f5ff55b64b3109491; presumption is that file was corrupted at flush and lost edits picked up by commit log replay. Verify!
> java.io.IOException: Cannot open filename /hbase/ufdr5/6231cf1c5312c87f5ff55b64b3109491/value/5446694695136514012
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1493)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1484)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:380)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:178)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:398)
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader.<init>(HFile.java:748)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:899)
> at org.apache.hadoop.hbase.io.HalfStoreFileReader.<init>(HalfStoreFileReader.java:65)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:375)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:266)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:208)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1963)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:343)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2506)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2492)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:262)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:94)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2011-02-25 13:22:38,138 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined ufdr5,8613815537639#2927,1298400219634.109de857846354947e0664b2edd5f544.; next sequenceid=60129692
> 2011-02-25 13:22:38,139 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12e5a47a4e4000a Attempting to transition node 109de857846354947e0664b2edd5f544 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2011-02-25 13:22:38,139 WARN org.apache.hadoop.hbase.regionserver.Store: Failed open of hdfs://C4C1:9000/hbase/ufdr5/86ac13dbf1e0287a9d6f5dc6caf452c8/value/3876832662736680001.9fe3127004d2ce5351da083fa2b824b5; presumption is that file was corrupted at flush and lost edits picked up by commit log replay. Verify!
> java.io.IOException: Cannot open filename /hbase/ufdr5/9fe3127004d2ce5351da083fa2b824b5/value/3876832662736680001
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1493)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1484)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:380)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:178)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:398)
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader.<init>(HFile.java:748)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:899)
> at org.apache.hadoop.hbase.io.HalfStoreFileReader.<init>(HalfStoreFileReader.java:65)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:375)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:266)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:208)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1963)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:343)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2506)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2492)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:262)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:94)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2011-02-25 13:22:38,150 WARN org.apache.hadoop.hbase.regionserver.Store: Failed open of hdfs://C4C1:9000/hbase/ufdr5/86ac13dbf1e0287a9d6f5dc6caf452c8/value/7758939538437314246.9fe3127004d2ce5351da083fa2b824b5; presumption is that file was corrupted at flush and lost edits picked up by commit log replay. Verify!
> java.io.IOException: Cannot open filename /hbase/ufdr5/9fe3127004d2ce5351da083fa2b824b5/value/7758939538437314246
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1493)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1484)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:380)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:178)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:398)
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader.<init>(HFile.java:748)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:899)
> at org.apache.hadoop.hbase.io.HalfStoreFileReader.<init>(HalfStoreFileReader.java:65)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:375)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:438)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:266)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:208)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1963)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:343)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2506)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2492)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:262)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:94)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
Re: 答复: the region server can't join in the new hmaster while the state become active from standy
Posted by Stack <st...@duboce.net>.
The below looks like https://issues.apache.org/jira/browse/HBASE-3545.
The fix will be in 0.90.2. Try upgrading to 0.90.1 and patching in
HBASE-3545. See if that fixes your issue.
Yours,
St.Ack