You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by James Kennedy <ja...@troove.net> on 2010/08/14 00:59:35 UTC
Can master detect sudden region server death?
For our system it is critical that there be no data loss and fast recovery time if any node goes down.
We've recently updated the hbase-transactional-tableindexed extension to work with the latest 0.89.20100726 version of HBase (still to be pushed).
All HBase tests are passing but then when we started to write our own and test true sudden HRegionServer death we ran into trouble.
It seems that the HMaster does not recognize the kill even after many minutes. Client requests are blocked and the log continues to repeat the logs below.
We realized that HBase's own tests that require RegionServer death use abort() and not kill() which does enough cleanup to inadequately simulate a sudden (e.g. JVM crash) death.
As an experiment I made HRegionServer.kill() public and modified HBaseMiniCluster to call that from abort() instead. Now a test like TestMasterTransitions will exhibit similar behaviour: The HMaster never notices the RegionServer is gone.
Could it really be that sudden region server death is not handled in hbase?
Or more likely is this a failure of the testing framework to adequately simulate kill -9?
James Kennedy
Project Manager
Troove Inc.
-------------------------------
[13/08/10 15:12:12] 259494 [n.serverMonitor] INFO oop.hbase.master.ServerManager - 2 region servers, 0 dead, average load 3.5
[13/08/10 15:12:12] 259560 [ger.metaScanner] INFO adoop.hbase.master.BaseScanner - RegionManager.metaScanner scanning meta region {server: 10.0.1.4:56908, regionname: .META.,,1.1028785192, startKey: <>}
[13/08/10 15:12:12] 259561 [ger.metaScanner] WARN adoop.hbase.master.BaseScanner - Scan one META region: {server: 10.0.1.4:56908, regionname: .META.,,1.1028785192, startKey: <>}
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:309)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:857)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:725)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:253)
at $Proxy10.openScanner(Unknown Source)
at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:182)
at org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:156)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
Re: Can master detect sudden region server death?
Posted by James Kennedy <ja...@troove.net>.
On 2010-08-13, at 5:04 PM, Stack wrote:
> On Fri, Aug 13, 2010 at 3:59 PM, James Kennedy <ja...@troove.net> wrote:
>> We've recently updated the hbase-transactional-tableindexed extension to work with the latest 0.89.20100726 version of HBase (still to be pushed).
>
> Good. Please update the list when you push out the change.
Will do. We're just working on getting an HBase patch up to snuff for review. We needed to make some things like the the HLog split code more extendible.
>
> Are you sure the master doesn't notice the server gone? Maybe the RS
> is dead in all but the ZK Client session thread so its lease is not
> timing out against the ZK ensemble?
Will verify.
>
> The other issue that can get in the way of a RS recovery is replay of
> logs; the master needs to grab the lease on the WAL log that the dead
> regionserver was writing. This can take a little while. It won't
> work at all if you are not using an hdfs that doesn't support
> 'append'; i.e. the hadoop that is in hbase lib dir or build your own
> out of apache hadoop branch-0.20-append branch (or get cdh3b2). Is
> this the issue (Tail the log and look for failure to obtain file
> lease).
Yeah we definitely have appends enabled. We did at one point have a separate issue where that lease recovery loop would never exit. Problem was that our transactional extension was failing to close it's own THLog HDFS file when region server was aborted. We fixed that. But it seemed to be that that comment in the loop about possibly continuing after lease expiry time should be followed up on (maybe it has been in trunk since last i looked).
>
> An incomplete server crash recovery will show the exceptions you pasted below.
>
>> Could it really be that sudden region server death is not handled in hbase?
>> Or more likely is this a failure of the testing framework to adequately simulate kill -9?
>>
>
> Its kinda hard to simulate kill -9 in unit tests. The 'kill' method
> above was a pale attempt at it in our unit testing context. I've
> manually kill -9'd nodes -- RS and DN -- to simulate node death
> testing 0.89.x builds and all seems to be basically working.
Ok, we'll do some more manual testing and explore gremlins as JD Cryans suggests.
I don't think kill -9 testing is critical for our hbase patch but it will be for guaranteeing that our transactional extension works.
> On making it so master notices downed servers the faster so it'll
> start up recovery the sooner, thats a case of playing with the zk
> session timeout but coming in the other direction is tuning GC so GC
> pause is not > ZK session expiration.
In the case i mentioned i waited 5 min which was well over the set 60 sec zk lease time.
>
> You also want to ensure that there is not too much of a backlog of WAL
> logs else recovery which involves splitting all WALs that had
> outstanding edits on the crashed server has less work to do when it
> runs. Default is maximum backlog of 32 WALs. You might want to tune
> this down though there is a currently an issue because it seems like
> we can overrun this limit.
What happens if the maximum is exceeded? Logs are lost? Our system is data critical and we can't afford to lose anything.
Thanks again,
James
Re: Can master detect sudden region server death?
Posted by Stack <st...@duboce.net>.
On Fri, Aug 13, 2010 at 3:59 PM, James Kennedy <ja...@troove.net> wrote:
> We've recently updated the hbase-transactional-tableindexed extension to work with the latest 0.89.20100726 version of HBase (still to be pushed).
Good. Please update the list when you push out the change.
> All HBase tests are passing but then when we started to write our own and test true sudden HRegionServer death we ran into trouble.
> It seems that the HMaster does not recognize the kill even after many minutes. Client requests are blocked and the log continues to repeat the logs below.
>
> We realized that HBase's own tests that require RegionServer death use abort() and not kill() which does enough cleanup to inadequately simulate a sudden (e.g. JVM crash) death.
>
> As an experiment I made HRegionServer.kill() public and modified HBaseMiniCluster to call that from abort() instead. Now a test like TestMasterTransitions will exhibit similar behaviour: The HMaster never notices the RegionServer is gone.
>
Odd. I added kill. Its used in TestMasterTransistions currently.
Are you sure the master doesn't notice the server gone? Maybe the RS
is dead in all but the ZK Client session thread so its lease is not
timing out against the ZK ensemble?
The other issue that can get in the way of a RS recovery is replay of
logs; the master needs to grab the lease on the WAL log that the dead
regionserver was writing. This can take a little while. It won't
work at all if you are not using an hdfs that doesn't support
'append'; i.e. the hadoop that is in hbase lib dir or build your own
out of apache hadoop branch-0.20-append branch (or get cdh3b2). Is
this the issue (Tail the log and look for failure to obtain file
lease).
An incomplete server crash recovery will show the exceptions you pasted below.
> Could it really be that sudden region server death is not handled in hbase?
> Or more likely is this a failure of the testing framework to adequately simulate kill -9?
>
Its kinda hard to simulate kill -9 in unit tests. The 'kill' method
above was a pale attempt at it in our unit testing context. I've
manually kill -9'd nodes -- RS and DN -- to simulate node death
testing 0.89.x builds and all seems to be basically working.
On making it so master notices downed servers the faster so it'll
start up recovery the sooner, thats a case of playing with the zk
session timeout but coming in the other direction is tuning GC so GC
pause is not > ZK session expiration.
You also want to ensure that there is not too much of a backlog of WAL
logs else recovery which involves splitting all WALs that had
outstanding edits on the crashed server has less work to do when it
runs. Default is maximum backlog of 32 WALs. You might want to tune
this down though there is a currently an issue because it seems like
we can overrun this limit.
Hope this helps James,
St.Ack
> James Kennedy
> Project Manager
> Troove Inc.
>
>
> -------------------------------
>
> [13/08/10 15:12:12] 259494 [n.serverMonitor] INFO oop.hbase.master.ServerManager - 2 region servers, 0 dead, average load 3.5
> [13/08/10 15:12:12] 259560 [ger.metaScanner] INFO adoop.hbase.master.BaseScanner - RegionManager.metaScanner scanning meta region {server: 10.0.1.4:56908, regionname: .META.,,1.1028785192, startKey: <>}
> [13/08/10 15:12:12] 259561 [ger.metaScanner] WARN adoop.hbase.master.BaseScanner - Scan one META region: {server: 10.0.1.4:56908, regionname: .META.,,1.1028785192, startKey: <>}
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:309)
> at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:857)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:725)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:253)
> at $Proxy10.openScanner(Unknown Source)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:182)
> at org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
> at org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:156)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
>
>
>
>
Re: Can master detect sudden region server death?
Posted by Todd Lipcon <to...@cloudera.com>.
Hey James,
I wrote Gremlins this spring when we were starting to test the new
durability features. I have a five node test cluster here, and I ran
Gremlins on one or two of the nodes, set up to fail a couple times an hour,
while also running some tests which insert and verify data. This found a
bunch of the bugs we fixed in HDFS as well as HBase, though there are still
some more lurking.
If you check out the code, you'll see it's really pretty simple - nothing
more than a sleep loop around some handy functions for killing things.
-Todd
On Tue, Aug 17, 2010 at 11:41 AM, James Kennedy <ja...@troove.net>wrote:
> Hmm, yeah i waited well over the zk lease time for the HMaster to come out
> of the exception loop but it never did.
>
> Thanks, i'll follow up with a look at Gremlins. How is it used within
> HBase? Do you continuously run gremlins on a cluster in an automated way?
>
> James Kennedy
> Project Manager
> Troove Inc.
>
> 1 877 330 8501
>
> On 2010-08-13, at 4:06 PM, Jean-Daniel Cryans wrote:
>
> > The master will get a Watcher event from ZooKeeper when the region
> > server's session is expired and its ephemeral znode is deleted. By
> > default the session timeout is really high to cope with users with
> > huge GC pauses problems, something like 1 minute (see
> > hbase-default.xml).
> >
> > For a good fault testing framework, please use
> > http://github.com/toddlipcon/gremlins. This was written by Todd Lipcon
> > to test HBase's handling of region servers' death.
> >
> > J-D
> >
> > On Fri, Aug 13, 2010 at 3:59 PM, James Kennedy <ja...@troove.net>
> wrote:
> >> For our system it is critical that there be no data loss and fast
> recovery time if any node goes down.
> >>
> >> We've recently updated the hbase-transactional-tableindexed extension to
> work with the latest 0.89.20100726 version of HBase (still to be pushed).
> >> All HBase tests are passing but then when we started to write our own
> and test true sudden HRegionServer death we ran into trouble.
> >> It seems that the HMaster does not recognize the kill even after many
> minutes. Client requests are blocked and the log continues to repeat the
> logs below.
> >>
> >> We realized that HBase's own tests that require RegionServer death use
> abort() and not kill() which does enough cleanup to inadequately simulate a
> sudden (e.g. JVM crash) death.
> >>
> >> As an experiment I made HRegionServer.kill() public and modified
> HBaseMiniCluster to call that from abort() instead. Now a test like
> TestMasterTransitions will exhibit similar behaviour: The HMaster never
> notices the RegionServer is gone.
> >>
> >> Could it really be that sudden region server death is not handled in
> hbase?
> >> Or more likely is this a failure of the testing framework to adequately
> simulate kill -9?
> >>
> >> James Kennedy
> >> Project Manager
> >> Troove Inc.
> >>
> >>
> >> -------------------------------
> >>
> >> [13/08/10 15:12:12] 259494 [n.serverMonitor] INFO
> oop.hbase.master.ServerManager - 2 region servers, 0 dead, average load
> 3.5
> >> [13/08/10 15:12:12] 259560 [ger.metaScanner] INFO
> adoop.hbase.master.BaseScanner - RegionManager.metaScanner scanning meta
> region {server: 10.0.1.4:56908, regionname: .META.,,1.1028785192,
> startKey: <>}
> >> [13/08/10 15:12:12] 259561 [ger.metaScanner] WARN
> adoop.hbase.master.BaseScanner - Scan one META region: {server:
> 10.0.1.4:56908, regionname: .META.,,1.1028785192, startKey: <>}
> >> java.net.ConnectException: Connection refused
> >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> >> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> >> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> >> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:309)
> >> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:857)
> >> at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:725)
> >> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:253)
> >> at $Proxy10.openScanner(Unknown Source)
> >> at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:182)
> >> at
> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
> >> at
> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
> >> at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:156)
> >> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
> >>
> >>
> >>
> >>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Can master detect sudden region server death?
Posted by James Kennedy <ja...@troove.net>.
Hmm, yeah i waited well over the zk lease time for the HMaster to come out of the exception loop but it never did.
Thanks, i'll follow up with a look at Gremlins. How is it used within HBase? Do you continuously run gremlins on a cluster in an automated way?
James Kennedy
Project Manager
Troove Inc.
1 877 330 8501
On 2010-08-13, at 4:06 PM, Jean-Daniel Cryans wrote:
> The master will get a Watcher event from ZooKeeper when the region
> server's session is expired and its ephemeral znode is deleted. By
> default the session timeout is really high to cope with users with
> huge GC pauses problems, something like 1 minute (see
> hbase-default.xml).
>
> For a good fault testing framework, please use
> http://github.com/toddlipcon/gremlins. This was written by Todd Lipcon
> to test HBase's handling of region servers' death.
>
> J-D
>
> On Fri, Aug 13, 2010 at 3:59 PM, James Kennedy <ja...@troove.net> wrote:
>> For our system it is critical that there be no data loss and fast recovery time if any node goes down.
>>
>> We've recently updated the hbase-transactional-tableindexed extension to work with the latest 0.89.20100726 version of HBase (still to be pushed).
>> All HBase tests are passing but then when we started to write our own and test true sudden HRegionServer death we ran into trouble.
>> It seems that the HMaster does not recognize the kill even after many minutes. Client requests are blocked and the log continues to repeat the logs below.
>>
>> We realized that HBase's own tests that require RegionServer death use abort() and not kill() which does enough cleanup to inadequately simulate a sudden (e.g. JVM crash) death.
>>
>> As an experiment I made HRegionServer.kill() public and modified HBaseMiniCluster to call that from abort() instead. Now a test like TestMasterTransitions will exhibit similar behaviour: The HMaster never notices the RegionServer is gone.
>>
>> Could it really be that sudden region server death is not handled in hbase?
>> Or more likely is this a failure of the testing framework to adequately simulate kill -9?
>>
>> James Kennedy
>> Project Manager
>> Troove Inc.
>>
>>
>> -------------------------------
>>
>> [13/08/10 15:12:12] 259494 [n.serverMonitor] INFO oop.hbase.master.ServerManager - 2 region servers, 0 dead, average load 3.5
>> [13/08/10 15:12:12] 259560 [ger.metaScanner] INFO adoop.hbase.master.BaseScanner - RegionManager.metaScanner scanning meta region {server: 10.0.1.4:56908, regionname: .META.,,1.1028785192, startKey: <>}
>> [13/08/10 15:12:12] 259561 [ger.metaScanner] WARN adoop.hbase.master.BaseScanner - Scan one META region: {server: 10.0.1.4:56908, regionname: .META.,,1.1028785192, startKey: <>}
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>> at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:309)
>> at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:857)
>> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:725)
>> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:253)
>> at $Proxy10.openScanner(Unknown Source)
>> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:182)
>> at org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
>> at org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
>> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:156)
>> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
>>
>>
>>
>>
Re: Can master detect sudden region server death?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
The master will get a Watcher event from ZooKeeper when the region
server's session is expired and its ephemeral znode is deleted. By
default the session timeout is really high to cope with users with
huge GC pauses problems, something like 1 minute (see
hbase-default.xml).
For a good fault testing framework, please use
http://github.com/toddlipcon/gremlins. This was written by Todd Lipcon
to test HBase's handling of region servers' death.
J-D
On Fri, Aug 13, 2010 at 3:59 PM, James Kennedy <ja...@troove.net> wrote:
> For our system it is critical that there be no data loss and fast recovery time if any node goes down.
>
> We've recently updated the hbase-transactional-tableindexed extension to work with the latest 0.89.20100726 version of HBase (still to be pushed).
> All HBase tests are passing but then when we started to write our own and test true sudden HRegionServer death we ran into trouble.
> It seems that the HMaster does not recognize the kill even after many minutes. Client requests are blocked and the log continues to repeat the logs below.
>
> We realized that HBase's own tests that require RegionServer death use abort() and not kill() which does enough cleanup to inadequately simulate a sudden (e.g. JVM crash) death.
>
> As an experiment I made HRegionServer.kill() public and modified HBaseMiniCluster to call that from abort() instead. Now a test like TestMasterTransitions will exhibit similar behaviour: The HMaster never notices the RegionServer is gone.
>
> Could it really be that sudden region server death is not handled in hbase?
> Or more likely is this a failure of the testing framework to adequately simulate kill -9?
>
> James Kennedy
> Project Manager
> Troove Inc.
>
>
> -------------------------------
>
> [13/08/10 15:12:12] 259494 [n.serverMonitor] INFO oop.hbase.master.ServerManager - 2 region servers, 0 dead, average load 3.5
> [13/08/10 15:12:12] 259560 [ger.metaScanner] INFO adoop.hbase.master.BaseScanner - RegionManager.metaScanner scanning meta region {server: 10.0.1.4:56908, regionname: .META.,,1.1028785192, startKey: <>}
> [13/08/10 15:12:12] 259561 [ger.metaScanner] WARN adoop.hbase.master.BaseScanner - Scan one META region: {server: 10.0.1.4:56908, regionname: .META.,,1.1028785192, startKey: <>}
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:309)
> at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:857)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:725)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:253)
> at $Proxy10.openScanner(Unknown Source)
> at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:182)
> at org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
> at org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
> at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:156)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
>
>
>
>