You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2010/05/08 08:28:49 UTC
[jira] Resolved: (HBASE-2413) Master does not respect generation
stamps, may result in meta getting permanently offlined
[ https://issues.apache.org/jira/browse/HBASE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-2413.
--------------------------
Hadoop Flags: [Reviewed]
Fix Version/s: 0.21.0
Resolution: Fixed
Applied branch and trunk. Thanks for the double-review over in http://review.hbase.org/r/36/ Todd.
> Master does not respect generation stamps, may result in meta getting permanently offlined
> ------------------------------------------------------------------------------------------
>
> Key: HBASE-2413
> URL: https://issues.apache.org/jira/browse/HBASE-2413
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.20.3
> Reporter: Karthik Ranganathan
> Assignee: stack
> Fix For: 0.20.5, 0.21.0
>
> Attachments: 2413-v10.patch, 2413-v4.txt, 2413-v6.patch, 2413-v7.patch, 2413-v8.patch, 2413-v9.patch, newserver-v3.txt, newserver.txt
>
>
> This happens if the RS is restarted before the zk node expires. The sequence is as follows:
> 1. RS1 dies - lets say its server string was HOST1:PORT1:TS1
> 2. In a few seconds RS1 is restarted, it comes up as HOST1:PORT1:TS2 (TS2 is more recent than TS1)
> 3. Master gets a start up message from RS1 with the server name as HOST1:PORT1:TS2
> 4. Master adds this as a new RS, tries to red
> ---- The master does not use the generation stamps to detect that RS1 has already restarted.
> ---- Also, if RS1 contained meta, master would try to go to HOST1:PORT1:TS1. It would end up talking to HOST1:PORT1:TS2, which spews a bunch of not serving region exceptions.
> 5. zk node expires for HOST1:PORT1:TS1
> 6. Master tries to process shutdown for HOST1:PORT1:TS1 - this probably interferes with HOST1:PORT1:TS2 and ends up somehow removing the reassign meta in the master's queue.
> ---- Meta never comes online and master continues logging the following exception indefinitely:
> 2010-04-06 11:02:23,988 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of test1,7094000000,1270220428234, false, reassign: true
> 2010-04-06 11:02:23,988 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation:
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:64)
> at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63)
> at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:494)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:429)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Resolved: (HBASE-2413) Master does not respect generation stamps, may result in meta getting permanently offlined
Posted by Andrew Purtell <ap...@apache.org>.
Yeah it's private but I can always make a branch and apply test patches.
I really like our private Hudson. It runs on a server particularly sensitive to timing related changes for some reason. I've looked into what that might be but have not found anything conclusive.
I like the direction our testing framework is headed, should be easier to introduce timing perturbations in controlled manner.
- Andy
> From: Stack <st...@duboce.net>
> Subject: Re: [jira] Resolved: (HBASE-2413) Master does not respect generation stamps, may result in meta getting permanently offlined
> To: hbase-dev@hadoop.apache.org, apurtell@apache.org
> Date: Saturday, May 8, 2010, 12:05 PM
> Excellent. Let me take a look
> later today. Your hudson is private I
> suppose? Your forwarding the likes of below is
> helpful.
> St.Ack
>
> On Sat, May 8, 2010 at 11:10 AM, Andrew Purtell <ap...@apache.org>
> wrote:
> > Our fickle Hudson doesn't like this:
> >
> > With this project we just take up 0.20 branch changes
> from git.apache.org and run them.
> >
> >>>>
> >
> > Build #33 (May 8, 2010 7:44:17 AM) FAILED
> >
> > Changes
> > HBASE-2413 Master does not respect generation stamps,
> may result in meta getting permanently offlined (detail)
> > HBASE-2413 Master does not respect generation stamps,
> may result in meta getting permanently offlined (detail)
> >
> > Started by an SCM change (2 times)
> >
> > Test Result (2 failures / +2)
> >
> org.apache.hadoop.hbase.TestZooKeeper.testRegionServerSessionExpired
> >
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileScanning
> >
> > <<<
> >
> > The ZK failure seems related:
> >
> >
> org.apache.hadoop.hbase.master.NotAllMetaRegionsOnlineException
> at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:709)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597) at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> > Stacktrace
> >
> >
> org.apache.hadoop.hbase.master.NotAllMetaRegionsOnlineException:
> org.apache.hadoop.hbase.master.NotAllMetaRegionsOnlineException
> > at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:709)
> > at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
> > at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> >
> > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> > at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:282)
> > at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:239)
> > at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:165)
> > at
> org.apache.hadoop.hbase.TestZooKeeper.sessionExpirationHelper(TestZooKeeper.java:152)
> > at
> org.apache.hadoop.hbase.TestZooKeeper.testRegionServerSessionExpired(TestZooKeeper.java:122)
> >
> >
> >
> >
> >
> >
>
Re: [jira] Resolved: (HBASE-2413) Master does not respect generation
stamps, may result in meta getting permanently offlined
Posted by Stack <st...@duboce.net>.
Excellent. Let me take a look later today. Your hudson is private I
suppose? Your forwarding the likes of below is helpful.
St.Ack
On Sat, May 8, 2010 at 11:10 AM, Andrew Purtell <ap...@apache.org> wrote:
> Our fickle Hudson doesn't like this:
>
> With this project we just take up 0.20 branch changes from git.apache.org and run them.
>
>>>>
>
> Build #33 (May 8, 2010 7:44:17 AM) FAILED
>
> Changes
> HBASE-2413 Master does not respect generation stamps, may result in meta getting permanently offlined (detail)
> HBASE-2413 Master does not respect generation stamps, may result in meta getting permanently offlined (detail)
>
> Started by an SCM change (2 times)
>
> Test Result (2 failures / +2)
> org.apache.hadoop.hbase.TestZooKeeper.testRegionServerSessionExpired
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileScanning
>
> <<<
>
> The ZK failure seems related:
>
> org.apache.hadoop.hbase.master.NotAllMetaRegionsOnlineException at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:709) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> Stacktrace
>
> org.apache.hadoop.hbase.master.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.master.NotAllMetaRegionsOnlineException
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:709)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
> at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:282)
> at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:239)
> at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:165)
> at org.apache.hadoop.hbase.TestZooKeeper.sessionExpirationHelper(TestZooKeeper.java:152)
> at org.apache.hadoop.hbase.TestZooKeeper.testRegionServerSessionExpired(TestZooKeeper.java:122)
>
>
>
>
>
>
Re: [jira] Resolved: (HBASE-2413) Master does not respect generation stamps, may result in meta getting permanently offlined
Posted by Andrew Purtell <ap...@apache.org>.
Our fickle Hudson doesn't like this:
With this project we just take up 0.20 branch changes from git.apache.org and run them.
>>>
Build #33 (May 8, 2010 7:44:17 AM) FAILED
Changes
HBASE-2413 Master does not respect generation stamps, may result in meta getting permanently offlined (detail)
HBASE-2413 Master does not respect generation stamps, may result in meta getting permanently offlined (detail)
Started by an SCM change (2 times)
Test Result (2 failures / +2)
org.apache.hadoop.hbase.TestZooKeeper.testRegionServerSessionExpired
org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileScanning
<<<
The ZK failure seems related:
org.apache.hadoop.hbase.master.NotAllMetaRegionsOnlineException at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:709) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Stacktrace
org.apache.hadoop.hbase.master.NotAllMetaRegionsOnlineException: org.apache.hadoop.hbase.master.NotAllMetaRegionsOnlineException
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:709)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:282)
at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:239)
at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:165)
at org.apache.hadoop.hbase.TestZooKeeper.sessionExpirationHelper(TestZooKeeper.java:152)
at org.apache.hadoop.hbase.TestZooKeeper.testRegionServerSessionExpired(TestZooKeeper.java:122)