You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Matt Corgan <mc...@hotpads.com> on 2010/09/21 22:07:29 UTC
lost .META. table
Is there a way to create a new .META. table if I lost it? For example, I'm
getting this exception everywhere:
org.apache.hadoop.hbase.TableNotFoundException: .META. (NativeException)
The add_table script doesn't appear to handle .META., giving this error:
java.lang.IllegalArgumentException: Illegal first character <46> at 0.
User-space table names can only start with 'word characters': i.e.
[a-zA-Z_0-9]: .META.
And the hbase shell will only create user-space table names. Restarting the
hbase master also does not appear to re-create it.
I'm using 0.20.6.
Re: lost .META. table
Posted by Matt Corgan <mc...@hotpads.com>.
Thanks Stack - worked like a charm, and I think all my data is there. So
does the content of -ROOT- get overwritten, because I think it was different
than it should have been?
On Tue, Sep 21, 2010 at 7:25 PM, Stack <st...@duboce.net> wrote:
> Ok. Thanks. I don't know why its not assigning .META. Trying
> starting the cluster with a different hbase.rootdir then move the
> -ROOT- from the new hbase.rootdir in place of the -ROOT- you currently
> have.
>
> St.Ack
>
> On Tue, Sep 21, 2010 at 4:17 PM, Stack <st...@duboce.net> wrote:
> > Please put the first 1k lines after startup up in pastebin. The below
> > is hard to read. Its scanning the -ROOT- but there is no mention of
> > .META. being assigned out.
> >
> > St.Ack
> >
> > On Tue, Sep 21, 2010 at 3:09 PM, Matt Corgan <mc...@hotpads.com>
> wrote:
> >> I think this is the relevant section:
> >>
> >> 2010-09-21 15:42:11,986 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> >> Received start message from: HadoopNode41,60020,1285098132069
> >> 2010-09-21 15:42:11,989 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> >> Received start message from: HadoopNode38,60020,1285098131987
> >> 2010-09-21 15:42:11,989 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> >> /hbase/rs/1285098132069 with data 10.214.7.223:60020
> >> 2010-09-21 15:42:11,996 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> >> /hbase/rs/1285098131987 with data 10.209.206.15:60020
> >> 2010-09-21 15:42:13,001 DEBUG
> org.apache.hadoop.hbase.master.RegionManager:
> >> Assigning for address: 10.209.206.15:60020, startcode: 1285098131987,
> load:
> >> (requests=0, regions=0, usedHeap=32, maxHeap=5195): total nregions to
> >> assign=1, regions to give other servers than this=0, isMetaAssign=true
> >> 2010-09-21 15:42:13,001 DEBUG
> org.apache.hadoop.hbase.master.RegionManager:
> >> Assigning address: 10.209.206.15:60020, startcode: 1285098131987, load:
> >> (requests=0, regions=0, usedHeap=32, maxHeap=5195) 1 regions
> >> 2010-09-21 15:42:13,001 INFO
> org.apache.hadoop.hbase.master.RegionManager:
> >> Assigning region -ROOT-,,0 to HadoopNode38,60020,1285098131987
> >> 2010-09-21 15:42:13,085 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> >> Processing MSG_REPORT_OPEN: -ROOT-,,0 from
> HadoopNode38,60020,1285098131987;
> >> 1 of 1
> >> 2010-09-21 15:42:13,130 WARN
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create out
> of
> >> safe mode in ZooKeeper:
> >> org.apache.zookeeper.KeeperException$NodeExistsException:
> KeeperErrorCode =
> >> NodeExists for /hbase/safe-mode
> >> 2010-09-21 15:42:13,130 INFO
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Node exists; just
> move
> >> on
> >> 2010-09-21 15:42:13,130 INFO
> org.apache.hadoop.hbase.master.RegionManager:
> >> exiting safe mode
> >> 2010-09-21 15:42:13,132 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: SetData of ZNode
> >> /hbase/root-region-server with 10.209.206.15:60020
> >> 2010-09-21 15:42:13,133 INFO org.apache.hadoop.hbase.master.BaseScanner:
> >> RegionManager.rootScanner scanning meta region {server:
> 10.209.206.15:60020,
> >> regionname: -ROOT-,,0, startKey: <>}
> >> 2010-09-21 15:42:13,223 INFO org.apache.hadoop.hbase.master.BaseScanner:
> >> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
> >> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
> >> 2010-09-21 15:42:13,466 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> >> Received start message from: HadoopNode40,60020,1285098133450
> >> 2010-09-21 15:42:13,468 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> >> /hbase/rs/1285098133450 with data 10.192.66.240:60020
> >> 2010-09-21 15:42:13,828 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> >> Received start message from: HadoopNode39,60020,1285098133654
> >> 2010-09-21 15:42:13,830 DEBUG
> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> >> /hbase/rs/1285098133654 with data 10.206.26.95:60020
> >> 2010-09-21 15:43:10,256 INFO
> org.apache.hadoop.hbase.master.ServerManager: 4
> >> region servers, 0 dead, average load 0.25
> >> 2010-09-21 15:43:10,341 INFO org.apache.hadoop.hbase.master.BaseScanner:
> >> RegionManager.rootScanner scanning meta region {server:
> 10.209.206.15:60020,
> >> regionname: -ROOT-,,0, startKey: <>}
> >> 2010-09-21 15:43:10,345 INFO org.apache.hadoop.hbase.master.BaseScanner:
> >> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
> >> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
> >> 2010-09-21 15:43:10,354 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All
> >> 0 .META. region(s) scanned
> >> 2010-09-21 15:43:56,351 INFO org.apache.zookeeper.ZooKeeper: Initiating
> >> client connection, connectString=10.249.107.223:2181sessionTimeout=60000
> >>
> watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@72898540
> >> 2010-09-21 15:43:56,352 INFO org.apache.zookeeper.ClientCnxn: Attempting
> >> connection to server /10.249.107.223:2181
> >> 2010-09-21 15:43:56,362 INFO org.apache.zookeeper.ClientCnxn: Priming
> >> connection to java.nio.channels.SocketChannel[connected local=/
> >> 10.249.107.223:56828remote=/10.249.107.223:2181]
> >> 2010-09-21 15:43:56,380 INFO org.apache.zookeeper.ClientCnxn: Server
> >> connection successful
> >>
> >>
> >> On Tue, Sep 21, 2010 at 6:05 PM, Stack <st...@duboce.net> wrote:
> >>
> >>> Ok. So .META. is there. What is in your master log on startup? Its
> >>> supposed to assign out -ROOT- and .META. (when .META. is assigned
> >>> it'll add row to -ROOT-). I'd say hold off on any of the trickery
> >>> described in the first mail till we see snippet from your master log.
> >>> Lets figure why not assigning .META. before we do any messing.
> >>>
> >>> Thanks,
> >>> St.Ack
> >>>
> >>> On Tue, Sep 21, 2010 at 2:52 PM, Matt Corgan <mc...@hotpads.com>
> wrote:
> >>> > Thanks for responding Stack. In a 4 slave cluster (rf=3) cluster we
> >>> > disabled an hdfs/hbase node which also seemed to make a second
> >>> regionserver
> >>> > crash. I'm not sure exactly what went wrong.
> >>> >
> >>> > There is data in the .META. directory
> >>> >
> >>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs
> -lsr
> >>> > /hbase/.META.
> >>> > drwxr-xr-x - root supergroup 0 2010-09-20 16:44
> >>> > /hbase/.META./1028785192
> >>> > -rw-r--r-- 3 root supergroup 932 2010-09-20 14:25
> >>> > /hbase/.META./1028785192/.regioninfo
> >>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41
> >>> > /hbase/.META./1028785192/info
> >>> > -rw-r--r-- 3 root supergroup 5018 2010-09-21 15:41
> >>> > /hbase/.META./1028785192/info/4694688683869779325
> >>> > -rw-r--r-- 3 root supergroup 0 2010-09-21 14:48
> >>> > /hbase/.META./1028785192/info/6832908118284006198
> >>> > -rw-r--r-- 3 root supergroup 228130 2010-09-21 14:44
> >>> > /hbase/.META./1028785192/info/7773537252698623585
> >>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:44
> >>> > /hbase/.META./compaction.dir
> >>> >
> >>> >
> >>> > but it appears the -ROOT- region is empty. If i scan '-ROOT-' in the
> >>> hbase
> >>> > shell i get:
> >>> >
> >>> > hbase(main):001:0> scan '-ROOT-'
> >>> > ROW COLUMN+CELL
> >>> > 0 row(s) in 0.0840 seconds
> >>> >
> >>> >
> >>> > even though it appears -ROOT- exists in hdfs. Any idea why ROOT
> would
> >>> not
> >>> > get loaded from hdfs after a master restart?
> >>> >
> >>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs
> -lsr
> >>> > /hbase/-ROOT-
> >>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:50
> >>> > /hbase/-ROOT-/70236052
> >>> > -rw-r--r-- 3 root supergroup 618 2010-09-21 14:50
> >>> > /hbase/-ROOT-/70236052/.regioninfo
> >>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41
> >>> > /hbase/-ROOT-/70236052/info
> >>> > -rw-r--r-- 3 root supergroup 489 2010-09-21 15:41
> >>> > /hbase/-ROOT-/70236052/info/308400263036654323
> >>> > -rw-r--r-- 3 root supergroup 462 2010-09-21 13:43
> >>> > /hbase/-ROOT-/70236052/info/8250773031047005400
> >>> >
> >>> >
> >>> > I'll just follow your suggestion of generating a new .META. and
> copying
> >>> > over. Thanks,
> >>> >
> >>> > Matt
> >>> >
> >>> >
> >>> > On Tue, Sep 21, 2010 at 5:20 PM, Stack <st...@duboce.net> wrote:
> >>> >
> >>> >> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <mc...@hotpads.com>
> >>> wrote:
> >>> >> > Is there a way to create a new .META. table if I lost it? For
> >>> example,
> >>> >> I'm
> >>> >> > getting this exception everywhere:
> >>> >> >
> >>> >> > org.apache.hadoop.hbase.TableNotFoundException: .META.
> >>> (NativeException)
> >>> >> >
> >>> >>
> >>> >> How'd you do that? Thats pretty impressive.
> >>> >>
> >>> >> >
> >>> >> > The add_table script doesn't appear to handle .META., giving this
> >>> error:
> >>> >> >
> >>> >> > java.lang.IllegalArgumentException: Illegal first character <46>
> at 0.
> >>> >> > User-space table names can only start with 'word characters': i.e.
> >>> >> > [a-zA-Z_0-9]: .META.
> >>> >> >
> >>> >> >
> >>> >> > And the hbase shell will only create user-space table names.
> >>> Restarting
> >>> >> the
> >>> >> > hbase master also does not appear to re-create it.
> >>> >> >
> >>> >> > I'm using 0.20.6.
> >>> >> >
> >>> >> Whats the master log look like when you start up hbase? Whats it
> say
> >>> >> about missing .META.?
> >>> >>
> >>> >> If you do a
> >>> >>
> >>> >> $ bin/hadoop fs -lsr /hbase/.META.
> >>> >>
> >>> >> ... is there anything in your filesystem?
> >>> >>
> >>> >> If nothing there, one thing you could do is run an hbase elsewhere.
> >>> >> Let it create a new "/hbase" dir and then copy the .META. table from
> >>> >> here up to the filesystem with data (Run add_table.rb after .META.
> is
> >>> >> in place to repopulate its rows).
> >>> >>
> >>> >> St.Ack
> >>> >>
> >>> >
> >>>
> >>
> >
>
Re: lost .META. table
Posted by Stack <st...@duboce.net>.
Ok. Thanks. I don't know why its not assigning .META. Trying
starting the cluster with a different hbase.rootdir then move the
-ROOT- from the new hbase.rootdir in place of the -ROOT- you currently
have.
St.Ack
On Tue, Sep 21, 2010 at 4:17 PM, Stack <st...@duboce.net> wrote:
> Please put the first 1k lines after startup up in pastebin. The below
> is hard to read. Its scanning the -ROOT- but there is no mention of
> .META. being assigned out.
>
> St.Ack
>
> On Tue, Sep 21, 2010 at 3:09 PM, Matt Corgan <mc...@hotpads.com> wrote:
>> I think this is the relevant section:
>>
>> 2010-09-21 15:42:11,986 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received start message from: HadoopNode41,60020,1285098132069
>> 2010-09-21 15:42:11,989 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received start message from: HadoopNode38,60020,1285098131987
>> 2010-09-21 15:42:11,989 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
>> /hbase/rs/1285098132069 with data 10.214.7.223:60020
>> 2010-09-21 15:42:11,996 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
>> /hbase/rs/1285098131987 with data 10.209.206.15:60020
>> 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager:
>> Assigning for address: 10.209.206.15:60020, startcode: 1285098131987, load:
>> (requests=0, regions=0, usedHeap=32, maxHeap=5195): total nregions to
>> assign=1, regions to give other servers than this=0, isMetaAssign=true
>> 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager:
>> Assigning address: 10.209.206.15:60020, startcode: 1285098131987, load:
>> (requests=0, regions=0, usedHeap=32, maxHeap=5195) 1 regions
>> 2010-09-21 15:42:13,001 INFO org.apache.hadoop.hbase.master.RegionManager:
>> Assigning region -ROOT-,,0 to HadoopNode38,60020,1285098131987
>> 2010-09-21 15:42:13,085 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Processing MSG_REPORT_OPEN: -ROOT-,,0 from HadoopNode38,60020,1285098131987;
>> 1 of 1
>> 2010-09-21 15:42:13,130 WARN
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create out of
>> safe mode in ZooKeeper:
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists for /hbase/safe-mode
>> 2010-09-21 15:42:13,130 INFO
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Node exists; just move
>> on
>> 2010-09-21 15:42:13,130 INFO org.apache.hadoop.hbase.master.RegionManager:
>> exiting safe mode
>> 2010-09-21 15:42:13,132 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: SetData of ZNode
>> /hbase/root-region-server with 10.209.206.15:60020
>> 2010-09-21 15:42:13,133 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.209.206.15:60020,
>> regionname: -ROOT-,,0, startKey: <>}
>> 2010-09-21 15:42:13,223 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
>> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
>> 2010-09-21 15:42:13,466 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received start message from: HadoopNode40,60020,1285098133450
>> 2010-09-21 15:42:13,468 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
>> /hbase/rs/1285098133450 with data 10.192.66.240:60020
>> 2010-09-21 15:42:13,828 INFO org.apache.hadoop.hbase.master.ServerManager:
>> Received start message from: HadoopNode39,60020,1285098133654
>> 2010-09-21 15:42:13,830 DEBUG
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
>> /hbase/rs/1285098133654 with data 10.206.26.95:60020
>> 2010-09-21 15:43:10,256 INFO org.apache.hadoop.hbase.master.ServerManager: 4
>> region servers, 0 dead, average load 0.25
>> 2010-09-21 15:43:10,341 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {server: 10.209.206.15:60020,
>> regionname: -ROOT-,,0, startKey: <>}
>> 2010-09-21 15:43:10,345 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
>> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
>> 2010-09-21 15:43:10,354 INFO org.apache.hadoop.hbase.master.BaseScanner: All
>> 0 .META. region(s) scanned
>> 2010-09-21 15:43:56,351 INFO org.apache.zookeeper.ZooKeeper: Initiating
>> client connection, connectString=10.249.107.223:2181 sessionTimeout=60000
>> watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@72898540
>> 2010-09-21 15:43:56,352 INFO org.apache.zookeeper.ClientCnxn: Attempting
>> connection to server /10.249.107.223:2181
>> 2010-09-21 15:43:56,362 INFO org.apache.zookeeper.ClientCnxn: Priming
>> connection to java.nio.channels.SocketChannel[connected local=/
>> 10.249.107.223:56828remote=/10.249.107.223:2181]
>> 2010-09-21 15:43:56,380 INFO org.apache.zookeeper.ClientCnxn: Server
>> connection successful
>>
>>
>> On Tue, Sep 21, 2010 at 6:05 PM, Stack <st...@duboce.net> wrote:
>>
>>> Ok. So .META. is there. What is in your master log on startup? Its
>>> supposed to assign out -ROOT- and .META. (when .META. is assigned
>>> it'll add row to -ROOT-). I'd say hold off on any of the trickery
>>> described in the first mail till we see snippet from your master log.
>>> Lets figure why not assigning .META. before we do any messing.
>>>
>>> Thanks,
>>> St.Ack
>>>
>>> On Tue, Sep 21, 2010 at 2:52 PM, Matt Corgan <mc...@hotpads.com> wrote:
>>> > Thanks for responding Stack. In a 4 slave cluster (rf=3) cluster we
>>> > disabled an hdfs/hbase node which also seemed to make a second
>>> regionserver
>>> > crash. I'm not sure exactly what went wrong.
>>> >
>>> > There is data in the .META. directory
>>> >
>>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr
>>> > /hbase/.META.
>>> > drwxr-xr-x - root supergroup 0 2010-09-20 16:44
>>> > /hbase/.META./1028785192
>>> > -rw-r--r-- 3 root supergroup 932 2010-09-20 14:25
>>> > /hbase/.META./1028785192/.regioninfo
>>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41
>>> > /hbase/.META./1028785192/info
>>> > -rw-r--r-- 3 root supergroup 5018 2010-09-21 15:41
>>> > /hbase/.META./1028785192/info/4694688683869779325
>>> > -rw-r--r-- 3 root supergroup 0 2010-09-21 14:48
>>> > /hbase/.META./1028785192/info/6832908118284006198
>>> > -rw-r--r-- 3 root supergroup 228130 2010-09-21 14:44
>>> > /hbase/.META./1028785192/info/7773537252698623585
>>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:44
>>> > /hbase/.META./compaction.dir
>>> >
>>> >
>>> > but it appears the -ROOT- region is empty. If i scan '-ROOT-' in the
>>> hbase
>>> > shell i get:
>>> >
>>> > hbase(main):001:0> scan '-ROOT-'
>>> > ROW COLUMN+CELL
>>> > 0 row(s) in 0.0840 seconds
>>> >
>>> >
>>> > even though it appears -ROOT- exists in hdfs. Any idea why ROOT would
>>> not
>>> > get loaded from hdfs after a master restart?
>>> >
>>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr
>>> > /hbase/-ROOT-
>>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:50
>>> > /hbase/-ROOT-/70236052
>>> > -rw-r--r-- 3 root supergroup 618 2010-09-21 14:50
>>> > /hbase/-ROOT-/70236052/.regioninfo
>>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41
>>> > /hbase/-ROOT-/70236052/info
>>> > -rw-r--r-- 3 root supergroup 489 2010-09-21 15:41
>>> > /hbase/-ROOT-/70236052/info/308400263036654323
>>> > -rw-r--r-- 3 root supergroup 462 2010-09-21 13:43
>>> > /hbase/-ROOT-/70236052/info/8250773031047005400
>>> >
>>> >
>>> > I'll just follow your suggestion of generating a new .META. and copying
>>> > over. Thanks,
>>> >
>>> > Matt
>>> >
>>> >
>>> > On Tue, Sep 21, 2010 at 5:20 PM, Stack <st...@duboce.net> wrote:
>>> >
>>> >> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <mc...@hotpads.com>
>>> wrote:
>>> >> > Is there a way to create a new .META. table if I lost it? For
>>> example,
>>> >> I'm
>>> >> > getting this exception everywhere:
>>> >> >
>>> >> > org.apache.hadoop.hbase.TableNotFoundException: .META.
>>> (NativeException)
>>> >> >
>>> >>
>>> >> How'd you do that? Thats pretty impressive.
>>> >>
>>> >> >
>>> >> > The add_table script doesn't appear to handle .META., giving this
>>> error:
>>> >> >
>>> >> > java.lang.IllegalArgumentException: Illegal first character <46> at 0.
>>> >> > User-space table names can only start with 'word characters': i.e.
>>> >> > [a-zA-Z_0-9]: .META.
>>> >> >
>>> >> >
>>> >> > And the hbase shell will only create user-space table names.
>>> Restarting
>>> >> the
>>> >> > hbase master also does not appear to re-create it.
>>> >> >
>>> >> > I'm using 0.20.6.
>>> >> >
>>> >> Whats the master log look like when you start up hbase? Whats it say
>>> >> about missing .META.?
>>> >>
>>> >> If you do a
>>> >>
>>> >> $ bin/hadoop fs -lsr /hbase/.META.
>>> >>
>>> >> ... is there anything in your filesystem?
>>> >>
>>> >> If nothing there, one thing you could do is run an hbase elsewhere.
>>> >> Let it create a new "/hbase" dir and then copy the .META. table from
>>> >> here up to the filesystem with data (Run add_table.rb after .META. is
>>> >> in place to repopulate its rows).
>>> >>
>>> >> St.Ack
>>> >>
>>> >
>>>
>>
>
Re: lost .META. table
Posted by Stack <st...@duboce.net>.
Please put the first 1k lines after startup up in pastebin. The below
is hard to read. Its scanning the -ROOT- but there is no mention of
.META. being assigned out.
St.Ack
On Tue, Sep 21, 2010 at 3:09 PM, Matt Corgan <mc...@hotpads.com> wrote:
> I think this is the relevant section:
>
> 2010-09-21 15:42:11,986 INFO org.apache.hadoop.hbase.master.ServerManager:
> Received start message from: HadoopNode41,60020,1285098132069
> 2010-09-21 15:42:11,989 INFO org.apache.hadoop.hbase.master.ServerManager:
> Received start message from: HadoopNode38,60020,1285098131987
> 2010-09-21 15:42:11,989 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> /hbase/rs/1285098132069 with data 10.214.7.223:60020
> 2010-09-21 15:42:11,996 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> /hbase/rs/1285098131987 with data 10.209.206.15:60020
> 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager:
> Assigning for address: 10.209.206.15:60020, startcode: 1285098131987, load:
> (requests=0, regions=0, usedHeap=32, maxHeap=5195): total nregions to
> assign=1, regions to give other servers than this=0, isMetaAssign=true
> 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager:
> Assigning address: 10.209.206.15:60020, startcode: 1285098131987, load:
> (requests=0, regions=0, usedHeap=32, maxHeap=5195) 1 regions
> 2010-09-21 15:42:13,001 INFO org.apache.hadoop.hbase.master.RegionManager:
> Assigning region -ROOT-,,0 to HadoopNode38,60020,1285098131987
> 2010-09-21 15:42:13,085 INFO org.apache.hadoop.hbase.master.ServerManager:
> Processing MSG_REPORT_OPEN: -ROOT-,,0 from HadoopNode38,60020,1285098131987;
> 1 of 1
> 2010-09-21 15:42:13,130 WARN
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create out of
> safe mode in ZooKeeper:
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists for /hbase/safe-mode
> 2010-09-21 15:42:13,130 INFO
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Node exists; just move
> on
> 2010-09-21 15:42:13,130 INFO org.apache.hadoop.hbase.master.RegionManager:
> exiting safe mode
> 2010-09-21 15:42:13,132 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: SetData of ZNode
> /hbase/root-region-server with 10.209.206.15:60020
> 2010-09-21 15:42:13,133 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 10.209.206.15:60020,
> regionname: -ROOT-,,0, startKey: <>}
> 2010-09-21 15:42:13,223 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
> 2010-09-21 15:42:13,466 INFO org.apache.hadoop.hbase.master.ServerManager:
> Received start message from: HadoopNode40,60020,1285098133450
> 2010-09-21 15:42:13,468 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> /hbase/rs/1285098133450 with data 10.192.66.240:60020
> 2010-09-21 15:42:13,828 INFO org.apache.hadoop.hbase.master.ServerManager:
> Received start message from: HadoopNode39,60020,1285098133654
> 2010-09-21 15:42:13,830 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> /hbase/rs/1285098133654 with data 10.206.26.95:60020
> 2010-09-21 15:43:10,256 INFO org.apache.hadoop.hbase.master.ServerManager: 4
> region servers, 0 dead, average load 0.25
> 2010-09-21 15:43:10,341 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 10.209.206.15:60020,
> regionname: -ROOT-,,0, startKey: <>}
> 2010-09-21 15:43:10,345 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
> 2010-09-21 15:43:10,354 INFO org.apache.hadoop.hbase.master.BaseScanner: All
> 0 .META. region(s) scanned
> 2010-09-21 15:43:56,351 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=10.249.107.223:2181 sessionTimeout=60000
> watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@72898540
> 2010-09-21 15:43:56,352 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server /10.249.107.223:2181
> 2010-09-21 15:43:56,362 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 10.249.107.223:56828remote=/10.249.107.223:2181]
> 2010-09-21 15:43:56,380 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
>
>
> On Tue, Sep 21, 2010 at 6:05 PM, Stack <st...@duboce.net> wrote:
>
>> Ok. So .META. is there. What is in your master log on startup? Its
>> supposed to assign out -ROOT- and .META. (when .META. is assigned
>> it'll add row to -ROOT-). I'd say hold off on any of the trickery
>> described in the first mail till we see snippet from your master log.
>> Lets figure why not assigning .META. before we do any messing.
>>
>> Thanks,
>> St.Ack
>>
>> On Tue, Sep 21, 2010 at 2:52 PM, Matt Corgan <mc...@hotpads.com> wrote:
>> > Thanks for responding Stack. In a 4 slave cluster (rf=3) cluster we
>> > disabled an hdfs/hbase node which also seemed to make a second
>> regionserver
>> > crash. I'm not sure exactly what went wrong.
>> >
>> > There is data in the .META. directory
>> >
>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr
>> > /hbase/.META.
>> > drwxr-xr-x - root supergroup 0 2010-09-20 16:44
>> > /hbase/.META./1028785192
>> > -rw-r--r-- 3 root supergroup 932 2010-09-20 14:25
>> > /hbase/.META./1028785192/.regioninfo
>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41
>> > /hbase/.META./1028785192/info
>> > -rw-r--r-- 3 root supergroup 5018 2010-09-21 15:41
>> > /hbase/.META./1028785192/info/4694688683869779325
>> > -rw-r--r-- 3 root supergroup 0 2010-09-21 14:48
>> > /hbase/.META./1028785192/info/6832908118284006198
>> > -rw-r--r-- 3 root supergroup 228130 2010-09-21 14:44
>> > /hbase/.META./1028785192/info/7773537252698623585
>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:44
>> > /hbase/.META./compaction.dir
>> >
>> >
>> > but it appears the -ROOT- region is empty. If i scan '-ROOT-' in the
>> hbase
>> > shell i get:
>> >
>> > hbase(main):001:0> scan '-ROOT-'
>> > ROW COLUMN+CELL
>> > 0 row(s) in 0.0840 seconds
>> >
>> >
>> > even though it appears -ROOT- exists in hdfs. Any idea why ROOT would
>> not
>> > get loaded from hdfs after a master restart?
>> >
>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr
>> > /hbase/-ROOT-
>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:50
>> > /hbase/-ROOT-/70236052
>> > -rw-r--r-- 3 root supergroup 618 2010-09-21 14:50
>> > /hbase/-ROOT-/70236052/.regioninfo
>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41
>> > /hbase/-ROOT-/70236052/info
>> > -rw-r--r-- 3 root supergroup 489 2010-09-21 15:41
>> > /hbase/-ROOT-/70236052/info/308400263036654323
>> > -rw-r--r-- 3 root supergroup 462 2010-09-21 13:43
>> > /hbase/-ROOT-/70236052/info/8250773031047005400
>> >
>> >
>> > I'll just follow your suggestion of generating a new .META. and copying
>> > over. Thanks,
>> >
>> > Matt
>> >
>> >
>> > On Tue, Sep 21, 2010 at 5:20 PM, Stack <st...@duboce.net> wrote:
>> >
>> >> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <mc...@hotpads.com>
>> wrote:
>> >> > Is there a way to create a new .META. table if I lost it? For
>> example,
>> >> I'm
>> >> > getting this exception everywhere:
>> >> >
>> >> > org.apache.hadoop.hbase.TableNotFoundException: .META.
>> (NativeException)
>> >> >
>> >>
>> >> How'd you do that? Thats pretty impressive.
>> >>
>> >> >
>> >> > The add_table script doesn't appear to handle .META., giving this
>> error:
>> >> >
>> >> > java.lang.IllegalArgumentException: Illegal first character <46> at 0.
>> >> > User-space table names can only start with 'word characters': i.e.
>> >> > [a-zA-Z_0-9]: .META.
>> >> >
>> >> >
>> >> > And the hbase shell will only create user-space table names.
>> Restarting
>> >> the
>> >> > hbase master also does not appear to re-create it.
>> >> >
>> >> > I'm using 0.20.6.
>> >> >
>> >> Whats the master log look like when you start up hbase? Whats it say
>> >> about missing .META.?
>> >>
>> >> If you do a
>> >>
>> >> $ bin/hadoop fs -lsr /hbase/.META.
>> >>
>> >> ... is there anything in your filesystem?
>> >>
>> >> If nothing there, one thing you could do is run an hbase elsewhere.
>> >> Let it create a new "/hbase" dir and then copy the .META. table from
>> >> here up to the filesystem with data (Run add_table.rb after .META. is
>> >> in place to repopulate its rows).
>> >>
>> >> St.Ack
>> >>
>> >
>>
>
Re: lost .META. table
Posted by Matt Corgan <mc...@hotpads.com>.
sorry - maybe this is better? http://pastebin.com/gerjBkMT
On Tue, Sep 21, 2010 at 6:09 PM, Matt Corgan <mc...@hotpads.com> wrote:
> I think this is the relevant section:
>
> 2010-09-21 15:42:11,986 INFO org.apache.hadoop.hbase.master.ServerManager:
> Received start message from: HadoopNode41,60020,1285098132069
> 2010-09-21 15:42:11,989 INFO org.apache.hadoop.hbase.master.ServerManager:
> Received start message from: HadoopNode38,60020,1285098131987
> 2010-09-21 15:42:11,989 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> /hbase/rs/1285098132069 with data 10.214.7.223:60020
> 2010-09-21 15:42:11,996 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> /hbase/rs/1285098131987 with data 10.209.206.15:60020
> 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager:
> Assigning for address: 10.209.206.15:60020, startcode: 1285098131987,
> load: (requests=0, regions=0, usedHeap=32, maxHeap=5195): total nregions to
> assign=1, regions to give other servers than this=0, isMetaAssign=true
> 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager:
> Assigning address: 10.209.206.15:60020, startcode: 1285098131987, load:
> (requests=0, regions=0, usedHeap=32, maxHeap=5195) 1 regions
> 2010-09-21 15:42:13,001 INFO org.apache.hadoop.hbase.master.RegionManager:
> Assigning region -ROOT-,,0 to HadoopNode38,60020,1285098131987
> 2010-09-21 15:42:13,085 INFO org.apache.hadoop.hbase.master.ServerManager:
> Processing MSG_REPORT_OPEN: -ROOT-,,0 from HadoopNode38,60020,1285098131987;
> 1 of 1
> 2010-09-21 15:42:13,130 WARN
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create out of
> safe mode in ZooKeeper:
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists for /hbase/safe-mode
> 2010-09-21 15:42:13,130 INFO
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Node exists; just move
> on
> 2010-09-21 15:42:13,130 INFO org.apache.hadoop.hbase.master.RegionManager:
> exiting safe mode
> 2010-09-21 15:42:13,132 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: SetData of ZNode
> /hbase/root-region-server with 10.209.206.15:60020
> 2010-09-21 15:42:13,133 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server:
> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-09-21 15:42:13,223 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
> 2010-09-21 15:42:13,466 INFO org.apache.hadoop.hbase.master.ServerManager:
> Received start message from: HadoopNode40,60020,1285098133450
> 2010-09-21 15:42:13,468 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> /hbase/rs/1285098133450 with data 10.192.66.240:60020
> 2010-09-21 15:42:13,828 INFO org.apache.hadoop.hbase.master.ServerManager:
> Received start message from: HadoopNode39,60020,1285098133654
> 2010-09-21 15:42:13,830 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
> /hbase/rs/1285098133654 with data 10.206.26.95:60020
> 2010-09-21 15:43:10,256 INFO org.apache.hadoop.hbase.master.ServerManager:
> 4 region servers, 0 dead, average load 0.25
> 2010-09-21 15:43:10,341 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server:
> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>}
> 2010-09-21 15:43:10,345 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 0 row(s) of meta region {server:
> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
> 2010-09-21 15:43:10,354 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 0 .META. region(s) scanned
> 2010-09-21 15:43:56,351 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=10.249.107.223:2181 sessionTimeout=60000
> watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@72898540
> 2010-09-21 15:43:56,352 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server /10.249.107.223:2181
> 2010-09-21 15:43:56,362 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 10.249.107.223:56828remote=/10.249.107.223:2181]
> 2010-09-21 15:43:56,380 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
>
>
>
> On Tue, Sep 21, 2010 at 6:05 PM, Stack <st...@duboce.net> wrote:
>
>> Ok. So .META. is there. What is in your master log on startup? Its
>> supposed to assign out -ROOT- and .META. (when .META. is assigned
>> it'll add row to -ROOT-). I'd say hold off on any of the trickery
>> described in the first mail till we see snippet from your master log.
>> Lets figure why not assigning .META. before we do any messing.
>>
>> Thanks,
>> St.Ack
>>
>> On Tue, Sep 21, 2010 at 2:52 PM, Matt Corgan <mc...@hotpads.com> wrote:
>> > Thanks for responding Stack. In a 4 slave cluster (rf=3) cluster we
>> > disabled an hdfs/hbase node which also seemed to make a second
>> regionserver
>> > crash. I'm not sure exactly what went wrong.
>> >
>> > There is data in the .META. directory
>> >
>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs
>> -lsr
>> > /hbase/.META.
>> > drwxr-xr-x - root supergroup 0 2010-09-20 16:44
>> > /hbase/.META./1028785192
>> > -rw-r--r-- 3 root supergroup 932 2010-09-20 14:25
>> > /hbase/.META./1028785192/.regioninfo
>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41
>> > /hbase/.META./1028785192/info
>> > -rw-r--r-- 3 root supergroup 5018 2010-09-21 15:41
>> > /hbase/.META./1028785192/info/4694688683869779325
>> > -rw-r--r-- 3 root supergroup 0 2010-09-21 14:48
>> > /hbase/.META./1028785192/info/6832908118284006198
>> > -rw-r--r-- 3 root supergroup 228130 2010-09-21 14:44
>> > /hbase/.META./1028785192/info/7773537252698623585
>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:44
>> > /hbase/.META./compaction.dir
>> >
>> >
>> > but it appears the -ROOT- region is empty. If i scan '-ROOT-' in the
>> hbase
>> > shell i get:
>> >
>> > hbase(main):001:0> scan '-ROOT-'
>> > ROW COLUMN+CELL
>> > 0 row(s) in 0.0840 seconds
>> >
>> >
>> > even though it appears -ROOT- exists in hdfs. Any idea why ROOT would
>> not
>> > get loaded from hdfs after a master restart?
>> >
>> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs
>> -lsr
>> > /hbase/-ROOT-
>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:50
>> > /hbase/-ROOT-/70236052
>> > -rw-r--r-- 3 root supergroup 618 2010-09-21 14:50
>> > /hbase/-ROOT-/70236052/.regioninfo
>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41
>> > /hbase/-ROOT-/70236052/info
>> > -rw-r--r-- 3 root supergroup 489 2010-09-21 15:41
>> > /hbase/-ROOT-/70236052/info/308400263036654323
>> > -rw-r--r-- 3 root supergroup 462 2010-09-21 13:43
>> > /hbase/-ROOT-/70236052/info/8250773031047005400
>> >
>> >
>> > I'll just follow your suggestion of generating a new .META. and copying
>> > over. Thanks,
>> >
>> > Matt
>> >
>> >
>> > On Tue, Sep 21, 2010 at 5:20 PM, Stack <st...@duboce.net> wrote:
>> >
>> >> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <mc...@hotpads.com>
>> wrote:
>> >> > Is there a way to create a new .META. table if I lost it? For
>> example,
>> >> I'm
>> >> > getting this exception everywhere:
>> >> >
>> >> > org.apache.hadoop.hbase.TableNotFoundException: .META.
>> (NativeException)
>> >> >
>> >>
>> >> How'd you do that? Thats pretty impressive.
>> >>
>> >> >
>> >> > The add_table script doesn't appear to handle .META., giving this
>> error:
>> >> >
>> >> > java.lang.IllegalArgumentException: Illegal first character <46> at
>> 0.
>> >> > User-space table names can only start with 'word characters': i.e.
>> >> > [a-zA-Z_0-9]: .META.
>> >> >
>> >> >
>> >> > And the hbase shell will only create user-space table names.
>> Restarting
>> >> the
>> >> > hbase master also does not appear to re-create it.
>> >> >
>> >> > I'm using 0.20.6.
>> >> >
>> >> Whats the master log look like when you start up hbase? Whats it say
>> >> about missing .META.?
>> >>
>> >> If you do a
>> >>
>> >> $ bin/hadoop fs -lsr /hbase/.META.
>> >>
>> >> ... is there anything in your filesystem?
>> >>
>> >> If nothing there, one thing you could do is run an hbase elsewhere.
>> >> Let it create a new "/hbase" dir and then copy the .META. table from
>> >> here up to the filesystem with data (Run add_table.rb after .META. is
>> >> in place to repopulate its rows).
>> >>
>> >> St.Ack
>> >>
>> >
>>
>
>
Re: lost .META. table
Posted by Matt Corgan <mc...@hotpads.com>.
I think this is the relevant section:
2010-09-21 15:42:11,986 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: HadoopNode41,60020,1285098132069
2010-09-21 15:42:11,989 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: HadoopNode38,60020,1285098131987
2010-09-21 15:42:11,989 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
/hbase/rs/1285098132069 with data 10.214.7.223:60020
2010-09-21 15:42:11,996 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
/hbase/rs/1285098131987 with data 10.209.206.15:60020
2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Assigning for address: 10.209.206.15:60020, startcode: 1285098131987, load:
(requests=0, regions=0, usedHeap=32, maxHeap=5195): total nregions to
assign=1, regions to give other servers than this=0, isMetaAssign=true
2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager:
Assigning address: 10.209.206.15:60020, startcode: 1285098131987, load:
(requests=0, regions=0, usedHeap=32, maxHeap=5195) 1 regions
2010-09-21 15:42:13,001 INFO org.apache.hadoop.hbase.master.RegionManager:
Assigning region -ROOT-,,0 to HadoopNode38,60020,1285098131987
2010-09-21 15:42:13,085 INFO org.apache.hadoop.hbase.master.ServerManager:
Processing MSG_REPORT_OPEN: -ROOT-,,0 from HadoopNode38,60020,1285098131987;
1 of 1
2010-09-21 15:42:13,130 WARN
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create out of
safe mode in ZooKeeper:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists for /hbase/safe-mode
2010-09-21 15:42:13,130 INFO
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Node exists; just move
on
2010-09-21 15:42:13,130 INFO org.apache.hadoop.hbase.master.RegionManager:
exiting safe mode
2010-09-21 15:42:13,132 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: SetData of ZNode
/hbase/root-region-server with 10.209.206.15:60020
2010-09-21 15:42:13,133 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 10.209.206.15:60020,
regionname: -ROOT-,,0, startKey: <>}
2010-09-21 15:42:13,223 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 0 row(s) of meta region {server:
10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
2010-09-21 15:42:13,466 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: HadoopNode40,60020,1285098133450
2010-09-21 15:42:13,468 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
/hbase/rs/1285098133450 with data 10.192.66.240:60020
2010-09-21 15:42:13,828 INFO org.apache.hadoop.hbase.master.ServerManager:
Received start message from: HadoopNode39,60020,1285098133654
2010-09-21 15:42:13,830 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
/hbase/rs/1285098133654 with data 10.206.26.95:60020
2010-09-21 15:43:10,256 INFO org.apache.hadoop.hbase.master.ServerManager: 4
region servers, 0 dead, average load 0.25
2010-09-21 15:43:10,341 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 10.209.206.15:60020,
regionname: -ROOT-,,0, startKey: <>}
2010-09-21 15:43:10,345 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 0 row(s) of meta region {server:
10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete
2010-09-21 15:43:10,354 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned
2010-09-21 15:43:56,351 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=10.249.107.223:2181 sessionTimeout=60000
watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@72898540
2010-09-21 15:43:56,352 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server /10.249.107.223:2181
2010-09-21 15:43:56,362 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
10.249.107.223:56828remote=/10.249.107.223:2181]
2010-09-21 15:43:56,380 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
On Tue, Sep 21, 2010 at 6:05 PM, Stack <st...@duboce.net> wrote:
> Ok. So .META. is there. What is in your master log on startup? Its
> supposed to assign out -ROOT- and .META. (when .META. is assigned
> it'll add row to -ROOT-). I'd say hold off on any of the trickery
> described in the first mail till we see snippet from your master log.
> Lets figure why not assigning .META. before we do any messing.
>
> Thanks,
> St.Ack
>
> On Tue, Sep 21, 2010 at 2:52 PM, Matt Corgan <mc...@hotpads.com> wrote:
> > Thanks for responding Stack. In a 4 slave cluster (rf=3) cluster we
> > disabled an hdfs/hbase node which also seemed to make a second
> regionserver
> > crash. I'm not sure exactly what went wrong.
> >
> > There is data in the .META. directory
> >
> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr
> > /hbase/.META.
> > drwxr-xr-x - root supergroup 0 2010-09-20 16:44
> > /hbase/.META./1028785192
> > -rw-r--r-- 3 root supergroup 932 2010-09-20 14:25
> > /hbase/.META./1028785192/.regioninfo
> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41
> > /hbase/.META./1028785192/info
> > -rw-r--r-- 3 root supergroup 5018 2010-09-21 15:41
> > /hbase/.META./1028785192/info/4694688683869779325
> > -rw-r--r-- 3 root supergroup 0 2010-09-21 14:48
> > /hbase/.META./1028785192/info/6832908118284006198
> > -rw-r--r-- 3 root supergroup 228130 2010-09-21 14:44
> > /hbase/.META./1028785192/info/7773537252698623585
> > drwxr-xr-x - root supergroup 0 2010-09-21 14:44
> > /hbase/.META./compaction.dir
> >
> >
> > but it appears the -ROOT- region is empty. If i scan '-ROOT-' in the
> hbase
> > shell i get:
> >
> > hbase(main):001:0> scan '-ROOT-'
> > ROW COLUMN+CELL
> > 0 row(s) in 0.0840 seconds
> >
> >
> > even though it appears -ROOT- exists in hdfs. Any idea why ROOT would
> not
> > get loaded from hdfs after a master restart?
> >
> > [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr
> > /hbase/-ROOT-
> > drwxr-xr-x - root supergroup 0 2010-09-21 14:50
> > /hbase/-ROOT-/70236052
> > -rw-r--r-- 3 root supergroup 618 2010-09-21 14:50
> > /hbase/-ROOT-/70236052/.regioninfo
> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41
> > /hbase/-ROOT-/70236052/info
> > -rw-r--r-- 3 root supergroup 489 2010-09-21 15:41
> > /hbase/-ROOT-/70236052/info/308400263036654323
> > -rw-r--r-- 3 root supergroup 462 2010-09-21 13:43
> > /hbase/-ROOT-/70236052/info/8250773031047005400
> >
> >
> > I'll just follow your suggestion of generating a new .META. and copying
> > over. Thanks,
> >
> > Matt
> >
> >
> > On Tue, Sep 21, 2010 at 5:20 PM, Stack <st...@duboce.net> wrote:
> >
> >> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <mc...@hotpads.com>
> wrote:
> >> > Is there a way to create a new .META. table if I lost it? For
> example,
> >> I'm
> >> > getting this exception everywhere:
> >> >
> >> > org.apache.hadoop.hbase.TableNotFoundException: .META.
> (NativeException)
> >> >
> >>
> >> How'd you do that? Thats pretty impressive.
> >>
> >> >
> >> > The add_table script doesn't appear to handle .META., giving this
> error:
> >> >
> >> > java.lang.IllegalArgumentException: Illegal first character <46> at 0.
> >> > User-space table names can only start with 'word characters': i.e.
> >> > [a-zA-Z_0-9]: .META.
> >> >
> >> >
> >> > And the hbase shell will only create user-space table names.
> Restarting
> >> the
> >> > hbase master also does not appear to re-create it.
> >> >
> >> > I'm using 0.20.6.
> >> >
> >> Whats the master log look like when you start up hbase? Whats it say
> >> about missing .META.?
> >>
> >> If you do a
> >>
> >> $ bin/hadoop fs -lsr /hbase/.META.
> >>
> >> ... is there anything in your filesystem?
> >>
> >> If nothing there, one thing you could do is run an hbase elsewhere.
> >> Let it create a new "/hbase" dir and then copy the .META. table from
> >> here up to the filesystem with data (Run add_table.rb after .META. is
> >> in place to repopulate its rows).
> >>
> >> St.Ack
> >>
> >
>
Re: lost .META. table
Posted by Stack <st...@duboce.net>.
Ok. So .META. is there. What is in your master log on startup? Its
supposed to assign out -ROOT- and .META. (when .META. is assigned
it'll add row to -ROOT-). I'd say hold off on any of the trickery
described in the first mail till we see snippet from your master log.
Lets figure why not assigning .META. before we do any messing.
Thanks,
St.Ack
On Tue, Sep 21, 2010 at 2:52 PM, Matt Corgan <mc...@hotpads.com> wrote:
> Thanks for responding Stack. In a 4 slave cluster (rf=3) cluster we
> disabled an hdfs/hbase node which also seemed to make a second regionserver
> crash. I'm not sure exactly what went wrong.
>
> There is data in the .META. directory
>
> [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr
> /hbase/.META.
> drwxr-xr-x - root supergroup 0 2010-09-20 16:44
> /hbase/.META./1028785192
> -rw-r--r-- 3 root supergroup 932 2010-09-20 14:25
> /hbase/.META./1028785192/.regioninfo
> drwxr-xr-x - root supergroup 0 2010-09-21 15:41
> /hbase/.META./1028785192/info
> -rw-r--r-- 3 root supergroup 5018 2010-09-21 15:41
> /hbase/.META./1028785192/info/4694688683869779325
> -rw-r--r-- 3 root supergroup 0 2010-09-21 14:48
> /hbase/.META./1028785192/info/6832908118284006198
> -rw-r--r-- 3 root supergroup 228130 2010-09-21 14:44
> /hbase/.META./1028785192/info/7773537252698623585
> drwxr-xr-x - root supergroup 0 2010-09-21 14:44
> /hbase/.META./compaction.dir
>
>
> but it appears the -ROOT- region is empty. If i scan '-ROOT-' in the hbase
> shell i get:
>
> hbase(main):001:0> scan '-ROOT-'
> ROW COLUMN+CELL
> 0 row(s) in 0.0840 seconds
>
>
> even though it appears -ROOT- exists in hdfs. Any idea why ROOT would not
> get loaded from hdfs after a master restart?
>
> [root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr
> /hbase/-ROOT-
> drwxr-xr-x - root supergroup 0 2010-09-21 14:50
> /hbase/-ROOT-/70236052
> -rw-r--r-- 3 root supergroup 618 2010-09-21 14:50
> /hbase/-ROOT-/70236052/.regioninfo
> drwxr-xr-x - root supergroup 0 2010-09-21 15:41
> /hbase/-ROOT-/70236052/info
> -rw-r--r-- 3 root supergroup 489 2010-09-21 15:41
> /hbase/-ROOT-/70236052/info/308400263036654323
> -rw-r--r-- 3 root supergroup 462 2010-09-21 13:43
> /hbase/-ROOT-/70236052/info/8250773031047005400
>
>
> I'll just follow your suggestion of generating a new .META. and copying
> over. Thanks,
>
> Matt
>
>
> On Tue, Sep 21, 2010 at 5:20 PM, Stack <st...@duboce.net> wrote:
>
>> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <mc...@hotpads.com> wrote:
>> > Is there a way to create a new .META. table if I lost it? For example,
>> I'm
>> > getting this exception everywhere:
>> >
>> > org.apache.hadoop.hbase.TableNotFoundException: .META. (NativeException)
>> >
>>
>> How'd you do that? Thats pretty impressive.
>>
>> >
>> > The add_table script doesn't appear to handle .META., giving this error:
>> >
>> > java.lang.IllegalArgumentException: Illegal first character <46> at 0.
>> > User-space table names can only start with 'word characters': i.e.
>> > [a-zA-Z_0-9]: .META.
>> >
>> >
>> > And the hbase shell will only create user-space table names. Restarting
>> the
>> > hbase master also does not appear to re-create it.
>> >
>> > I'm using 0.20.6.
>> >
>> Whats the master log look like when you start up hbase? Whats it say
>> about missing .META.?
>>
>> If you do a
>>
>> $ bin/hadoop fs -lsr /hbase/.META.
>>
>> ... is there anything in your filesystem?
>>
>> If nothing there, one thing you could do is run an hbase elsewhere.
>> Let it create a new "/hbase" dir and then copy the .META. table from
>> here up to the filesystem with data (Run add_table.rb after .META. is
>> in place to repopulate its rows).
>>
>> St.Ack
>>
>
Re: lost .META. table
Posted by Matt Corgan <mc...@hotpads.com>.
Thanks for responding Stack. In a 4 slave cluster (rf=3) cluster we
disabled an hdfs/hbase node which also seemed to make a second regionserver
crash. I'm not sure exactly what went wrong.
There is data in the .META. directory
[root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr
/hbase/.META.
drwxr-xr-x - root supergroup 0 2010-09-20 16:44
/hbase/.META./1028785192
-rw-r--r-- 3 root supergroup 932 2010-09-20 14:25
/hbase/.META./1028785192/.regioninfo
drwxr-xr-x - root supergroup 0 2010-09-21 15:41
/hbase/.META./1028785192/info
-rw-r--r-- 3 root supergroup 5018 2010-09-21 15:41
/hbase/.META./1028785192/info/4694688683869779325
-rw-r--r-- 3 root supergroup 0 2010-09-21 14:48
/hbase/.META./1028785192/info/6832908118284006198
-rw-r--r-- 3 root supergroup 228130 2010-09-21 14:44
/hbase/.META./1028785192/info/7773537252698623585
drwxr-xr-x - root supergroup 0 2010-09-21 14:44
/hbase/.META./compaction.dir
but it appears the -ROOT- region is empty. If i scan '-ROOT-' in the hbase
shell i get:
hbase(main):001:0> scan '-ROOT-'
ROW COLUMN+CELL
0 row(s) in 0.0840 seconds
even though it appears -ROOT- exists in hdfs. Any idea why ROOT would not
get loaded from hdfs after a master restart?
[root@Hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr
/hbase/-ROOT-
drwxr-xr-x - root supergroup 0 2010-09-21 14:50
/hbase/-ROOT-/70236052
-rw-r--r-- 3 root supergroup 618 2010-09-21 14:50
/hbase/-ROOT-/70236052/.regioninfo
drwxr-xr-x - root supergroup 0 2010-09-21 15:41
/hbase/-ROOT-/70236052/info
-rw-r--r-- 3 root supergroup 489 2010-09-21 15:41
/hbase/-ROOT-/70236052/info/308400263036654323
-rw-r--r-- 3 root supergroup 462 2010-09-21 13:43
/hbase/-ROOT-/70236052/info/8250773031047005400
I'll just follow your suggestion of generating a new .META. and copying
over. Thanks,
Matt
On Tue, Sep 21, 2010 at 5:20 PM, Stack <st...@duboce.net> wrote:
> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <mc...@hotpads.com> wrote:
> > Is there a way to create a new .META. table if I lost it? For example,
> I'm
> > getting this exception everywhere:
> >
> > org.apache.hadoop.hbase.TableNotFoundException: .META. (NativeException)
> >
>
> How'd you do that? Thats pretty impressive.
>
> >
> > The add_table script doesn't appear to handle .META., giving this error:
> >
> > java.lang.IllegalArgumentException: Illegal first character <46> at 0.
> > User-space table names can only start with 'word characters': i.e.
> > [a-zA-Z_0-9]: .META.
> >
> >
> > And the hbase shell will only create user-space table names. Restarting
> the
> > hbase master also does not appear to re-create it.
> >
> > I'm using 0.20.6.
> >
> Whats the master log look like when you start up hbase? Whats it say
> about missing .META.?
>
> If you do a
>
> $ bin/hadoop fs -lsr /hbase/.META.
>
> ... is there anything in your filesystem?
>
> If nothing there, one thing you could do is run an hbase elsewhere.
> Let it create a new "/hbase" dir and then copy the .META. table from
> here up to the filesystem with data (Run add_table.rb after .META. is
> in place to repopulate its rows).
>
> St.Ack
>
Re: lost .META. table
Posted by Stack <st...@duboce.net>.
On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <mc...@hotpads.com> wrote:
> Is there a way to create a new .META. table if I lost it? For example, I'm
> getting this exception everywhere:
>
> org.apache.hadoop.hbase.TableNotFoundException: .META. (NativeException)
>
How'd you do that? Thats pretty impressive.
>
> The add_table script doesn't appear to handle .META., giving this error:
>
> java.lang.IllegalArgumentException: Illegal first character <46> at 0.
> User-space table names can only start with 'word characters': i.e.
> [a-zA-Z_0-9]: .META.
>
>
> And the hbase shell will only create user-space table names. Restarting the
> hbase master also does not appear to re-create it.
>
> I'm using 0.20.6.
>
Whats the master log look like when you start up hbase? Whats it say
about missing .META.?
If you do a
$ bin/hadoop fs -lsr /hbase/.META.
... is there anything in your filesystem?
If nothing there, one thing you could do is run an hbase elsewhere.
Let it create a new "/hbase" dir and then copy the .META. table from
here up to the filesystem with data (Run add_table.rb after .META. is
in place to repopulate its rows).
St.Ack