You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Shane O'Donnell <sh...@knownormal.com> on 2016/01/04 04:21:21 UTC
Hbase master not starting
Hi -
My cluster has been running perfectly until the other day when I found it
down.
The error seems to be related not being able to get the ClusterID from
zookeeper, but I'm stumped as to what to do about it. This seems to be the
relevant part of the master's log:
http://pastebin.com/C3iaxM3p
starting at line 235 (also highlighted).
Any help is appreciated!
Thanks -
Shane O.
========================
Shane O'Donnell
<http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!
Re: Hbase master not starting
Posted by Shane O'Donnell <sh...@knownormal.com>.
And the error was (no surprise) not finding 'namespace':
2016-01-04 16:28:45,323 INFO
[PriorityRpcServer.handler=7,queue=1,port=60020]
regionserver.RSRpcServices: Open
hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
2016-01-04 16:28:45,326 ERROR [RS_OPEN_REGION-ip-10-0-1-151:60020-1]
handler.OpenRegionHandler: Failed open of
region=hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.,
starting to roll back the global memstore size.
java.lang.IllegalStateException: Could not instantiate a region instance.
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5666)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5973)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5945)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5901)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5852)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc
========================
Shane O'Donnell
<http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9de88312-9164-4f22-c6ca-b4c867938fc3>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!
On Mon, Jan 4, 2016 at 12:26 PM, Shane O'Donnell <sh...@knownormal.com>
wrote:
> I looked for region 82432aca9ede964943b40753cb64e808 on each of my region
> servers and none of them had it. They all had identical failures in the
> log of attempting to open it, but none had it (or at least were successful
> in opening it).
>
> My solution to "finding it" was grepping for
> "82432aca9ede964943b40753cb64e808" in the logs. If there's a better or
> more reliable way of searching for this, let me know.
>
> Thanks -
>
> Shane O.
>
> ========================
> Shane O'Donnell
>
> <http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=27fa6676-ade7-41f1-f304-76654be07bdd>
> 4819 Emperor Blvd., Ste 400
> Durham, North Carolina 27703
> tel: +1.424.262.KNOW x703
> skype: shaneodonnell
> email: shaneo@knownormal.com
> ========================
> :wq!
>
> On Mon, Jan 4, 2016 at 12:03 PM, Shane O'Donnell <sh...@knownormal.com>
> wrote:
>
>> It's not there. The directory listing was for the right directory, but
>> the namespace directory is not there.
>>
>> Here it is one from directory level up:
>>
>> [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls
>> -R /hbase/data
>>
>> drwxr-xr-x - hdfs hadoop 0 2016-01-04 14:47 /hbase/data/hbase
>>
>> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
>> /hbase/data/hbase/meta
>>
>> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
>> /hbase/data/hbase/meta/.tabledesc
>>
>> -rw-r--r-- 2 hbase hadoop 372 2016-01-04 14:48
>> /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
>>
>> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
>> /hbase/data/hbase/meta/.tmp
>>
>> drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
>> /hbase/data/hbase/meta/1588230740
>>
>> -rw-r--r-- 3 hdfs hadoop 32 2016-01-04 16:28
>> /hbase/data/hbase/meta/1588230740/.regioninfo
>>
>> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:47
>> /hbase/data/hbase/meta/1588230740/info
>>
>> drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
>> /hbase/data/hbase/meta/1588230740/recovered.edits
>>
>> -rw-r--r-- 3 hdfs hadoop 0 2016-01-04 16:28
>> /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
>>
>> ========================
>> Shane O'Donnell
>>
>> <http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6d1d2ed3-3d92-4821-eb9f-f589f936d34c>
>> 4819 Emperor Blvd., Ste 400
>> Durham, North Carolina 27703
>> tel: +1.424.262.KNOW x703
>> skype: shaneodonnell
>> email: shaneo@knownormal.com
>> ========================
>> :wq!
>>
>> On Mon, Jan 4, 2016 at 11:59 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> In your listing, meta table directory structure was shown.
>>>
>>> Please look under /hbase/data/hbase for namespace table.
>>>
>>> Cheers
>>>
>>> On Mon, Jan 4, 2016 at 8:42 AM, Shane O'Donnell <sh...@knownormal.com>
>>> wrote:
>>>
>>> > I attempted to restore a backed-up "meta" directory from a copy made by
>>> > 'hbase hbck', so my directory structure may be messed up:
>>> >
>>> > [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs
>>> -ls -R
>>> > /hbase/data/hbase
>>> >
>>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
>>> > /hbase/data/hbase/meta
>>> >
>>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
>>> > /hbase/data/hbase/meta/.tabledesc
>>> >
>>> > -rw-r--r-- 2 hbase hadoop 372 2016-01-04 14:48
>>> > /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
>>> >
>>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
>>> > /hbase/data/hbase/meta/.tmp
>>> >
>>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
>>> > /hbase/data/hbase/meta/1588230740
>>> >
>>> > -rw-r--r-- 3 hdfs hadoop 32 2016-01-04 16:28
>>> > /hbase/data/hbase/meta/1588230740/.regioninfo
>>> >
>>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:47
>>> > /hbase/data/hbase/meta/1588230740/info
>>> >
>>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
>>> > /hbase/data/hbase/meta/1588230740/recovered.edits
>>> >
>>> > -rw-r--r-- 3 hdfs hadoop 0 2016-01-04 16:28
>>> > /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
>>> >
>>> > And I'm not seeing any "namespace" directory.
>>> >
>>> > Will check out the server now...
>>> >
>>> > Shane O.
>>> >
>>> > ========================
>>> > Shane O'Donnell
>>> > <
>>> >
>>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6a0326e0-41e4-46fd-bcce-114f44302f69
>>> > >
>>> > 4819 Emperor Blvd., Ste 400
>>> > Durham, North Carolina 27703
>>> > tel: +1.424.262.KNOW x703
>>> > skype: shaneodonnell
>>> > email: shaneo@knownormal.com
>>> > ========================
>>> > :wq!
>>> >
>>> > On Mon, Jan 4, 2016 at 9:44 AM, Ted Yu <yu...@gmail.com> wrote:
>>> >
>>> > > Can you log onto the server hosting region
>>> > 82432aca9ede964943b40753cb64e808
>>> > > and see what happened ?
>>> > >
>>> > > See if the namespace table can be found under rootdir.
>>> > > e.g. assuming /apps/hbase/data is the rootdir, you should see
>>> something
>>> > > similar to the following:
>>> > >
>>> > > hdfs dfs -ls /apps/hbase/data/data/hbase/namespace
>>> > > Found 3 items
>>> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
>>> > > /apps/hbase/data/data/hbase/namespace/.tabledesc
>>> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
>>> > > /apps/hbase/data/data/hbase/namespace/.tmp
>>> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:51
>>> > >
>>> /apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8
>>> > >
>>> > > On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell <
>>> shaneo@knownormal.com>
>>> > > wrote:
>>> > >
>>> > > > Some progress...
>>> > > >
>>> > > > /hbase did NOT have either the hbase.id or hbase.version files so
>>> I
>>> > > > temporarily changed hbase.rootdir and started the master so the
>>> files
>>> > > would
>>> > > > be recreated elsewhere and copied them in.
>>> > > >
>>> > > > Now it starts fine, but my hbase tables are gone. Specifically,
>>> I'm
>>> > > > getting this error:
>>> > > >
>>> > > > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
>>> > > > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
>>> > > > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
>>> > > >
>>> > > > Is my existing data toast, or is there a crafty way out of this?
>>> > > >
>>> > > > Thanks -
>>> > > >
>>> > > > Shane O.
>>> > > >
>>> > > > ========================
>>> > > > Shane O'Donnell
>>> > > > <
>>> > > >
>>> > >
>>> >
>>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
>>> > > > >
>>> > > > 4819 Emperor Blvd., Ste 400
>>> > > > Durham, North Carolina 27703
>>> > > > tel: +1.424.262.KNOW x703
>>> > > > skype: shaneodonnell
>>> > > > email: shaneo@knownormal.com
>>> > > > ========================
>>> > > > :wq!
>>> > > >
>>> > > > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <
>>> > shaneo@knownormal.com>
>>> > > > wrote:
>>> > > >
>>> > > > > Hi -
>>> > > > >
>>> > > > > My cluster has been running perfectly until the other day when I
>>> > found
>>> > > it
>>> > > > > down.
>>> > > > >
>>> > > > > The error seems to be related not being able to get the ClusterID
>>> > from
>>> > > > > zookeeper, but I'm stumped as to what to do about it. This
>>> seems to
>>> > be
>>> > > > the
>>> > > > > relevant part of the master's log:
>>> > > > >
>>> > > > > http://pastebin.com/C3iaxM3p
>>> > > > >
>>> > > > > starting at line 235 (also highlighted).
>>> > > > >
>>> > > > > Any help is appreciated!
>>> > > > >
>>> > > > > Thanks -
>>> > > > >
>>> > > > > Shane O.
>>> > > > > ========================
>>> > > > > Shane O'Donnell
>>> > > > >
>>> > > > > <
>>> > > >
>>> > >
>>> >
>>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
>>> > > > >
>>> > > > > 4819 Emperor Blvd., Ste 400
>>> > > > > Durham, North Carolina 27703
>>> > > > > tel: +1.424.262.KNOW x703
>>> > > > > skype: shaneodonnell
>>> > > > > email: shaneo@knownormal.com
>>> > > > > ========================
>>> > > > > :wq!
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: Hbase master not starting
Posted by Shane O'Donnell <sh...@knownormal.com>.
I looked for region 82432aca9ede964943b40753cb64e808 on each of my region
servers and none of them had it. They all had identical failures in the
log of attempting to open it, but none had it (or at least were successful
in opening it).
My solution to "finding it" was grepping for
"82432aca9ede964943b40753cb64e808" in the logs. If there's a better or
more reliable way of searching for this, let me know.
Thanks -
Shane O.
========================
Shane O'Donnell
<http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=27fa6676-ade7-41f1-f304-76654be07bdd>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!
On Mon, Jan 4, 2016 at 12:03 PM, Shane O'Donnell <sh...@knownormal.com>
wrote:
> It's not there. The directory listing was for the right directory, but
> the namespace directory is not there.
>
> Here it is one from directory level up:
>
> [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls -R
> /hbase/data
>
> drwxr-xr-x - hdfs hadoop 0 2016-01-04 14:47 /hbase/data/hbase
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
> /hbase/data/hbase/meta
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
> /hbase/data/hbase/meta/.tabledesc
>
> -rw-r--r-- 2 hbase hadoop 372 2016-01-04 14:48
> /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
> /hbase/data/hbase/meta/.tmp
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740
>
> -rw-r--r-- 3 hdfs hadoop 32 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740/.regioninfo
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:47
> /hbase/data/hbase/meta/1588230740/info
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740/recovered.edits
>
> -rw-r--r-- 3 hdfs hadoop 0 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
>
> ========================
> Shane O'Donnell
>
> <http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6d1d2ed3-3d92-4821-eb9f-f589f936d34c>
> 4819 Emperor Blvd., Ste 400
> Durham, North Carolina 27703
> tel: +1.424.262.KNOW x703
> skype: shaneodonnell
> email: shaneo@knownormal.com
> ========================
> :wq!
>
> On Mon, Jan 4, 2016 at 11:59 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> In your listing, meta table directory structure was shown.
>>
>> Please look under /hbase/data/hbase for namespace table.
>>
>> Cheers
>>
>> On Mon, Jan 4, 2016 at 8:42 AM, Shane O'Donnell <sh...@knownormal.com>
>> wrote:
>>
>> > I attempted to restore a backed-up "meta" directory from a copy made by
>> > 'hbase hbck', so my directory structure may be messed up:
>> >
>> > [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls
>> -R
>> > /hbase/data/hbase
>> >
>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
>> > /hbase/data/hbase/meta
>> >
>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
>> > /hbase/data/hbase/meta/.tabledesc
>> >
>> > -rw-r--r-- 2 hbase hadoop 372 2016-01-04 14:48
>> > /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
>> >
>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
>> > /hbase/data/hbase/meta/.tmp
>> >
>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
>> > /hbase/data/hbase/meta/1588230740
>> >
>> > -rw-r--r-- 3 hdfs hadoop 32 2016-01-04 16:28
>> > /hbase/data/hbase/meta/1588230740/.regioninfo
>> >
>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:47
>> > /hbase/data/hbase/meta/1588230740/info
>> >
>> > drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
>> > /hbase/data/hbase/meta/1588230740/recovered.edits
>> >
>> > -rw-r--r-- 3 hdfs hadoop 0 2016-01-04 16:28
>> > /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
>> >
>> > And I'm not seeing any "namespace" directory.
>> >
>> > Will check out the server now...
>> >
>> > Shane O.
>> >
>> > ========================
>> > Shane O'Donnell
>> > <
>> >
>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6a0326e0-41e4-46fd-bcce-114f44302f69
>> > >
>> > 4819 Emperor Blvd., Ste 400
>> > Durham, North Carolina 27703
>> > tel: +1.424.262.KNOW x703
>> > skype: shaneodonnell
>> > email: shaneo@knownormal.com
>> > ========================
>> > :wq!
>> >
>> > On Mon, Jan 4, 2016 at 9:44 AM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> > > Can you log onto the server hosting region
>> > 82432aca9ede964943b40753cb64e808
>> > > and see what happened ?
>> > >
>> > > See if the namespace table can be found under rootdir.
>> > > e.g. assuming /apps/hbase/data is the rootdir, you should see
>> something
>> > > similar to the following:
>> > >
>> > > hdfs dfs -ls /apps/hbase/data/data/hbase/namespace
>> > > Found 3 items
>> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
>> > > /apps/hbase/data/data/hbase/namespace/.tabledesc
>> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
>> > > /apps/hbase/data/data/hbase/namespace/.tmp
>> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:51
>> > > /apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8
>> > >
>> > > On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell <
>> shaneo@knownormal.com>
>> > > wrote:
>> > >
>> > > > Some progress...
>> > > >
>> > > > /hbase did NOT have either the hbase.id or hbase.version files so I
>> > > > temporarily changed hbase.rootdir and started the master so the
>> files
>> > > would
>> > > > be recreated elsewhere and copied them in.
>> > > >
>> > > > Now it starts fine, but my hbase tables are gone. Specifically, I'm
>> > > > getting this error:
>> > > >
>> > > > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
>> > > > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
>> > > > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
>> > > >
>> > > > Is my existing data toast, or is there a crafty way out of this?
>> > > >
>> > > > Thanks -
>> > > >
>> > > > Shane O.
>> > > >
>> > > > ========================
>> > > > Shane O'Donnell
>> > > > <
>> > > >
>> > >
>> >
>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
>> > > > >
>> > > > 4819 Emperor Blvd., Ste 400
>> > > > Durham, North Carolina 27703
>> > > > tel: +1.424.262.KNOW x703
>> > > > skype: shaneodonnell
>> > > > email: shaneo@knownormal.com
>> > > > ========================
>> > > > :wq!
>> > > >
>> > > > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <
>> > shaneo@knownormal.com>
>> > > > wrote:
>> > > >
>> > > > > Hi -
>> > > > >
>> > > > > My cluster has been running perfectly until the other day when I
>> > found
>> > > it
>> > > > > down.
>> > > > >
>> > > > > The error seems to be related not being able to get the ClusterID
>> > from
>> > > > > zookeeper, but I'm stumped as to what to do about it. This seems
>> to
>> > be
>> > > > the
>> > > > > relevant part of the master's log:
>> > > > >
>> > > > > http://pastebin.com/C3iaxM3p
>> > > > >
>> > > > > starting at line 235 (also highlighted).
>> > > > >
>> > > > > Any help is appreciated!
>> > > > >
>> > > > > Thanks -
>> > > > >
>> > > > > Shane O.
>> > > > > ========================
>> > > > > Shane O'Donnell
>> > > > >
>> > > > > <
>> > > >
>> > >
>> >
>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
>> > > > >
>> > > > > 4819 Emperor Blvd., Ste 400
>> > > > > Durham, North Carolina 27703
>> > > > > tel: +1.424.262.KNOW x703
>> > > > > skype: shaneodonnell
>> > > > > email: shaneo@knownormal.com
>> > > > > ========================
>> > > > > :wq!
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Hbase master not starting
Posted by Shane O'Donnell <sh...@knownormal.com>.
It's not there. The directory listing was for the right directory, but the
namespace directory is not there.
Here it is one from directory level up:
[user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls -R
/hbase/data
drwxr-xr-x - hdfs hadoop 0 2016-01-04 14:47 /hbase/data/hbase
drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
/hbase/data/hbase/meta
drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
/hbase/data/hbase/meta/.tabledesc
-rw-r--r-- 2 hbase hadoop 372 2016-01-04 14:48
/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
/hbase/data/hbase/meta/.tmp
drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740
-rw-r--r-- 3 hdfs hadoop 32 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/.regioninfo
drwxr-xr-x - hbase hadoop 0 2016-01-04 14:47
/hbase/data/hbase/meta/1588230740/info
drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/recovered.edits
-rw-r--r-- 3 hdfs hadoop 0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
========================
Shane O'Donnell
<http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6d1d2ed3-3d92-4821-eb9f-f589f936d34c>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!
On Mon, Jan 4, 2016 at 11:59 AM, Ted Yu <yu...@gmail.com> wrote:
> In your listing, meta table directory structure was shown.
>
> Please look under /hbase/data/hbase for namespace table.
>
> Cheers
>
> On Mon, Jan 4, 2016 at 8:42 AM, Shane O'Donnell <sh...@knownormal.com>
> wrote:
>
> > I attempted to restore a backed-up "meta" directory from a copy made by
> > 'hbase hbck', so my directory structure may be messed up:
> >
> > [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls
> -R
> > /hbase/data/hbase
> >
> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
> > /hbase/data/hbase/meta
> >
> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
> > /hbase/data/hbase/meta/.tabledesc
> >
> > -rw-r--r-- 2 hbase hadoop 372 2016-01-04 14:48
> > /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
> >
> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
> > /hbase/data/hbase/meta/.tmp
> >
> > drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
> > /hbase/data/hbase/meta/1588230740
> >
> > -rw-r--r-- 3 hdfs hadoop 32 2016-01-04 16:28
> > /hbase/data/hbase/meta/1588230740/.regioninfo
> >
> > drwxr-xr-x - hbase hadoop 0 2016-01-04 14:47
> > /hbase/data/hbase/meta/1588230740/info
> >
> > drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
> > /hbase/data/hbase/meta/1588230740/recovered.edits
> >
> > -rw-r--r-- 3 hdfs hadoop 0 2016-01-04 16:28
> > /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
> >
> > And I'm not seeing any "namespace" directory.
> >
> > Will check out the server now...
> >
> > Shane O.
> >
> > ========================
> > Shane O'Donnell
> > <
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6a0326e0-41e4-46fd-bcce-114f44302f69
> > >
> > 4819 Emperor Blvd., Ste 400
> > Durham, North Carolina 27703
> > tel: +1.424.262.KNOW x703
> > skype: shaneodonnell
> > email: shaneo@knownormal.com
> > ========================
> > :wq!
> >
> > On Mon, Jan 4, 2016 at 9:44 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Can you log onto the server hosting region
> > 82432aca9ede964943b40753cb64e808
> > > and see what happened ?
> > >
> > > See if the namespace table can be found under rootdir.
> > > e.g. assuming /apps/hbase/data is the rootdir, you should see something
> > > similar to the following:
> > >
> > > hdfs dfs -ls /apps/hbase/data/data/hbase/namespace
> > > Found 3 items
> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
> > > /apps/hbase/data/data/hbase/namespace/.tabledesc
> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
> > > /apps/hbase/data/data/hbase/namespace/.tmp
> > > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:51
> > > /apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8
> > >
> > > On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell <shaneo@knownormal.com
> >
> > > wrote:
> > >
> > > > Some progress...
> > > >
> > > > /hbase did NOT have either the hbase.id or hbase.version files so I
> > > > temporarily changed hbase.rootdir and started the master so the files
> > > would
> > > > be recreated elsewhere and copied them in.
> > > >
> > > > Now it starts fine, but my hbase tables are gone. Specifically, I'm
> > > > getting this error:
> > > >
> > > > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
> > > > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
> > > > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
> > > >
> > > > Is my existing data toast, or is there a crafty way out of this?
> > > >
> > > > Thanks -
> > > >
> > > > Shane O.
> > > >
> > > > ========================
> > > > Shane O'Donnell
> > > > <
> > > >
> > >
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
> > > > >
> > > > 4819 Emperor Blvd., Ste 400
> > > > Durham, North Carolina 27703
> > > > tel: +1.424.262.KNOW x703
> > > > skype: shaneodonnell
> > > > email: shaneo@knownormal.com
> > > > ========================
> > > > :wq!
> > > >
> > > > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <
> > shaneo@knownormal.com>
> > > > wrote:
> > > >
> > > > > Hi -
> > > > >
> > > > > My cluster has been running perfectly until the other day when I
> > found
> > > it
> > > > > down.
> > > > >
> > > > > The error seems to be related not being able to get the ClusterID
> > from
> > > > > zookeeper, but I'm stumped as to what to do about it. This seems
> to
> > be
> > > > the
> > > > > relevant part of the master's log:
> > > > >
> > > > > http://pastebin.com/C3iaxM3p
> > > > >
> > > > > starting at line 235 (also highlighted).
> > > > >
> > > > > Any help is appreciated!
> > > > >
> > > > > Thanks -
> > > > >
> > > > > Shane O.
> > > > > ========================
> > > > > Shane O'Donnell
> > > > >
> > > > > <
> > > >
> > >
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> > > > >
> > > > > 4819 Emperor Blvd., Ste 400
> > > > > Durham, North Carolina 27703
> > > > > tel: +1.424.262.KNOW x703
> > > > > skype: shaneodonnell
> > > > > email: shaneo@knownormal.com
> > > > > ========================
> > > > > :wq!
> > > > >
> > > >
> > >
> >
>
Re: Hbase master not starting
Posted by Ted Yu <yu...@gmail.com>.
In your listing, meta table directory structure was shown.
Please look under /hbase/data/hbase for namespace table.
Cheers
On Mon, Jan 4, 2016 at 8:42 AM, Shane O'Donnell <sh...@knownormal.com>
wrote:
> I attempted to restore a backed-up "meta" directory from a copy made by
> 'hbase hbck', so my directory structure may be messed up:
>
> [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls -R
> /hbase/data/hbase
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
> /hbase/data/hbase/meta
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
> /hbase/data/hbase/meta/.tabledesc
>
> -rw-r--r-- 2 hbase hadoop 372 2016-01-04 14:48
> /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
> /hbase/data/hbase/meta/.tmp
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740
>
> -rw-r--r-- 3 hdfs hadoop 32 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740/.regioninfo
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 14:47
> /hbase/data/hbase/meta/1588230740/info
>
> drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740/recovered.edits
>
> -rw-r--r-- 3 hdfs hadoop 0 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
>
> And I'm not seeing any "namespace" directory.
>
> Will check out the server now...
>
> Shane O.
>
> ========================
> Shane O'Donnell
> <
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6a0326e0-41e4-46fd-bcce-114f44302f69
> >
> 4819 Emperor Blvd., Ste 400
> Durham, North Carolina 27703
> tel: +1.424.262.KNOW x703
> skype: shaneodonnell
> email: shaneo@knownormal.com
> ========================
> :wq!
>
> On Mon, Jan 4, 2016 at 9:44 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Can you log onto the server hosting region
> 82432aca9ede964943b40753cb64e808
> > and see what happened ?
> >
> > See if the namespace table can be found under rootdir.
> > e.g. assuming /apps/hbase/data is the rootdir, you should see something
> > similar to the following:
> >
> > hdfs dfs -ls /apps/hbase/data/data/hbase/namespace
> > Found 3 items
> > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
> > /apps/hbase/data/data/hbase/namespace/.tabledesc
> > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
> > /apps/hbase/data/data/hbase/namespace/.tmp
> > drwxr-xr-x - hbase hdfs 0 2015-12-15 19:51
> > /apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8
> >
> > On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell <sh...@knownormal.com>
> > wrote:
> >
> > > Some progress...
> > >
> > > /hbase did NOT have either the hbase.id or hbase.version files so I
> > > temporarily changed hbase.rootdir and started the master so the files
> > would
> > > be recreated elsewhere and copied them in.
> > >
> > > Now it starts fine, but my hbase tables are gone. Specifically, I'm
> > > getting this error:
> > >
> > > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
> > > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
> > > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
> > >
> > > Is my existing data toast, or is there a crafty way out of this?
> > >
> > > Thanks -
> > >
> > > Shane O.
> > >
> > > ========================
> > > Shane O'Donnell
> > > <
> > >
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
> > > >
> > > 4819 Emperor Blvd., Ste 400
> > > Durham, North Carolina 27703
> > > tel: +1.424.262.KNOW x703
> > > skype: shaneodonnell
> > > email: shaneo@knownormal.com
> > > ========================
> > > :wq!
> > >
> > > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <
> shaneo@knownormal.com>
> > > wrote:
> > >
> > > > Hi -
> > > >
> > > > My cluster has been running perfectly until the other day when I
> found
> > it
> > > > down.
> > > >
> > > > The error seems to be related not being able to get the ClusterID
> from
> > > > zookeeper, but I'm stumped as to what to do about it. This seems to
> be
> > > the
> > > > relevant part of the master's log:
> > > >
> > > > http://pastebin.com/C3iaxM3p
> > > >
> > > > starting at line 235 (also highlighted).
> > > >
> > > > Any help is appreciated!
> > > >
> > > > Thanks -
> > > >
> > > > Shane O.
> > > > ========================
> > > > Shane O'Donnell
> > > >
> > > > <
> > >
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> > > >
> > > > 4819 Emperor Blvd., Ste 400
> > > > Durham, North Carolina 27703
> > > > tel: +1.424.262.KNOW x703
> > > > skype: shaneodonnell
> > > > email: shaneo@knownormal.com
> > > > ========================
> > > > :wq!
> > > >
> > >
> >
>
Re: Hbase master not starting
Posted by Shane O'Donnell <sh...@knownormal.com>.
I attempted to restore a backed-up "meta" directory from a copy made by
'hbase hbck', so my directory structure may be messed up:
[user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls -R
/hbase/data/hbase
drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
/hbase/data/hbase/meta
drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
/hbase/data/hbase/meta/.tabledesc
-rw-r--r-- 2 hbase hadoop 372 2016-01-04 14:48
/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
drwxr-xr-x - hbase hadoop 0 2016-01-04 14:48
/hbase/data/hbase/meta/.tmp
drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740
-rw-r--r-- 3 hdfs hadoop 32 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/.regioninfo
drwxr-xr-x - hbase hadoop 0 2016-01-04 14:47
/hbase/data/hbase/meta/1588230740/info
drwxr-xr-x - hbase hadoop 0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/recovered.edits
-rw-r--r-- 3 hdfs hadoop 0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
And I'm not seeing any "namespace" directory.
Will check out the server now...
Shane O.
========================
Shane O'Donnell
<http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6a0326e0-41e4-46fd-bcce-114f44302f69>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!
On Mon, Jan 4, 2016 at 9:44 AM, Ted Yu <yu...@gmail.com> wrote:
> Can you log onto the server hosting region 82432aca9ede964943b40753cb64e808
> and see what happened ?
>
> See if the namespace table can be found under rootdir.
> e.g. assuming /apps/hbase/data is the rootdir, you should see something
> similar to the following:
>
> hdfs dfs -ls /apps/hbase/data/data/hbase/namespace
> Found 3 items
> drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
> /apps/hbase/data/data/hbase/namespace/.tabledesc
> drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
> /apps/hbase/data/data/hbase/namespace/.tmp
> drwxr-xr-x - hbase hdfs 0 2015-12-15 19:51
> /apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8
>
> On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell <sh...@knownormal.com>
> wrote:
>
> > Some progress...
> >
> > /hbase did NOT have either the hbase.id or hbase.version files so I
> > temporarily changed hbase.rootdir and started the master so the files
> would
> > be recreated elsewhere and copied them in.
> >
> > Now it starts fine, but my hbase tables are gone. Specifically, I'm
> > getting this error:
> >
> > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
> > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
> > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
> >
> > Is my existing data toast, or is there a crafty way out of this?
> >
> > Thanks -
> >
> > Shane O.
> >
> > ========================
> > Shane O'Donnell
> > <
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
> > >
> > 4819 Emperor Blvd., Ste 400
> > Durham, North Carolina 27703
> > tel: +1.424.262.KNOW x703
> > skype: shaneodonnell
> > email: shaneo@knownormal.com
> > ========================
> > :wq!
> >
> > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <sh...@knownormal.com>
> > wrote:
> >
> > > Hi -
> > >
> > > My cluster has been running perfectly until the other day when I found
> it
> > > down.
> > >
> > > The error seems to be related not being able to get the ClusterID from
> > > zookeeper, but I'm stumped as to what to do about it. This seems to be
> > the
> > > relevant part of the master's log:
> > >
> > > http://pastebin.com/C3iaxM3p
> > >
> > > starting at line 235 (also highlighted).
> > >
> > > Any help is appreciated!
> > >
> > > Thanks -
> > >
> > > Shane O.
> > > ========================
> > > Shane O'Donnell
> > >
> > > <
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> > >
> > > 4819 Emperor Blvd., Ste 400
> > > Durham, North Carolina 27703
> > > tel: +1.424.262.KNOW x703
> > > skype: shaneodonnell
> > > email: shaneo@knownormal.com
> > > ========================
> > > :wq!
> > >
> >
>
Re: Hbase master not starting
Posted by Ted Yu <yu...@gmail.com>.
Can you log onto the server hosting region 82432aca9ede964943b40753cb64e808
and see what happened ?
See if the namespace table can be found under rootdir.
e.g. assuming /apps/hbase/data is the rootdir, you should see something
similar to the following:
hdfs dfs -ls /apps/hbase/data/data/hbase/namespace
Found 3 items
drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
/apps/hbase/data/data/hbase/namespace/.tabledesc
drwxr-xr-x - hbase hdfs 0 2015-12-15 19:15
/apps/hbase/data/data/hbase/namespace/.tmp
drwxr-xr-x - hbase hdfs 0 2015-12-15 19:51
/apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8
On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell <sh...@knownormal.com>
wrote:
> Some progress...
>
> /hbase did NOT have either the hbase.id or hbase.version files so I
> temporarily changed hbase.rootdir and started the master so the files would
> be recreated elsewhere and copied them in.
>
> Now it starts fine, but my hbase tables are gone. Specifically, I'm
> getting this error:
>
> hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
> state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
> server=ip-10-0-1-29.ec2.internal,60020,1451524442749
>
> Is my existing data toast, or is there a crafty way out of this?
>
> Thanks -
>
> Shane O.
>
> ========================
> Shane O'Donnell
> <
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
> >
> 4819 Emperor Blvd., Ste 400
> Durham, North Carolina 27703
> tel: +1.424.262.KNOW x703
> skype: shaneodonnell
> email: shaneo@knownormal.com
> ========================
> :wq!
>
> On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <sh...@knownormal.com>
> wrote:
>
> > Hi -
> >
> > My cluster has been running perfectly until the other day when I found it
> > down.
> >
> > The error seems to be related not being able to get the ClusterID from
> > zookeeper, but I'm stumped as to what to do about it. This seems to be
> the
> > relevant part of the master's log:
> >
> > http://pastebin.com/C3iaxM3p
> >
> > starting at line 235 (also highlighted).
> >
> > Any help is appreciated!
> >
> > Thanks -
> >
> > Shane O.
> > ========================
> > Shane O'Donnell
> >
> > <
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> >
> > 4819 Emperor Blvd., Ste 400
> > Durham, North Carolina 27703
> > tel: +1.424.262.KNOW x703
> > skype: shaneodonnell
> > email: shaneo@knownormal.com
> > ========================
> > :wq!
> >
>
Re: Hbase master not starting
Posted by Dejan Menges <de...@gmail.com>.
Hi Shane,
What was our case - at some point we couldn't bring up HBase anymore having
exactly same issues as you did - not being able to open hbase:namespace. On
HDFS I also couldn't find hbase:namespace anymore for that specific
database, so what I did was next:
- Let's say that your hbase.root was on your hdfs://somewhere:8020/hbase
- Back up all HBase related files on HDFS to different folder
- Change hbase.rootdir and zookeeper.znode.parent to something new, like
hbasetemp
- Start hbase, it will set up empty namespace and meta for you, with no
tables, on your new location
- Copy first table from ${old_hbase_root}/data/default/${table_name} to
${new_hbase_root}/data/default/
- Run hbck until it's saying all is good. You will most likely have to run
it multiple times with different options. Hopefully, if your tables are not
corrupted, and you have just issue with namespace, hbck will rebuild just
meta and namespace, and you will be good.
- Repeat this for every table, verify your data is still there in the end
of course. In the end if all is good you should have your tables recovered.
If you really want, you can take HBase down again, rename two parameters
from above and rename hbase.root on HDFS and restart HBase again, to have
stuff up and running on the old location.
Just to say it again: this was desperate man try, and in our case it
worked. Ok, we also wanted to test if it will work, but this wasn't
production at all, but wasn't also environment where we could loose any
data.
Good luck!
On Mon, Jan 4, 2016 at 9:42 PM Shane O'Donnell <sh...@knownormal.com>
wrote:
> Thanks for this, Dejan!
>
> Where did you have your tables to move them in? I know the names of all of
> my tables, but I'm not sure how I'd go about "copying my table to the new
> Hbase location". Can you explain?
>
> Shane O.
>
> ========================
> Shane O'Donnell
> <https://www.knownormal.com/>
> 4819 Emperor Blvd., Ste 400
> Durham, North Carolina 27703
> tel: +1.424.262.KNOW x703
> skype: shaneodonnell
> email: shaneo@knownormal.com
> ========================
> :wq!
>
> On Mon, Jan 4, 2016 at 9:44 AM, Dejan Menges <de...@gmail.com>
> wrote:
>
> > I had once exactly this situation, and was able (but that was some
> > development database, so wasn't super production big deal) to get over it
> > with:
> >
> > - Stop HBase
> > - Copy your table to your new HBase location
> > - Start your HBase
> > - Run HBCK until it says all is good and you are sure you have your data
> > - Repeat for every table you want to move.
> >
> > I'm quite sure this is not recommended way for anything, but that was the
> > only way for me once I hit hbase:namespace issue like yours...
> >
> > Good luck!
> >
> > On Mon, Jan 4, 2016 at 3:37 PM Shane O'Donnell <sh...@knownormal.com>
> > wrote:
> >
> > > Some progress...
> > >
> > > /hbase did NOT have either the hbase.id or hbase.version files so I
> > > temporarily changed hbase.rootdir and started the master so the files
> > would
> > > be recreated elsewhere and copied them in.
> > >
> > > Now it starts fine, but my hbase tables are gone. Specifically, I'm
> > > getting this error:
> > >
> > > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
> > > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
> > > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
> > >
> > > Is my existing data toast, or is there a crafty way out of this?
> > >
> > > Thanks -
> > >
> > > Shane O.
> > >
> > > ========================
> > > Shane O'Donnell
> > > <
> > >
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
> > > >
> > > 4819 Emperor Blvd., Ste 400
> > > Durham, North Carolina 27703
> > > tel: +1.424.262.KNOW x703
> > > skype: shaneodonnell
> > > email: shaneo@knownormal.com
> > > ========================
> > > :wq!
> > >
> > > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <
> shaneo@knownormal.com>
> > > wrote:
> > >
> > > > Hi -
> > > >
> > > > My cluster has been running perfectly until the other day when I
> found
> > it
> > > > down.
> > > >
> > > > The error seems to be related not being able to get the ClusterID
> from
> > > > zookeeper, but I'm stumped as to what to do about it. This seems to
> be
> > > the
> > > > relevant part of the master's log:
> > > >
> > > > http://pastebin.com/C3iaxM3p
> > > >
> > > > starting at line 235 (also highlighted).
> > > >
> > > > Any help is appreciated!
> > > >
> > > > Thanks -
> > > >
> > > > Shane O.
> > > > ========================
> > > > Shane O'Donnell
> > > >
> > > > <
> > >
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> > > >
> > > > 4819 Emperor Blvd., Ste 400
> > > > Durham, North Carolina 27703
> > > > tel: +1.424.262.KNOW x703
> > > > skype: shaneodonnell
> > > > email: shaneo@knownormal.com
> > > > ========================
> > > > :wq!
> > > >
> > >
> >
>
Re: Hbase master not starting
Posted by Shane O'Donnell <sh...@knownormal.com>.
Thanks for this, Dejan!
Where did you have your tables to move them in? I know the names of all of
my tables, but I'm not sure how I'd go about "copying my table to the new
Hbase location". Can you explain?
Shane O.
========================
Shane O'Donnell
<https://www.knownormal.com/>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!
On Mon, Jan 4, 2016 at 9:44 AM, Dejan Menges <de...@gmail.com> wrote:
> I had once exactly this situation, and was able (but that was some
> development database, so wasn't super production big deal) to get over it
> with:
>
> - Stop HBase
> - Copy your table to your new HBase location
> - Start your HBase
> - Run HBCK until it says all is good and you are sure you have your data
> - Repeat for every table you want to move.
>
> I'm quite sure this is not recommended way for anything, but that was the
> only way for me once I hit hbase:namespace issue like yours...
>
> Good luck!
>
> On Mon, Jan 4, 2016 at 3:37 PM Shane O'Donnell <sh...@knownormal.com>
> wrote:
>
> > Some progress...
> >
> > /hbase did NOT have either the hbase.id or hbase.version files so I
> > temporarily changed hbase.rootdir and started the master so the files
> would
> > be recreated elsewhere and copied them in.
> >
> > Now it starts fine, but my hbase tables are gone. Specifically, I'm
> > getting this error:
> >
> > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
> > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
> > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
> >
> > Is my existing data toast, or is there a crafty way out of this?
> >
> > Thanks -
> >
> > Shane O.
> >
> > ========================
> > Shane O'Donnell
> > <
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
> > >
> > 4819 Emperor Blvd., Ste 400
> > Durham, North Carolina 27703
> > tel: +1.424.262.KNOW x703
> > skype: shaneodonnell
> > email: shaneo@knownormal.com
> > ========================
> > :wq!
> >
> > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <sh...@knownormal.com>
> > wrote:
> >
> > > Hi -
> > >
> > > My cluster has been running perfectly until the other day when I found
> it
> > > down.
> > >
> > > The error seems to be related not being able to get the ClusterID from
> > > zookeeper, but I'm stumped as to what to do about it. This seems to be
> > the
> > > relevant part of the master's log:
> > >
> > > http://pastebin.com/C3iaxM3p
> > >
> > > starting at line 235 (also highlighted).
> > >
> > > Any help is appreciated!
> > >
> > > Thanks -
> > >
> > > Shane O.
> > > ========================
> > > Shane O'Donnell
> > >
> > > <
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> > >
> > > 4819 Emperor Blvd., Ste 400
> > > Durham, North Carolina 27703
> > > tel: +1.424.262.KNOW x703
> > > skype: shaneodonnell
> > > email: shaneo@knownormal.com
> > > ========================
> > > :wq!
> > >
> >
>
Re: Hbase master not starting
Posted by Dejan Menges <de...@gmail.com>.
I had once exactly this situation, and was able (but that was some
development database, so wasn't super production big deal) to get over it
with:
- Stop HBase
- Copy your table to your new HBase location
- Start your HBase
- Run HBCK until it says all is good and you are sure you have your data
- Repeat for every table you want to move.
I'm quite sure this is not recommended way for anything, but that was the
only way for me once I hit hbase:namespace issue like yours...
Good luck!
On Mon, Jan 4, 2016 at 3:37 PM Shane O'Donnell <sh...@knownormal.com>
wrote:
> Some progress...
>
> /hbase did NOT have either the hbase.id or hbase.version files so I
> temporarily changed hbase.rootdir and started the master so the files would
> be recreated elsewhere and copied them in.
>
> Now it starts fine, but my hbase tables are gone. Specifically, I'm
> getting this error:
>
> hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
> state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
> server=ip-10-0-1-29.ec2.internal,60020,1451524442749
>
> Is my existing data toast, or is there a crafty way out of this?
>
> Thanks -
>
> Shane O.
>
> ========================
> Shane O'Donnell
> <
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
> >
> 4819 Emperor Blvd., Ste 400
> Durham, North Carolina 27703
> tel: +1.424.262.KNOW x703
> skype: shaneodonnell
> email: shaneo@knownormal.com
> ========================
> :wq!
>
> On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <sh...@knownormal.com>
> wrote:
>
> > Hi -
> >
> > My cluster has been running perfectly until the other day when I found it
> > down.
> >
> > The error seems to be related not being able to get the ClusterID from
> > zookeeper, but I'm stumped as to what to do about it. This seems to be
> the
> > relevant part of the master's log:
> >
> > http://pastebin.com/C3iaxM3p
> >
> > starting at line 235 (also highlighted).
> >
> > Any help is appreciated!
> >
> > Thanks -
> >
> > Shane O.
> > ========================
> > Shane O'Donnell
> >
> > <
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> >
> > 4819 Emperor Blvd., Ste 400
> > Durham, North Carolina 27703
> > tel: +1.424.262.KNOW x703
> > skype: shaneodonnell
> > email: shaneo@knownormal.com
> > ========================
> > :wq!
> >
>
Re: Hbase master not starting
Posted by Shane O'Donnell <sh...@knownormal.com>.
Some progress...
/hbase did NOT have either the hbase.id or hbase.version files so I
temporarily changed hbase.rootdir and started the master so the files would
be recreated elsewhere and copied them in.
Now it starts fine, but my hbase tables are gone. Specifically, I'm
getting this error:
hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
server=ip-10-0-1-29.ec2.internal,60020,1451524442749
Is my existing data toast, or is there a crafty way out of this?
Thanks -
Shane O.
========================
Shane O'Donnell
<http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!
On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <sh...@knownormal.com>
wrote:
> Hi -
>
> My cluster has been running perfectly until the other day when I found it
> down.
>
> The error seems to be related not being able to get the ClusterID from
> zookeeper, but I'm stumped as to what to do about it. This seems to be the
> relevant part of the master's log:
>
> http://pastebin.com/C3iaxM3p
>
> starting at line 235 (also highlighted).
>
> Any help is appreciated!
>
> Thanks -
>
> Shane O.
> ========================
> Shane O'Donnell
>
> <http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac>
> 4819 Emperor Blvd., Ste 400
> Durham, North Carolina 27703
> tel: +1.424.262.KNOW x703
> skype: shaneodonnell
> email: shaneo@knownormal.com
> ========================
> :wq!
>
Re: Hbase master not starting
Posted by Shane O'Donnell <sh...@knownormal.com>.
My rootdir is /hbase, but my directory structure under it doesn't contain
hbase.version. Here's what I've got:
[user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls -R
/hbase/data
drwxr-xr-x - hdfs hadoop 0 2016-01-01 21:52 /hbase/data/hbase
drwxr-xr-x - hdfs hadoop 0 2016-01-01 21:52
/hbase/data/hbase/meta
drwxr-xr-x - hdfs hadoop 0 2016-01-01 21:52
/hbase/data/hbase/meta/1588230740
-rw-r--r-- 2 hdfs hadoop 32 2016-01-01 21:52
/hbase/data/hbase/meta/1588230740/.regioninfo
drwxr-xr-x - hdfs hadoop 0 2016-01-01 21:52
/hbase/data/hbase/meta/1588230740/info
drwxr-xr-x - hdfs hadoop 0 2016-01-01 21:52
/hbase/data/hbase/meta/1588230740/recovered.edits
-rw-r--r-- 2 hdfs hadoop 0 2016-01-01 21:52
/hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
I've run the OfflineMetaRepair and tried various forms of "hbase hbck".
Hope this helps -
Shane O.
========================
Shane O'Donnell
<https://www.knownormal.com/>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!
On Sun, Jan 3, 2016 at 10:54 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. HBase file layout needs to be upgraded. You have version null and I
> want version 8.
> Can you check hbase.version under hbase rootdir ?
> On a healthy system, you should see something like the following:
>
> hdfs dfs -cat /apps/hbase/data/hbase.version
> PBUF
> 8
>
> Cheers
>
> On Sun, Jan 3, 2016 at 7:21 PM, Shane O'Donnell <sh...@knownormal.com>
> wrote:
>
> > Hi -
> >
> > My cluster has been running perfectly until the other day when I found it
> > down.
> >
> > The error seems to be related not being able to get the ClusterID from
> > zookeeper, but I'm stumped as to what to do about it. This seems to be
> the
> > relevant part of the master's log:
> >
> > http://pastebin.com/C3iaxM3p
> >
> > starting at line 235 (also highlighted).
> >
> > Any help is appreciated!
> >
> > Thanks -
> >
> > Shane O.
> > ========================
> > Shane O'Donnell
> > <
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> > >
> > 4819 Emperor Blvd., Ste 400
> > Durham, North Carolina 27703
> > tel: +1.424.262.KNOW x703
> > skype: shaneodonnell
> > email: shaneo@knownormal.com
> > ========================
> > :wq!
> >
>
Re: Hbase master not starting
Posted by Ted Yu <yu...@gmail.com>.
bq. HBase file layout needs to be upgraded. You have version null and I
want version 8.
Can you check hbase.version under hbase rootdir ?
On a healthy system, you should see something like the following:
hdfs dfs -cat /apps/hbase/data/hbase.version
PBUF
8
Cheers
On Sun, Jan 3, 2016 at 7:21 PM, Shane O'Donnell <sh...@knownormal.com>
wrote:
> Hi -
>
> My cluster has been running perfectly until the other day when I found it
> down.
>
> The error seems to be related not being able to get the ClusterID from
> zookeeper, but I'm stumped as to what to do about it. This seems to be the
> relevant part of the master's log:
>
> http://pastebin.com/C3iaxM3p
>
> starting at line 235 (also highlighted).
>
> Any help is appreciated!
>
> Thanks -
>
> Shane O.
> ========================
> Shane O'Donnell
> <
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> >
> 4819 Emperor Blvd., Ste 400
> Durham, North Carolina 27703
> tel: +1.424.262.KNOW x703
> skype: shaneodonnell
> email: shaneo@knownormal.com
> ========================
> :wq!
>