You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Kristoffer Sjögren <st...@gmail.com> on 2014/10/14 09:27:07 UTC
Force remove table
Hi
I accidentally created a few tables with 'lzo' compression without the jar
dependencies in hbase 0.94.7. This caused havoc on the master which
eventually crashed and now I can't start it again [1].
I tried to remove the tables but I can't - hbase shell is complaining about
the master is not running so i'm stuck.
Is there a way to forcefully delete these tables or any other way to get
the master back up?
Cheers,
-Kristoffer
1. http://pastebin.com/Jprr6j39
Re: Force remove table
Posted by Kristoffer Sjögren <st...@gmail.com>.
The master did start after doing the following from hbase zkcli
$ rmr /hbase/unassigned
After this is was able to use hbase shell to remove the tables.
Thanks guys!
On Tue, Oct 14, 2014 at 10:27 AM, Dima Spivak <ds...@cloudera.com> wrote:
> Kristoffer,
>
> It looks like this is described in
> https://issues.apache.org/jira/browse/HBASE-8912. In short, stop HBase,
> remove the affected znode, restart HBase, and then run hbase hbck
> -fixAssignments . Longer term, it may be worth upgrading past 0.94.16, if
> possible, because this issue has since been resolved.
>
> Let us know how it goes!
>
> -Dima
>
> On Tue, Oct 14, 2014 at 1:11 AM, Kristoffer Sjögren <st...@gmail.com>
> wrote:
>
> > I was thinking of doing that but I suspect that zookeeper keeps metadata
> of
> > tables also. Seems like region servers are fine for now without the
> master
> > and I don't want to make the problem worse by taking chances.
> >
> > If I run 'hbase hbck' it complains about master is not running also.
> >
> > I found the following two posts which indicate that I should remove the
> > zookeeper nodes. Can somebody confirm?
> >
> > 1. http://stackoverflow.com/a/14557937/1195960
> > 2. http://arnon.me/2013/01/killing-hbase-zombie-table/
> >
> >
> > On Tue, Oct 14, 2014 at 9:51 AM, lagend <la...@foxmail.com> wrote:
> >
> > > Maybe you can remove the hdfs files of those bad tables.
> > > like： bin/hadoop fs -rm /hbase/TABLENAME
> > >
> > >
> > >
> > >
> > > ------------------ 原始邮件 ------------------
> > > 发件人: "Kristoffer Sjögren";<st...@gmail.com>;
> > > 发送时间: 2014年10月14日(星期二) 下午3:27
> > > 收件人: "user"<us...@hbase.apache.org>;
> > >
> > > 主题: Force remove table
> > >
> > >
> > >
> > > Hi
> > >
> > > I accidentally created a few tables with 'lzo' compression without the
> > jar
> > > dependencies in hbase 0.94.7. This caused havoc on the master which
> > > eventually crashed and now I can't start it again [1].
> > >
> > > I tried to remove the tables but I can't - hbase shell is complaining
> > about
> > > the master is not running so i'm stuck.
> > >
> > > Is there a way to forcefully delete these tables or any other way to
> get
> > > the master back up?
> > >
> > > Cheers,
> > > -Kristoffer
> > >
> > >
> > > 1. http://pastebin.com/Jprr6j39
> > >
> >
>
Re: Force remove table
Posted by Dima Spivak <ds...@cloudera.com>.
Kristoffer,
It looks like this is described in
https://issues.apache.org/jira/browse/HBASE-8912. In short, stop HBase,
remove the affected znode, restart HBase, and then run hbase hbck
-fixAssignments . Longer term, it may be worth upgrading past 0.94.16, if
possible, because this issue has since been resolved.
Let us know how it goes!
-Dima
On Tue, Oct 14, 2014 at 1:11 AM, Kristoffer Sjögren <st...@gmail.com>
wrote:
> I was thinking of doing that but I suspect that zookeeper keeps metadata of
> tables also. Seems like region servers are fine for now without the master
> and I don't want to make the problem worse by taking chances.
>
> If I run 'hbase hbck' it complains about master is not running also.
>
> I found the following two posts which indicate that I should remove the
> zookeeper nodes. Can somebody confirm?
>
> 1. http://stackoverflow.com/a/14557937/1195960
> 2. http://arnon.me/2013/01/killing-hbase-zombie-table/
>
>
> On Tue, Oct 14, 2014 at 9:51 AM, lagend <la...@foxmail.com> wrote:
>
> > Maybe you can remove the hdfs files of those bad tables.
> > like： bin/hadoop fs -rm /hbase/TABLENAME
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "Kristoffer Sjögren";<st...@gmail.com>;
> > 发送时间: 2014年10月14日(星期二) 下午3:27
> > 收件人: "user"<us...@hbase.apache.org>;
> >
> > 主题: Force remove table
> >
> >
> >
> > Hi
> >
> > I accidentally created a few tables with 'lzo' compression without the
> jar
> > dependencies in hbase 0.94.7. This caused havoc on the master which
> > eventually crashed and now I can't start it again [1].
> >
> > I tried to remove the tables but I can't - hbase shell is complaining
> about
> > the master is not running so i'm stuck.
> >
> > Is there a way to forcefully delete these tables or any other way to get
> > the master back up?
> >
> > Cheers,
> > -Kristoffer
> >
> >
> > 1. http://pastebin.com/Jprr6j39
> >
>
Re: Force remove table
Posted by Samir Ahmic <ah...@gmail.com>.
HI Kristoffer,
Just removing table data from HDFS will leave hbase in inconsistent state.
Yes, you are right you will have to remove table znode from zookeeper to.
This is what would i do in your case:
1. remove table data form hdfs (hadoop fs -rmr /hbase/TABLE_NAME)
2. remove znode from zookeeper (hbase zkcli; rmr /hbase/table/TABLE_NAME)
3. start master (it should start now since offending table is removed)
4, run hbase hbck to check state
5. if needed run hbase hbck -repair -repairHoles
6. check zookeeper /hbase/table to see is table is gone if not remove it
again and restart master
This should fix issue
Good luck
Samir
On Tue, Oct 14, 2014 at 10:11 AM, Kristoffer Sjögren <st...@gmail.com>
wrote:
> I was thinking of doing that but I suspect that zookeeper keeps metadata of
> tables also. Seems like region servers are fine for now without the master
> and I don't want to make the problem worse by taking chances.
>
> If I run 'hbase hbck' it complains about master is not running also.
>
> I found the following two posts which indicate that I should remove the
> zookeeper nodes. Can somebody confirm?
>
> 1. http://stackoverflow.com/a/14557937/1195960
> 2. http://arnon.me/2013/01/killing-hbase-zombie-table/
>
>
> On Tue, Oct 14, 2014 at 9:51 AM, lagend <la...@foxmail.com> wrote:
>
> > Maybe you can remove the hdfs files of those bad tables.
> > like： bin/hadoop fs -rm /hbase/TABLENAME
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "Kristoffer Sjögren";<st...@gmail.com>;
> > 发送时间: 2014年10月14日(星期二) 下午3:27
> > 收件人: "user"<us...@hbase.apache.org>;
> >
> > 主题: Force remove table
> >
> >
> >
> > Hi
> >
> > I accidentally created a few tables with 'lzo' compression without the
> jar
> > dependencies in hbase 0.94.7. This caused havoc on the master which
> > eventually crashed and now I can't start it again [1].
> >
> > I tried to remove the tables but I can't - hbase shell is complaining
> about
> > the master is not running so i'm stuck.
> >
> > Is there a way to forcefully delete these tables or any other way to get
> > the master back up?
> >
> > Cheers,
> > -Kristoffer
> >
> >
> > 1. http://pastebin.com/Jprr6j39
> >
>
Re: Force remove table
Posted by Kristoffer Sjögren <st...@gmail.com>.
I was thinking of doing that but I suspect that zookeeper keeps metadata of
tables also. Seems like region servers are fine for now without the master
and I don't want to make the problem worse by taking chances.
If I run 'hbase hbck' it complains about master is not running also.
I found the following two posts which indicate that I should remove the
zookeeper nodes. Can somebody confirm?
1. http://stackoverflow.com/a/14557937/1195960
2. http://arnon.me/2013/01/killing-hbase-zombie-table/
On Tue, Oct 14, 2014 at 9:51 AM, lagend <la...@foxmail.com> wrote:
> Maybe you can remove the hdfs files of those bad tables.
> like： bin/hadoop fs -rm /hbase/TABLENAME
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Kristoffer Sjögren";<st...@gmail.com>;
> 发送时间: 2014年10月14日(星期二) 下午3:27
> 收件人: "user"<us...@hbase.apache.org>;
>
> 主题: Force remove table
>
>
>
> Hi
>
> I accidentally created a few tables with 'lzo' compression without the jar
> dependencies in hbase 0.94.7. This caused havoc on the master which
> eventually crashed and now I can't start it again [1].
>
> I tried to remove the tables but I can't - hbase shell is complaining about
> the master is not running so i'm stuck.
>
> Is there a way to forcefully delete these tables or any other way to get
> the master back up?
>
> Cheers,
> -Kristoffer
>
>
> 1. http://pastebin.com/Jprr6j39
>
回复：Force remove table
Posted by lagend <la...@foxmail.com>.
Maybe you can remove the hdfs files of those bad tables.
like： bin/hadoop fs -rm /hbase/TABLENAME
------------------ 原始邮件 ------------------
发件人: "Kristoffer Sjögren";<st...@gmail.com>;
发送时间: 2014年10月14日(星期二) 下午3:27
收件人: "user"<us...@hbase.apache.org>;
主题: Force remove table
Hi
I accidentally created a few tables with 'lzo' compression without the jar
dependencies in hbase 0.94.7. This caused havoc on the master which
eventually crashed and now I can't start it again [1].
I tried to remove the tables but I can't - hbase shell is complaining about
the master is not running so i'm stuck.
Is there a way to forcefully delete these tables or any other way to get
the master back up?
Cheers,
-Kristoffer
1. http://pastebin.com/Jprr6j39