You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Cecile, Adam" <Ad...@hitec.lu> on 2016/11/22 12:16:44 UTC
Table is disabled an no way to get it back online
Hello,
We're having a table stuck in disabled state. First I'd like to start with what I tried already:
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
Now more details anout what happens:
* When enabling from hbase shell:
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's talking about itself (and it gets stuck for a while before saying anything)
In the log, all I can see is:
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0] procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure: Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0] procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure: Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on a single active master. You'll see the table is found as well as the regions but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
RE: Table is disabled an no way to get it back online
Posted by "Cecile, Adam" <Ad...@hitec.lu>.
Hello,
Thanks a lot, the table is back online. One last question ? Can you provide a log pattern to spot this ? Just in case it occurs again ;-)
Regards, Adam.
________________________________________
De : Matteo Bertozzi <th...@gmail.com>
Envoyé : mardi 22 novembre 2016 19:48
À : user@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online
hadoop fs -rmr /hbase/MasterProcWALs
Matteo
On Tue, Nov 22, 2016 at 10:42 AM, Cecile, Adam <Ad...@hitec.lu> wrote:
> Can you explain me how to delete masterwals directory?
>
>
>
> Sent from my Samsung device
>
>
> -------- Original message --------
> From: Matteo Bertozzi <th...@gmail.com>
> Date: 22/11/2016 19:12 (GMT+01:00)
> To: user@hbase.apache.org
> Subject: Re: Table is disabled an no way to get it back online
>
> I don't think this has anything to do with HBASE-13415 or the bugfix it
> related to it i'm working on.
>
> this is probably the usual case of mismatch state with zk. Disable is
> saying that the table is already not disabled.
> so, enableTable() is the one that should give any exception in case. but I
> don't see any.
> I suggest to just drop the MasterWALs directory, drop the znode
> /hbase/table/sentinel-meta, restart the master and try disable and then
> enable. that should bring you back with the table online
>
> Matteo
>
>
> On Tue, Nov 22, 2016 at 10:05 AM, Cecile, Adam <Ad...@hitec.lu>
> wrote:
>
> > Thanks for everything. As you said, this bug is supposed to be fixed in
> > 1.2.0. Matteo is reading this list as well ?
> >
> > Regards, Adam.
> > ________________________________________
> > De : Ted Yu <yu...@gmail.com>
> > Envoyé : mardi 22 novembre 2016 19:00
> > À : user@hbase.apache.org
> > Objet : Re: Table is disabled an no way to get it back online
> >
> > Please take a look at HBASE-13415
> >
> > From the log, you're using hbase 1.2.0 already. But I heard there is a
> > subtle bug which is being fixed.
> >
> > Matteo is the person with best knowledge in this regard.
> >
> > On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <Ad...@hitec.lu>
> > wrote:
> >
> > > Another one, because I'm not sure the log is overwritten when
> restarting.
> > > This one has been cleared before service start.
> > > ________________________________________
> > > De : Cecile, Adam <Ad...@hitec.lu>
> > > Envoyé : mardi 22 novembre 2016 18:42
> > > À : user@hbase.apache.org
> > > Objet : RE: Table is disabled an no way to get it back online
> > >
> > > Hello,
> > >
> > > Sadly I could not use the webui, it killed my firefox (probably way too
> > > much time). Here is the debug log... (11Mb uncompressed for maybe two
> > > minutes running !!)
> > >
> > > Best regards, Adam.
> > > ________________________________________
> > > De : Ted Yu <yu...@gmail.com>
> > > Envoyé : mardi 22 novembre 2016 17:05
> > > À : user@hbase.apache.org
> > > Objet : Re: Table is disabled an no way to get it back online
> > >
> > > In log4j.properties :
> > >
> > > log4j.logger.org.apache.hadoop.hbase=DEBUG
> > >
> > > On master UI, you can select the Procedures tab. Pastebin what you see
> > > (text is enough).
> > >
> > > Thanks
> > >
> > > On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <Ad...@hitec.lu>
> > > wrote:
> > >
> > > > Hey Ted,
> > > >
> > > > Thank you. Heading home right now but I'll start the laptop again.
> Not
> > > > sure exactly how I should turn debug log on so if you have the
> > > information
> > > > it'd be appreciated, otherwise I'll look at the xml files.
> > > >
> > > > Regards, Adam.
> > > > ________________________________________
> > > > De : Ted Yu <yu...@gmail.com>
> > > > Envoyé : mardi 22 novembre 2016 15:46
> > > > À : user@hbase.apache.org
> > > > Objet : Re: Table is disabled an no way to get it back online
> > > >
> > > > Master log contained entries in the following form:
> > > >
> > > > 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> > DisableTableProcedure
> > > > (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
> > > > exec-time=242hrs, 10mins, 28.896sec
> > > > exception=org.apache.hadoop.hbase.TableNotEnabledException:
> > > sentinel-meta
> > > >
> > > > Note the procedure Id was around 43000, far lower than 147464.
> > > >
> > > > Can you turn debug log on and repost master log ?
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Ad...@hitec.lu>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > >
> > > > > We're having a table stuck in disabled state. First I'd like to
> start
> > > > with
> > > > > what I tried already:
> > > > >
> > > > >
> > > > > * Restart all machines involved in HBase cluster
> > > > >
> > > > > * hbase hbck with varios arguments
> > > > >
> > > > > * hdfs fsck
> > > > >
> > > > > * Purge ZK /hbase and restart masters
> > > > >
> > > > >
> > > > > Now more details anout what happens:
> > > > >
> > > > > * When enabling from hbase shell:
> > > > >
> > > > >
> > > > > hbase(main):002:0> enable "sentinel-meta"
> > > > > ERROR: The procedure 147464 is still running
> > > > >
> > > > >
> > > > > The task ID changes every time I run the command so I think it's
> > > talking
> > > > > about itself (and it gets stuck for a while before saying anything)
> > > > >
> > > > >
> > > > > In the log, all I can see is:
> > > > >
> > > > > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
> > > > > procedure2.ProcedureExecutor: Rolledback procedure
> > > DisableTableProcedure
> > > > > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> > > > > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.
> > hbase.
> > > > TableNotEnabledException:
> > > > > sentinel-meta
> > > > > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
> > > > DisableTableProcedure:
> > > > > Table sentinel-meta isn't enabled; skipping disable
> > > > > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
> > > > > procedure2.ProcedureExecutor: Rolledback procedure
> > > DisableTableProcedure
> > > > > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> > > > > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.
> > hbase.
> > > > TableNotEnabledException:
> > > > > sentinel-meta
> > > > > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
> > > > DisableTableProcedure:
> > > > > Table sentinel-meta isn't enabled; skipping disable
> > > > >
> > > > >
> > > > > Please also find attached a complete log from startup to shutdown
> on
> > a
> > > > > single active master. You'll see the table is found as well as the
> > > > regions
> > > > > but it gets deactivated with no reason.
> > > > >
> > > > >
> > > > > Thanks a lot for your help, we're kinda running out if ideas here.
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > >
> > > > > Adam.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Table is disabled an no way to get it back online
Posted by Matteo Bertozzi <th...@gmail.com>.
hadoop fs -rmr /hbase/MasterProcWALs
Matteo
On Tue, Nov 22, 2016 at 10:42 AM, Cecile, Adam <Ad...@hitec.lu> wrote:
> Can you explain me how to delete masterwals directory?
>
>
>
> Sent from my Samsung device
>
>
> -------- Original message --------
> From: Matteo Bertozzi <th...@gmail.com>
> Date: 22/11/2016 19:12 (GMT+01:00)
> To: user@hbase.apache.org
> Subject: Re: Table is disabled an no way to get it back online
>
> I don't think this has anything to do with HBASE-13415 or the bugfix it
> related to it i'm working on.
>
> this is probably the usual case of mismatch state with zk. Disable is
> saying that the table is already not disabled.
> so, enableTable() is the one that should give any exception in case. but I
> don't see any.
> I suggest to just drop the MasterWALs directory, drop the znode
> /hbase/table/sentinel-meta, restart the master and try disable and then
> enable. that should bring you back with the table online
>
> Matteo
>
>
> On Tue, Nov 22, 2016 at 10:05 AM, Cecile, Adam <Ad...@hitec.lu>
> wrote:
>
> > Thanks for everything. As you said, this bug is supposed to be fixed in
> > 1.2.0. Matteo is reading this list as well ?
> >
> > Regards, Adam.
> > ________________________________________
> > De : Ted Yu <yu...@gmail.com>
> > Envoyé : mardi 22 novembre 2016 19:00
> > À : user@hbase.apache.org
> > Objet : Re: Table is disabled an no way to get it back online
> >
> > Please take a look at HBASE-13415
> >
> > From the log, you're using hbase 1.2.0 already. But I heard there is a
> > subtle bug which is being fixed.
> >
> > Matteo is the person with best knowledge in this regard.
> >
> > On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <Ad...@hitec.lu>
> > wrote:
> >
> > > Another one, because I'm not sure the log is overwritten when
> restarting.
> > > This one has been cleared before service start.
> > > ________________________________________
> > > De : Cecile, Adam <Ad...@hitec.lu>
> > > Envoyé : mardi 22 novembre 2016 18:42
> > > À : user@hbase.apache.org
> > > Objet : RE: Table is disabled an no way to get it back online
> > >
> > > Hello,
> > >
> > > Sadly I could not use the webui, it killed my firefox (probably way too
> > > much time). Here is the debug log... (11Mb uncompressed for maybe two
> > > minutes running !!)
> > >
> > > Best regards, Adam.
> > > ________________________________________
> > > De : Ted Yu <yu...@gmail.com>
> > > Envoyé : mardi 22 novembre 2016 17:05
> > > À : user@hbase.apache.org
> > > Objet : Re: Table is disabled an no way to get it back online
> > >
> > > In log4j.properties :
> > >
> > > log4j.logger.org.apache.hadoop.hbase=DEBUG
> > >
> > > On master UI, you can select the Procedures tab. Pastebin what you see
> > > (text is enough).
> > >
> > > Thanks
> > >
> > > On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <Ad...@hitec.lu>
> > > wrote:
> > >
> > > > Hey Ted,
> > > >
> > > > Thank you. Heading home right now but I'll start the laptop again.
> Not
> > > > sure exactly how I should turn debug log on so if you have the
> > > information
> > > > it'd be appreciated, otherwise I'll look at the xml files.
> > > >
> > > > Regards, Adam.
> > > > ________________________________________
> > > > De : Ted Yu <yu...@gmail.com>
> > > > Envoyé : mardi 22 novembre 2016 15:46
> > > > À : user@hbase.apache.org
> > > > Objet : Re: Table is disabled an no way to get it back online
> > > >
> > > > Master log contained entries in the following form:
> > > >
> > > > 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> > DisableTableProcedure
> > > > (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
> > > > exec-time=242hrs, 10mins, 28.896sec
> > > > exception=org.apache.hadoop.hbase.TableNotEnabledException:
> > > sentinel-meta
> > > >
> > > > Note the procedure Id was around 43000, far lower than 147464.
> > > >
> > > > Can you turn debug log on and repost master log ?
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Ad...@hitec.lu>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > >
> > > > > We're having a table stuck in disabled state. First I'd like to
> start
> > > > with
> > > > > what I tried already:
> > > > >
> > > > >
> > > > > * Restart all machines involved in HBase cluster
> > > > >
> > > > > * hbase hbck with varios arguments
> > > > >
> > > > > * hdfs fsck
> > > > >
> > > > > * Purge ZK /hbase and restart masters
> > > > >
> > > > >
> > > > > Now more details anout what happens:
> > > > >
> > > > > * When enabling from hbase shell:
> > > > >
> > > > >
> > > > > hbase(main):002:0> enable "sentinel-meta"
> > > > > ERROR: The procedure 147464 is still running
> > > > >
> > > > >
> > > > > The task ID changes every time I run the command so I think it's
> > > talking
> > > > > about itself (and it gets stuck for a while before saying anything)
> > > > >
> > > > >
> > > > > In the log, all I can see is:
> > > > >
> > > > > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
> > > > > procedure2.ProcedureExecutor: Rolledback procedure
> > > DisableTableProcedure
> > > > > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> > > > > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.
> > hbase.
> > > > TableNotEnabledException:
> > > > > sentinel-meta
> > > > > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
> > > > DisableTableProcedure:
> > > > > Table sentinel-meta isn't enabled; skipping disable
> > > > > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
> > > > > procedure2.ProcedureExecutor: Rolledback procedure
> > > DisableTableProcedure
> > > > > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> > > > > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.
> > hbase.
> > > > TableNotEnabledException:
> > > > > sentinel-meta
> > > > > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
> > > > DisableTableProcedure:
> > > > > Table sentinel-meta isn't enabled; skipping disable
> > > > >
> > > > >
> > > > > Please also find attached a complete log from startup to shutdown
> on
> > a
> > > > > single active master. You'll see the table is found as well as the
> > > > regions
> > > > > but it gets deactivated with no reason.
> > > > >
> > > > >
> > > > > Thanks a lot for your help, we're kinda running out if ideas here.
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > >
> > > > > Adam.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
RE: Table is disabled an no way to get it back online
Posted by "Cecile, Adam" <Ad...@hitec.lu>.
Can you explain me how to delete masterwals directory?
Sent from my Samsung device
-------- Original message --------
From: Matteo Bertozzi <th...@gmail.com>
Date: 22/11/2016 19:12 (GMT+01:00)
To: user@hbase.apache.org
Subject: Re: Table is disabled an no way to get it back online
I don't think this has anything to do with HBASE-13415 or the bugfix it
related to it i'm working on.
this is probably the usual case of mismatch state with zk. Disable is
saying that the table is already not disabled.
so, enableTable() is the one that should give any exception in case. but I
don't see any.
I suggest to just drop the MasterWALs directory, drop the znode
/hbase/table/sentinel-meta, restart the master and try disable and then
enable. that should bring you back with the table online
Matteo
On Tue, Nov 22, 2016 at 10:05 AM, Cecile, Adam <Ad...@hitec.lu> wrote:
> Thanks for everything. As you said, this bug is supposed to be fixed in
> 1.2.0. Matteo is reading this list as well ?
>
> Regards, Adam.
> ________________________________________
> De : Ted Yu <yu...@gmail.com>
> Envoyé : mardi 22 novembre 2016 19:00
> À : user@hbase.apache.org
> Objet : Re: Table is disabled an no way to get it back online
>
> Please take a look at HBASE-13415
>
> From the log, you're using hbase 1.2.0 already. But I heard there is a
> subtle bug which is being fixed.
>
> Matteo is the person with best knowledge in this regard.
>
> On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <Ad...@hitec.lu>
> wrote:
>
> > Another one, because I'm not sure the log is overwritten when restarting.
> > This one has been cleared before service start.
> > ________________________________________
> > De : Cecile, Adam <Ad...@hitec.lu>
> > Envoyé : mardi 22 novembre 2016 18:42
> > À : user@hbase.apache.org
> > Objet : RE: Table is disabled an no way to get it back online
> >
> > Hello,
> >
> > Sadly I could not use the webui, it killed my firefox (probably way too
> > much time). Here is the debug log... (11Mb uncompressed for maybe two
> > minutes running !!)
> >
> > Best regards, Adam.
> > ________________________________________
> > De : Ted Yu <yu...@gmail.com>
> > Envoyé : mardi 22 novembre 2016 17:05
> > À : user@hbase.apache.org
> > Objet : Re: Table is disabled an no way to get it back online
> >
> > In log4j.properties :
> >
> > log4j.logger.org.apache.hadoop.hbase=DEBUG
> >
> > On master UI, you can select the Procedures tab. Pastebin what you see
> > (text is enough).
> >
> > Thanks
> >
> > On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <Ad...@hitec.lu>
> > wrote:
> >
> > > Hey Ted,
> > >
> > > Thank you. Heading home right now but I'll start the laptop again. Not
> > > sure exactly how I should turn debug log on so if you have the
> > information
> > > it'd be appreciated, otherwise I'll look at the xml files.
> > >
> > > Regards, Adam.
> > > ________________________________________
> > > De : Ted Yu <yu...@gmail.com>
> > > Envoyé : mardi 22 novembre 2016 15:46
> > > À : user@hbase.apache.org
> > > Objet : Re: Table is disabled an no way to get it back online
> > >
> > > Master log contained entries in the following form:
> > >
> > > 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
> > > procedure2.ProcedureExecutor: Rolledback procedure
> DisableTableProcedure
> > > (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
> > > exec-time=242hrs, 10mins, 28.896sec
> > > exception=org.apache.hadoop.hbase.TableNotEnabledException:
> > sentinel-meta
> > >
> > > Note the procedure Id was around 43000, far lower than 147464.
> > >
> > > Can you turn debug log on and repost master log ?
> > >
> > > Thanks
> > >
> > > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Ad...@hitec.lu>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > >
> > > > We're having a table stuck in disabled state. First I'd like to start
> > > with
> > > > what I tried already:
> > > >
> > > >
> > > > * Restart all machines involved in HBase cluster
> > > >
> > > > * hbase hbck with varios arguments
> > > >
> > > > * hdfs fsck
> > > >
> > > > * Purge ZK /hbase and restart masters
> > > >
> > > >
> > > > Now more details anout what happens:
> > > >
> > > > * When enabling from hbase shell:
> > > >
> > > >
> > > > hbase(main):002:0> enable "sentinel-meta"
> > > > ERROR: The procedure 147464 is still running
> > > >
> > > >
> > > > The task ID changes every time I run the command so I think it's
> > talking
> > > > about itself (and it gets stuck for a while before saying anything)
> > > >
> > > >
> > > > In the log, all I can see is:
> > > >
> > > > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> > DisableTableProcedure
> > > > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> > > > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.
> hbase.
> > > TableNotEnabledException:
> > > > sentinel-meta
> > > > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
> > > DisableTableProcedure:
> > > > Table sentinel-meta isn't enabled; skipping disable
> > > > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> > DisableTableProcedure
> > > > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> > > > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.
> hbase.
> > > TableNotEnabledException:
> > > > sentinel-meta
> > > > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
> > > DisableTableProcedure:
> > > > Table sentinel-meta isn't enabled; skipping disable
> > > >
> > > >
> > > > Please also find attached a complete log from startup to shutdown on
> a
> > > > single active master. You'll see the table is found as well as the
> > > regions
> > > > but it gets deactivated with no reason.
> > > >
> > > >
> > > > Thanks a lot for your help, we're kinda running out if ideas here.
> > > >
> > > >
> > > > Best regards,
> > > >
> > > >
> > > > Adam.
> > > >
> > > >
> > > >
> > >
> >
>
Re: Table is disabled an no way to get it back online
Posted by Matteo Bertozzi <th...@gmail.com>.
I don't think this has anything to do with HBASE-13415 or the bugfix it
related to it i'm working on.
this is probably the usual case of mismatch state with zk. Disable is
saying that the table is already not disabled.
so, enableTable() is the one that should give any exception in case. but I
don't see any.
I suggest to just drop the MasterWALs directory, drop the znode
/hbase/table/sentinel-meta, restart the master and try disable and then
enable. that should bring you back with the table online
Matteo
On Tue, Nov 22, 2016 at 10:05 AM, Cecile, Adam <Ad...@hitec.lu> wrote:
> Thanks for everything. As you said, this bug is supposed to be fixed in
> 1.2.0. Matteo is reading this list as well ?
>
> Regards, Adam.
> ________________________________________
> De : Ted Yu <yu...@gmail.com>
> Envoyé : mardi 22 novembre 2016 19:00
> À : user@hbase.apache.org
> Objet : Re: Table is disabled an no way to get it back online
>
> Please take a look at HBASE-13415
>
> From the log, you're using hbase 1.2.0 already. But I heard there is a
> subtle bug which is being fixed.
>
> Matteo is the person with best knowledge in this regard.
>
> On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <Ad...@hitec.lu>
> wrote:
>
> > Another one, because I'm not sure the log is overwritten when restarting.
> > This one has been cleared before service start.
> > ________________________________________
> > De : Cecile, Adam <Ad...@hitec.lu>
> > Envoyé : mardi 22 novembre 2016 18:42
> > À : user@hbase.apache.org
> > Objet : RE: Table is disabled an no way to get it back online
> >
> > Hello,
> >
> > Sadly I could not use the webui, it killed my firefox (probably way too
> > much time). Here is the debug log... (11Mb uncompressed for maybe two
> > minutes running !!)
> >
> > Best regards, Adam.
> > ________________________________________
> > De : Ted Yu <yu...@gmail.com>
> > Envoyé : mardi 22 novembre 2016 17:05
> > À : user@hbase.apache.org
> > Objet : Re: Table is disabled an no way to get it back online
> >
> > In log4j.properties :
> >
> > log4j.logger.org.apache.hadoop.hbase=DEBUG
> >
> > On master UI, you can select the Procedures tab. Pastebin what you see
> > (text is enough).
> >
> > Thanks
> >
> > On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <Ad...@hitec.lu>
> > wrote:
> >
> > > Hey Ted,
> > >
> > > Thank you. Heading home right now but I'll start the laptop again. Not
> > > sure exactly how I should turn debug log on so if you have the
> > information
> > > it'd be appreciated, otherwise I'll look at the xml files.
> > >
> > > Regards, Adam.
> > > ________________________________________
> > > De : Ted Yu <yu...@gmail.com>
> > > Envoyé : mardi 22 novembre 2016 15:46
> > > À : user@hbase.apache.org
> > > Objet : Re: Table is disabled an no way to get it back online
> > >
> > > Master log contained entries in the following form:
> > >
> > > 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
> > > procedure2.ProcedureExecutor: Rolledback procedure
> DisableTableProcedure
> > > (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
> > > exec-time=242hrs, 10mins, 28.896sec
> > > exception=org.apache.hadoop.hbase.TableNotEnabledException:
> > sentinel-meta
> > >
> > > Note the procedure Id was around 43000, far lower than 147464.
> > >
> > > Can you turn debug log on and repost master log ?
> > >
> > > Thanks
> > >
> > > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Ad...@hitec.lu>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > >
> > > > We're having a table stuck in disabled state. First I'd like to start
> > > with
> > > > what I tried already:
> > > >
> > > >
> > > > * Restart all machines involved in HBase cluster
> > > >
> > > > * hbase hbck with varios arguments
> > > >
> > > > * hdfs fsck
> > > >
> > > > * Purge ZK /hbase and restart masters
> > > >
> > > >
> > > > Now more details anout what happens:
> > > >
> > > > * When enabling from hbase shell:
> > > >
> > > >
> > > > hbase(main):002:0> enable "sentinel-meta"
> > > > ERROR: The procedure 147464 is still running
> > > >
> > > >
> > > > The task ID changes every time I run the command so I think it's
> > talking
> > > > about itself (and it gets stuck for a while before saying anything)
> > > >
> > > >
> > > > In the log, all I can see is:
> > > >
> > > > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> > DisableTableProcedure
> > > > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> > > > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.
> hbase.
> > > TableNotEnabledException:
> > > > sentinel-meta
> > > > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
> > > DisableTableProcedure:
> > > > Table sentinel-meta isn't enabled; skipping disable
> > > > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> > DisableTableProcedure
> > > > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> > > > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.
> hbase.
> > > TableNotEnabledException:
> > > > sentinel-meta
> > > > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
> > > DisableTableProcedure:
> > > > Table sentinel-meta isn't enabled; skipping disable
> > > >
> > > >
> > > > Please also find attached a complete log from startup to shutdown on
> a
> > > > single active master. You'll see the table is found as well as the
> > > regions
> > > > but it gets deactivated with no reason.
> > > >
> > > >
> > > > Thanks a lot for your help, we're kinda running out if ideas here.
> > > >
> > > >
> > > > Best regards,
> > > >
> > > >
> > > > Adam.
> > > >
> > > >
> > > >
> > >
> >
>
RE: Table is disabled an no way to get it back online
Posted by "Cecile, Adam" <Ad...@hitec.lu>.
Thanks for everything. As you said, this bug is supposed to be fixed in 1.2.0. Matteo is reading this list as well ?
Regards, Adam.
________________________________________
De : Ted Yu <yu...@gmail.com>
Envoyé : mardi 22 novembre 2016 19:00
À : user@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online
Please take a look at HBASE-13415
From the log, you're using hbase 1.2.0 already. But I heard there is a
subtle bug which is being fixed.
Matteo is the person with best knowledge in this regard.
On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <Ad...@hitec.lu> wrote:
> Another one, because I'm not sure the log is overwritten when restarting.
> This one has been cleared before service start.
> ________________________________________
> De : Cecile, Adam <Ad...@hitec.lu>
> Envoyé : mardi 22 novembre 2016 18:42
> À : user@hbase.apache.org
> Objet : RE: Table is disabled an no way to get it back online
>
> Hello,
>
> Sadly I could not use the webui, it killed my firefox (probably way too
> much time). Here is the debug log... (11Mb uncompressed for maybe two
> minutes running !!)
>
> Best regards, Adam.
> ________________________________________
> De : Ted Yu <yu...@gmail.com>
> Envoyé : mardi 22 novembre 2016 17:05
> À : user@hbase.apache.org
> Objet : Re: Table is disabled an no way to get it back online
>
> In log4j.properties :
>
> log4j.logger.org.apache.hadoop.hbase=DEBUG
>
> On master UI, you can select the Procedures tab. Pastebin what you see
> (text is enough).
>
> Thanks
>
> On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <Ad...@hitec.lu>
> wrote:
>
> > Hey Ted,
> >
> > Thank you. Heading home right now but I'll start the laptop again. Not
> > sure exactly how I should turn debug log on so if you have the
> information
> > it'd be appreciated, otherwise I'll look at the xml files.
> >
> > Regards, Adam.
> > ________________________________________
> > De : Ted Yu <yu...@gmail.com>
> > Envoyé : mardi 22 novembre 2016 15:46
> > À : user@hbase.apache.org
> > Objet : Re: Table is disabled an no way to get it back online
> >
> > Master log contained entries in the following form:
> >
> > 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
> > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> > (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
> > exec-time=242hrs, 10mins, 28.896sec
> > exception=org.apache.hadoop.hbase.TableNotEnabledException:
> sentinel-meta
> >
> > Note the procedure Id was around 43000, far lower than 147464.
> >
> > Can you turn debug log on and repost master log ?
> >
> > Thanks
> >
> > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Ad...@hitec.lu>
> > wrote:
> >
> > > Hello,
> > >
> > >
> > > We're having a table stuck in disabled state. First I'd like to start
> > with
> > > what I tried already:
> > >
> > >
> > > * Restart all machines involved in HBase cluster
> > >
> > > * hbase hbck with varios arguments
> > >
> > > * hdfs fsck
> > >
> > > * Purge ZK /hbase and restart masters
> > >
> > >
> > > Now more details anout what happens:
> > >
> > > * When enabling from hbase shell:
> > >
> > >
> > > hbase(main):002:0> enable "sentinel-meta"
> > > ERROR: The procedure 147464 is still running
> > >
> > >
> > > The task ID changes every time I run the command so I think it's
> talking
> > > about itself (and it gets stuck for a while before saying anything)
> > >
> > >
> > > In the log, all I can see is:
> > >
> > > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
> > > procedure2.ProcedureExecutor: Rolledback procedure
> DisableTableProcedure
> > > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> > > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.
> > TableNotEnabledException:
> > > sentinel-meta
> > > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
> > DisableTableProcedure:
> > > Table sentinel-meta isn't enabled; skipping disable
> > > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
> > > procedure2.ProcedureExecutor: Rolledback procedure
> DisableTableProcedure
> > > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> > > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.
> > TableNotEnabledException:
> > > sentinel-meta
> > > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
> > DisableTableProcedure:
> > > Table sentinel-meta isn't enabled; skipping disable
> > >
> > >
> > > Please also find attached a complete log from startup to shutdown on a
> > > single active master. You'll see the table is found as well as the
> > regions
> > > but it gets deactivated with no reason.
> > >
> > >
> > > Thanks a lot for your help, we're kinda running out if ideas here.
> > >
> > >
> > > Best regards,
> > >
> > >
> > > Adam.
> > >
> > >
> > >
> >
>
Re: Table is disabled an no way to get it back online
Posted by Ted Yu <yu...@gmail.com>.
Please take a look at HBASE-13415
From the log, you're using hbase 1.2.0 already. But I heard there is a
subtle bug which is being fixed.
Matteo is the person with best knowledge in this regard.
On Tue, Nov 22, 2016 at 9:48 AM, Cecile, Adam <Ad...@hitec.lu> wrote:
> Another one, because I'm not sure the log is overwritten when restarting.
> This one has been cleared before service start.
> ________________________________________
> De : Cecile, Adam <Ad...@hitec.lu>
> Envoyé : mardi 22 novembre 2016 18:42
> À : user@hbase.apache.org
> Objet : RE: Table is disabled an no way to get it back online
>
> Hello,
>
> Sadly I could not use the webui, it killed my firefox (probably way too
> much time). Here is the debug log... (11Mb uncompressed for maybe two
> minutes running !!)
>
> Best regards, Adam.
> ________________________________________
> De : Ted Yu <yu...@gmail.com>
> Envoyé : mardi 22 novembre 2016 17:05
> À : user@hbase.apache.org
> Objet : Re: Table is disabled an no way to get it back online
>
> In log4j.properties :
>
> log4j.logger.org.apache.hadoop.hbase=DEBUG
>
> On master UI, you can select the Procedures tab. Pastebin what you see
> (text is enough).
>
> Thanks
>
> On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <Ad...@hitec.lu>
> wrote:
>
> > Hey Ted,
> >
> > Thank you. Heading home right now but I'll start the laptop again. Not
> > sure exactly how I should turn debug log on so if you have the
> information
> > it'd be appreciated, otherwise I'll look at the xml files.
> >
> > Regards, Adam.
> > ________________________________________
> > De : Ted Yu <yu...@gmail.com>
> > Envoyé : mardi 22 novembre 2016 15:46
> > À : user@hbase.apache.org
> > Objet : Re: Table is disabled an no way to get it back online
> >
> > Master log contained entries in the following form:
> >
> > 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
> > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> > (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
> > exec-time=242hrs, 10mins, 28.896sec
> > exception=org.apache.hadoop.hbase.TableNotEnabledException:
> sentinel-meta
> >
> > Note the procedure Id was around 43000, far lower than 147464.
> >
> > Can you turn debug log on and repost master log ?
> >
> > Thanks
> >
> > On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Ad...@hitec.lu>
> > wrote:
> >
> > > Hello,
> > >
> > >
> > > We're having a table stuck in disabled state. First I'd like to start
> > with
> > > what I tried already:
> > >
> > >
> > > * Restart all machines involved in HBase cluster
> > >
> > > * hbase hbck with varios arguments
> > >
> > > * hdfs fsck
> > >
> > > * Purge ZK /hbase and restart masters
> > >
> > >
> > > Now more details anout what happens:
> > >
> > > * When enabling from hbase shell:
> > >
> > >
> > > hbase(main):002:0> enable "sentinel-meta"
> > > ERROR: The procedure 147464 is still running
> > >
> > >
> > > The task ID changes every time I run the command so I think it's
> talking
> > > about itself (and it gets stuck for a while before saying anything)
> > >
> > >
> > > In the log, all I can see is:
> > >
> > > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
> > > procedure2.ProcedureExecutor: Rolledback procedure
> DisableTableProcedure
> > > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> > > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.
> > TableNotEnabledException:
> > > sentinel-meta
> > > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
> > DisableTableProcedure:
> > > Table sentinel-meta isn't enabled; skipping disable
> > > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
> > > procedure2.ProcedureExecutor: Rolledback procedure
> DisableTableProcedure
> > > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> > > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.
> > TableNotEnabledException:
> > > sentinel-meta
> > > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
> > DisableTableProcedure:
> > > Table sentinel-meta isn't enabled; skipping disable
> > >
> > >
> > > Please also find attached a complete log from startup to shutdown on a
> > > single active master. You'll see the table is found as well as the
> > regions
> > > but it gets deactivated with no reason.
> > >
> > >
> > > Thanks a lot for your help, we're kinda running out if ideas here.
> > >
> > >
> > > Best regards,
> > >
> > >
> > > Adam.
> > >
> > >
> > >
> >
>
RE: Table is disabled an no way to get it back online
Posted by "Cecile, Adam" <Ad...@hitec.lu>.
Another one, because I'm not sure the log is overwritten when restarting. This one has been cleared before service start.
________________________________________
De : Cecile, Adam <Ad...@hitec.lu>
Envoyé : mardi 22 novembre 2016 18:42
À : user@hbase.apache.org
Objet : RE: Table is disabled an no way to get it back online
Hello,
Sadly I could not use the webui, it killed my firefox (probably way too much time). Here is the debug log... (11Mb uncompressed for maybe two minutes running !!)
Best regards, Adam.
________________________________________
De : Ted Yu <yu...@gmail.com>
Envoyé : mardi 22 novembre 2016 17:05
À : user@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online
In log4j.properties :
log4j.logger.org.apache.hadoop.hbase=DEBUG
On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).
Thanks
On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <Ad...@hitec.lu> wrote:
> Hey Ted,
>
> Thank you. Heading home right now but I'll start the laptop again. Not
> sure exactly how I should turn debug log on so if you have the information
> it'd be appreciated, otherwise I'll look at the xml files.
>
> Regards, Adam.
> ________________________________________
> De : Ted Yu <yu...@gmail.com>
> Envoyé : mardi 22 novembre 2016 15:46
> À : user@hbase.apache.org
> Objet : Re: Table is disabled an no way to get it back online
>
> Master log contained entries in the following form:
>
> 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
> procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
> exec-time=242hrs, 10mins, 28.896sec
> exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
>
> Note the procedure Id was around 43000, far lower than 147464.
>
> Can you turn debug log on and repost master log ?
>
> Thanks
>
> On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Ad...@hitec.lu>
> wrote:
>
> > Hello,
> >
> >
> > We're having a table stuck in disabled state. First I'd like to start
> with
> > what I tried already:
> >
> >
> > * Restart all machines involved in HBase cluster
> >
> > * hbase hbck with varios arguments
> >
> > * hdfs fsck
> >
> > * Purge ZK /hbase and restart masters
> >
> >
> > Now more details anout what happens:
> >
> > * When enabling from hbase shell:
> >
> >
> > hbase(main):002:0> enable "sentinel-meta"
> > ERROR: The procedure 147464 is still running
> >
> >
> > The task ID changes every time I run the command so I think it's talking
> > about itself (and it gets stuck for a while before saying anything)
> >
> >
> > In the log, all I can see is:
> >
> > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
> > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.
> TableNotEnabledException:
> > sentinel-meta
> > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
> DisableTableProcedure:
> > Table sentinel-meta isn't enabled; skipping disable
> > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
> > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.
> TableNotEnabledException:
> > sentinel-meta
> > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
> DisableTableProcedure:
> > Table sentinel-meta isn't enabled; skipping disable
> >
> >
> > Please also find attached a complete log from startup to shutdown on a
> > single active master. You'll see the table is found as well as the
> regions
> > but it gets deactivated with no reason.
> >
> >
> > Thanks a lot for your help, we're kinda running out if ideas here.
> >
> >
> > Best regards,
> >
> >
> > Adam.
> >
> >
> >
>
RE: Table is disabled an no way to get it back online
Posted by "Cecile, Adam" <Ad...@hitec.lu>.
Hello,
Sadly I could not use the webui, it killed my firefox (probably way too much time). Here is the debug log... (11Mb uncompressed for maybe two minutes running !!)
Best regards, Adam.
________________________________________
De : Ted Yu <yu...@gmail.com>
Envoyé : mardi 22 novembre 2016 17:05
À : user@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online
In log4j.properties :
log4j.logger.org.apache.hadoop.hbase=DEBUG
On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).
Thanks
On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <Ad...@hitec.lu> wrote:
> Hey Ted,
>
> Thank you. Heading home right now but I'll start the laptop again. Not
> sure exactly how I should turn debug log on so if you have the information
> it'd be appreciated, otherwise I'll look at the xml files.
>
> Regards, Adam.
> ________________________________________
> De : Ted Yu <yu...@gmail.com>
> Envoyé : mardi 22 novembre 2016 15:46
> À : user@hbase.apache.org
> Objet : Re: Table is disabled an no way to get it back online
>
> Master log contained entries in the following form:
>
> 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
> procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
> exec-time=242hrs, 10mins, 28.896sec
> exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
>
> Note the procedure Id was around 43000, far lower than 147464.
>
> Can you turn debug log on and repost master log ?
>
> Thanks
>
> On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Ad...@hitec.lu>
> wrote:
>
> > Hello,
> >
> >
> > We're having a table stuck in disabled state. First I'd like to start
> with
> > what I tried already:
> >
> >
> > * Restart all machines involved in HBase cluster
> >
> > * hbase hbck with varios arguments
> >
> > * hdfs fsck
> >
> > * Purge ZK /hbase and restart masters
> >
> >
> > Now more details anout what happens:
> >
> > * When enabling from hbase shell:
> >
> >
> > hbase(main):002:0> enable "sentinel-meta"
> > ERROR: The procedure 147464 is still running
> >
> >
> > The task ID changes every time I run the command so I think it's talking
> > about itself (and it gets stuck for a while before saying anything)
> >
> >
> > In the log, all I can see is:
> >
> > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
> > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.
> TableNotEnabledException:
> > sentinel-meta
> > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
> DisableTableProcedure:
> > Table sentinel-meta isn't enabled; skipping disable
> > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
> > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.
> TableNotEnabledException:
> > sentinel-meta
> > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
> DisableTableProcedure:
> > Table sentinel-meta isn't enabled; skipping disable
> >
> >
> > Please also find attached a complete log from startup to shutdown on a
> > single active master. You'll see the table is found as well as the
> regions
> > but it gets deactivated with no reason.
> >
> >
> > Thanks a lot for your help, we're kinda running out if ideas here.
> >
> >
> > Best regards,
> >
> >
> > Adam.
> >
> >
> >
>
Re: Table is disabled an no way to get it back online
Posted by Ted Yu <yu...@gmail.com>.
In log4j.properties :
log4j.logger.org.apache.hadoop.hbase=DEBUG
On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).
Thanks
On Tue, Nov 22, 2016 at 7:16 AM, Cecile, Adam <Ad...@hitec.lu> wrote:
> Hey Ted,
>
> Thank you. Heading home right now but I'll start the laptop again. Not
> sure exactly how I should turn debug log on so if you have the information
> it'd be appreciated, otherwise I'll look at the xml files.
>
> Regards, Adam.
> ________________________________________
> De : Ted Yu <yu...@gmail.com>
> Envoyé : mardi 22 novembre 2016 15:46
> À : user@hbase.apache.org
> Objet : Re: Table is disabled an no way to get it back online
>
> Master log contained entries in the following form:
>
> 2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
> procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> (table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
> exec-time=242hrs, 10mins, 28.896sec
> exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
>
> Note the procedure Id was around 43000, far lower than 147464.
>
> Can you turn debug log on and repost master log ?
>
> Thanks
>
> On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Ad...@hitec.lu>
> wrote:
>
> > Hello,
> >
> >
> > We're having a table stuck in disabled state. First I'd like to start
> with
> > what I tried already:
> >
> >
> > * Restart all machines involved in HBase cluster
> >
> > * hbase hbck with varios arguments
> >
> > * hdfs fsck
> >
> > * Purge ZK /hbase and restart masters
> >
> >
> > Now more details anout what happens:
> >
> > * When enabling from hbase shell:
> >
> >
> > hbase(main):002:0> enable "sentinel-meta"
> > ERROR: The procedure 147464 is still running
> >
> >
> > The task ID changes every time I run the command so I think it's talking
> > about itself (and it gets stuck for a while before saying anything)
> >
> >
> > In the log, all I can see is:
> >
> > 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
> > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> > (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> > exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.
> TableNotEnabledException:
> > sentinel-meta
> > 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
> DisableTableProcedure:
> > Table sentinel-meta isn't enabled; skipping disable
> > 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
> > procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> > (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> > exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.
> TableNotEnabledException:
> > sentinel-meta
> > 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
> DisableTableProcedure:
> > Table sentinel-meta isn't enabled; skipping disable
> >
> >
> > Please also find attached a complete log from startup to shutdown on a
> > single active master. You'll see the table is found as well as the
> regions
> > but it gets deactivated with no reason.
> >
> >
> > Thanks a lot for your help, we're kinda running out if ideas here.
> >
> >
> > Best regards,
> >
> >
> > Adam.
> >
> >
> >
>
RE: Table is disabled an no way to get it back online
Posted by "Cecile, Adam" <Ad...@hitec.lu>.
Hey Ted,
Thank you. Heading home right now but I'll start the laptop again. Not sure exactly how I should turn debug log on so if you have the information it'd be appreciated, otherwise I'll look at the xml files.
Regards, Adam.
________________________________________
De : Ted Yu <yu...@gmail.com>
Envoyé : mardi 22 novembre 2016 15:46
À : user@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online
Master log contained entries in the following form:
2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
Note the procedure Id was around 43000, far lower than 147464.
Can you turn debug log on and repost master log ?
Thanks
On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Ad...@hitec.lu> wrote:
> Hello,
>
>
> We're having a table stuck in disabled state. First I'd like to start with
> what I tried already:
>
>
> * Restart all machines involved in HBase cluster
>
> * hbase hbck with varios arguments
>
> * hdfs fsck
>
> * Purge ZK /hbase and restart masters
>
>
> Now more details anout what happens:
>
> * When enabling from hbase shell:
>
>
> hbase(main):002:0> enable "sentinel-meta"
> ERROR: The procedure 147464 is still running
>
>
> The task ID changes every time I run the command so I think it's talking
> about itself (and it gets stuck for a while before saying anything)
>
>
> In the log, all I can see is:
>
> 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
> procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.TableNotEnabledException:
> sentinel-meta
> 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure:
> Table sentinel-meta isn't enabled; skipping disable
> 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
> procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.TableNotEnabledException:
> sentinel-meta
> 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure:
> Table sentinel-meta isn't enabled; skipping disable
>
>
> Please also find attached a complete log from startup to shutdown on a
> single active master. You'll see the table is found as well as the regions
> but it gets deactivated with no reason.
>
>
> Thanks a lot for your help, we're kinda running out if ideas here.
>
>
> Best regards,
>
>
> Adam.
>
>
>
Re: Table is disabled an no way to get it back online
Posted by Ted Yu <yu...@gmail.com>.
Master log contained entries in the following form:
2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
Note the procedure Id was around 43000, far lower than 147464.
Can you turn debug log on and repost master log ?
Thanks
On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Ad...@hitec.lu> wrote:
> Hello,
>
>
> We're having a table stuck in disabled state. First I'd like to start with
> what I tried already:
>
>
> * Restart all machines involved in HBase cluster
>
> * hbase hbck with varios arguments
>
> * hdfs fsck
>
> * Purge ZK /hbase and restart masters
>
>
> Now more details anout what happens:
>
> * When enabling from hbase shell:
>
>
> hbase(main):002:0> enable "sentinel-meta"
> ERROR: The procedure 147464 is still running
>
>
> The task ID changes every time I run the command so I think it's talking
> about itself (and it gets stuck for a while before saying anything)
>
>
> In the log, all I can see is:
>
> 2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
> procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.TableNotEnabledException:
> sentinel-meta
> 2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure:
> Table sentinel-meta isn't enabled; skipping disable
> 2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
> procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.TableNotEnabledException:
> sentinel-meta
> 2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure:
> Table sentinel-meta isn't enabled; skipping disable
>
>
> Please also find attached a complete log from startup to shutdown on a
> single active master. You'll see the table is found as well as the regions
> but it gets deactivated with no reason.
>
>
> Thanks a lot for your help, we're kinda running out if ideas here.
>
>
> Best regards,
>
>
> Adam.
>
>
>