HBase Cannot Start Again After Sudden Power-Off

[Bing Li <lb...@gmail.com>]

Dear all,

I have used HBase in the pseudo-distributed mode for four months. Last
night a sudden power-off shut down my HBase machine abnormally. Thereafter,
HBase could be started.

I had to reinstall my HBase again. This is a very bad experience. It is
fortunate that my system is in a development stage. If it works for a
business, that must be a disaster.

I guess the problem is caused because some configuration data is written in
/tmp. I really DON'T understand why the designers of Hadoop/HBase did that!
I have to change a lot of configurations to avoid configuration data loss
in /tmp.

After reinstallation, HBase can be started normally. Then, I shut down
HBase/Hadoop and turned the machine off.

But, today I got another problem. When starting the system, HBase could not
be started again! I checked the files in /tmp and noticed that HBase wrote
some Jetty-related files in it.

   ...
   drwxr-xr-x  4 libing libing 4096 2012-07-22 16:27
Jetty_0_0_0_0_50070_hdfs____w2cu08
   drwxr-xr-x  4 libing libing 4096 2012-07-22 16:27
Jetty_0_0_0_0_50090_secondary____y6aanv
   drwxr-xr-x  4 libing libing 4096 2012-07-22 16:27
Jetty_0_0_0_0_50075_datanode____hwtdwq
   ...

I didn't see them before. I guess that's the problem why HBase cannot be
started? If so, could you please tell me how to force HBase to write the
Jetty-related files to the directories other than /tmp?

Your help is highly appreciated!

Best regards,
Bing

Re: HBase Cannot Start Again After Sudden Power-Off

[Minh Duc Nguyen <md...@gmail.com>]

Bing, as far as why HBase is configured as such, it's by design.  The HBase
documentation is very clear in explaining that pseudo-distributed mode is
for "configuration testing and prototyping."

http://hbase.apache.org/book.html#standalone_dist

In order to make it as easy as possible for people to get up and running
with HBase, the default configuration for HBase in pseudo-distributed mode
is to use /tmp.  Of course, when linux restarts, it will clear out the /tmp
directory.  Since you've been running HBase for 4 months now, it may be
worth your while to setup a small HBase cluster in fully-distributed mode.
 While convenient, HBase in pseudo-distributed mode will not be as reliable
as fully-distributed.

   ~ Minh

On Sun, Jul 22, 2012 at 5:29 AM, Bing Li <lb...@gmail.com> wrote:

> Dear all,
>
> I have used HBase in the pseudo-distributed mode for four months. Last
> night a sudden power-off shut down my HBase machine abnormally. Thereafter,
> HBase could be started.
>
> I had to reinstall my HBase again. This is a very bad experience. It is
> fortunate that my system is in a development stage. If it works for a
> business, that must be a disaster.
>
> I guess the problem is caused because some configuration data is written in
> /tmp. I really DON'T understand why the designers of Hadoop/HBase did that!
> I have to change a lot of configurations to avoid configuration data loss
> in /tmp.
>
> After reinstallation, HBase can be started normally. Then, I shut down
> HBase/Hadoop and turned the machine off.
>
> But, today I got another problem. When starting the system, HBase could not
> be started again! I checked the files in /tmp and noticed that HBase wrote
> some Jetty-related files in it.
>
>    ...
>    drwxr-xr-x  4 libing libing 4096 2012-07-22 16:27
> Jetty_0_0_0_0_50070_hdfs____w2cu08
>    drwxr-xr-x  4 libing libing 4096 2012-07-22 16:27
> Jetty_0_0_0_0_50090_secondary____y6aanv
>    drwxr-xr-x  4 libing libing 4096 2012-07-22 16:27
> Jetty_0_0_0_0_50075_datanode____hwtdwq
>    ...
>
> I didn't see them before. I guess that's the problem why HBase cannot be
> started? If so, could you please tell me how to force HBase to write the
> Jetty-related files to the directories other than /tmp?
>
> Your help is highly appreciated!
>
> Best regards,
> Bing
>

Re: HBase Cannot Start Again After Sudden Power-Off

[Alok Kumar <al...@gmail.com>]

hi,

you should try looking into log dir ( $HBASE_HOME/logs ), look for HMaster
logs in "hbase-<$user.name>-master-<domain>.log files.

It'll tell you exact region why HBase is not coming up.. or try cleaning
/tmp directory..

cheers,
Alok


On Sun, Jul 22, 2012 at 2:59 PM, Bing Li <lb...@gmail.com> wrote:

> Dear all,
>
> I have used HBase in the pseudo-distributed mode for four months. Last
> night a sudden power-off shut down my HBase machine abnormally. Thereafter,
> HBase could be started.
>
> I had to reinstall my HBase again. This is a very bad experience. It is
> fortunate that my system is in a development stage. If it works for a
> business, that must be a disaster.
>
> I guess the problem is caused because some configuration data is written in
> /tmp. I really DON'T understand why the designers of Hadoop/HBase did that!
> I have to change a lot of configurations to avoid configuration data loss
> in /tmp.
>
> After reinstallation, HBase can be started normally. Then, I shut down
> HBase/Hadoop and turned the machine off.
>
> But, today I got another problem. When starting the system, HBase could not
> be started again! I checked the files in /tmp and noticed that HBase wrote
> some Jetty-related files in it.
>
>    ...
>    drwxr-xr-x  4 libing libing 4096 2012-07-22 16:27
> Jetty_0_0_0_0_50070_hdfs____w2cu08
>    drwxr-xr-x  4 libing libing 4096 2012-07-22 16:27
> Jetty_0_0_0_0_50090_secondary____y6aanv
>    drwxr-xr-x  4 libing libing 4096 2012-07-22 16:27
> Jetty_0_0_0_0_50075_datanode____hwtdwq
>    ...
>
> I didn't see them before. I guess that's the problem why HBase cannot be
> started? If so, could you please tell me how to force HBase to write the
> Jetty-related files to the directories other than /tmp?
>
> Your help is highly appreciated!
>
> Best regards,
> Bing
>