You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Dru Jensen <dr...@gmail.com> on 2008/09/29 22:16:47 UTC

hbase 0.2.1 failing to start

HBase was not responding to Thrift requests so I tried to restart but  
it still looks frozen.  I am seeing several error messages in the  
hmaster logs after I attempted to restart hbase:

2008-09-29 12:55:23,744 ERROR  
org.apache.hadoop.hbase.regionserver.HRegionServer: error opening  
region {table},{key},1222453917858
java.io.IOException: Premeture EOF from inputStream
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
	at org.apache.hadoop.dfs.DFSClient 
$BlockReader.readChunk(DFSClient.java:967)
	at  
org 
.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java: 
236)

and:

2008-09-29 12:58:50,067 ERROR  
org.apache.hadoop.hbase.regionserver.HRegionServer: error opening  
region {table},{key},1222453917858
java.io.IOException: Could not obtain block: blk_-2905695662732817278

Any ideas what would cause this? Can I recover this data?

HBase 0.2.1
Hadoop 0.17.2.1
Cluster of 3 boxes, dfs replication = 1.

Re: hbase 0.2.1 failing to start

Posted by stack <st...@duboce.net>.
Dru Jensen wrote:
> St.Ack.  Thanks for your response.
>
> I will enable DEBUG and rerun the MR processes to try and reproduce this.
> Hadoop is reporting everything is healthy using fsck.
> This is a test platform so the data is not critical but my confidence 
> is shaken (I sound like a day trader).
Understood.

> Questions:
> 1. Is there anything specific I should be looking for when I enable 
> DEBUG?
>
Thats a bit of a tough question.  Requires study to be able to 
interpret.  In short, look at lines before ERRORs and WARNINGs for 
anything that might explain why the ERROR or WARNING exception 
(NotServingRegionExceptions are part of 'normal' operation -- its the 
other exception-types you are interested in).

> 2. Does "bad" mean I cannot recover and i need to delete 
> /hbase.rootdir and start over?
>
Not if fsck says all is ok.  I said 'bad' because I thought you would 
have to do the above.

Any events on your cluster that might have effected HDFS?  A tsunami 
hit?  Or in your case, it wouldn't take much since replication was set 
to one -- did a host crash?

> 3. Does HBase depend on replication for normal operation?  In other 
> words, will it work without replication enabled?
>
It'll work fine without replication until you lose data.  Thereafter, 
it'll be hobbled by files with holes in them -- where the holes are 
blocks that sat on the downed server.

Would suggest running with replication of 3 unless you have very good 
reason -- and insurance against failure -- for doing otherwise.

Go easy Dru,
St.Ack

>
> On Sep 29, 2008, at 1:27 PM, stack wrote:
>
>> Dru Jensen wrote:
>>> HBase was not responding to Thrift requests so I tried to restart 
>>> but it still looks frozen.  I am seeing several error messages in 
>>> the hmaster logs after I attempted to restart hbase:
>>>
>>> 2008-09-29 12:55:23,744 ERROR 
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: error opening 
>>> region {table},{key},1222453917858
>>> java.io.IOException: Premeture EOF from inputStream
>>>    at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
>>>    at 
>>> org.apache.hadoop.dfs.DFSClient$BlockReader.readChunk(DFSClient.java:967) 
>>>
>>>    at 
>>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236) 
>>>
>>
>> Enable DEBUG and it might tell you what it was trying to open at time 
>> of the exception.
>>
>>>
>>> and:
>>>
>>> 2008-09-29 12:58:50,067 ERROR 
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: error opening 
>>> region {table},{key},1222453917858
>>> java.io.IOException: Could not obtain block: blk_-2905695662732817278
>>
>> This is bad.   What happens if you run './bin/hadoop fsck 
>> /hbase.rootdir'?  Your replication is one.  Means if any hdfs hiccup, 
>> data is lost.  You might putting replication back to the default?
>>
>
>
>> St.Ack
>>
>

Re: hbase 0.2.1 failing to start

Posted by Dru Jensen <dr...@gmail.com>.
St.Ack.  Thanks for your response.

I will enable DEBUG and rerun the MR processes to try and reproduce  
this.
Hadoop is reporting everything is healthy using fsck.
This is a test platform so the data is not critical but my confidence  
is shaken (I sound like a day trader).

Questions:
1. Is there anything specific I should be looking for when I enable  
DEBUG?

2. Does "bad" mean I cannot recover and i need to delete / 
hbase.rootdir and start over?

3. Does HBase depend on replication for normal operation?  In other  
words, will it work without replication enabled?
>


On Sep 29, 2008, at 1:27 PM, stack wrote:

> Dru Jensen wrote:
>> HBase was not responding to Thrift requests so I tried to restart  
>> but it still looks frozen.  I am seeing several error messages in  
>> the hmaster logs after I attempted to restart hbase:
>>
>> 2008-09-29 12:55:23,744 ERROR  
>> org.apache.hadoop.hbase.regionserver.HRegionServer: error opening  
>> region {table},{key},1222453917858
>> java.io.IOException: Premeture EOF from inputStream
>>    at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
>>    at org.apache.hadoop.dfs.DFSClient 
>> $BlockReader.readChunk(DFSClient.java:967)
>>    at  
>> org 
>> .apache 
>> .hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236)
>
> Enable DEBUG and it might tell you what it was trying to open at  
> time of the exception.
>
>>
>> and:
>>
>> 2008-09-29 12:58:50,067 ERROR  
>> org.apache.hadoop.hbase.regionserver.HRegionServer: error opening  
>> region {table},{key},1222453917858
>> java.io.IOException: Could not obtain block: blk_-2905695662732817278
>
> This is bad.   What happens if you run './bin/hadoop fsck / 
> hbase.rootdir'?  Your replication is one.  Means if any hdfs hiccup,  
> data is lost.  You might putting replication back to the default?
>


> St.Ack
>

Re: hbase 0.2.1 failing to start

Posted by stack <st...@duboce.net>.
Dru Jensen wrote:
> HBase was not responding to Thrift requests so I tried to restart but 
> it still looks frozen.  I am seeing several error messages in the 
> hmaster logs after I attempted to restart hbase:
>
> 2008-09-29 12:55:23,744 ERROR 
> org.apache.hadoop.hbase.regionserver.HRegionServer: error opening 
> region {table},{key},1222453917858
> java.io.IOException: Premeture EOF from inputStream
>     at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
>     at 
> org.apache.hadoop.dfs.DFSClient$BlockReader.readChunk(DFSClient.java:967)
>     at 
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236) 
>

Enable DEBUG and it might tell you what it was trying to open at time of 
the exception.

>
> and:
>
> 2008-09-29 12:58:50,067 ERROR 
> org.apache.hadoop.hbase.regionserver.HRegionServer: error opening 
> region {table},{key},1222453917858
> java.io.IOException: Could not obtain block: blk_-2905695662732817278

This is bad.   What happens if you run './bin/hadoop fsck 
/hbase.rootdir'?  Your replication is one.  Means if any hdfs hiccup, 
data is lost.  You might putting replication back to the default?

St.Ack