HBase recovery failuer problem

[KyuHwan Kim <ky...@gmail.com>]

Hi all.

I am testing on hadoop/hbase system.
I simply inserted into Hbase but there was some recovery problem.
The disk remaining space is small (1.2GB) and it is just used for
hbase data not log or others.

My test environment is:
Hadoop/Hbase version: 0.18.0
Data node (1), Name node (1)
Changed Hbase configuration:
    <property>
        <name>hbase.regionserver.class</name>
        <value>org.apache.hadoop.hbase.ipc.TransactionalRegionInterface</value>
    </property>
    <property>
        <name>hbase.regionserver.impl</name>
        <value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer</value>
    </property>

Here is the scenario.
1. Insert records into Hbase batch update (commit for each 1000 records)
2. While loading data, the following exception occurred in client side:
java.io.IOException: java.io.IOException: Could not get block
locations. Aborting...
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
3. After the client programs are terminated, I stopped Hbase and got
the error message:
2008-10-09 15:12:26,319 WARN org.apache.hadoop.dfs.DFSClient: Error
Recovery for block null bad datanode[0]
2008-10-09 15:12:26,320 FATAL
org.apache.hadoop.hbase.regionserver.Flusher: Replay of hlog required.
Forcing server restart
org.apache.hadoop.hbase.DroppedSnapshotException: region: -ROOT-,,0
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1071)
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:967)
        at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:172)
        at org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:90)
Caused by: java.io.IOException: Could not get block locations. Aborting...
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
4. After restart Hbase, the result of "count 'Table'" in hbase shell is 0.

It seems that there is some problem in recovery when the disk space is
not enough.
Is this known problem? How can I avoid this problem? except keeping
enough disk space?