Re: RegionServer can't recover after a failure

[Naresh Rapolu <nr...@purdue.edu>]

Hello James,

Can you explain a bit more about the design for this global transaction 
log table ?  When is it supposed to be created and how does it solve the 
problem of missing COMMIT statement in the region server WAL, before 
dying ?

Thanks,
Naresh.

On 08/27/2010 07:26 PM, James Kennedy wrote:
> I can help. I'm a developer on the transactional hbase extension which you must be using.
>
> The issue is that the global transaction log table is not created yet. You can do so simply by calling
> HBaseBackedTransactionLogger.createTable() at the time when you are seeding the rest of your tables.
>
> I apologize that the extension as given in GitHub is not yet mature.  While it works (HBase 0.21) it is poorly documented and needs more thorough testing.
>
> We have recently updated it to work with HBase 0.89.20100726 and it is much more stable and very slightly better documented. We are waiting for an HBase patch submission before we push it to hbase-trx at github.
>
> Thanks,
>
> James Kennedy
> Project Manager
> Troove Inc.
>
> On 2010-08-26, at 8:16 AM, Andrey Timerbaev wrote:
>
>    
>> Dear experts,
>>
>> Could you kindly suggest, how to help the RegionServer to complete
>> initialization in the following situation:
>>
>> After a failure of one or RegionServers, which is running on a dedicated node in
>> a HBase/Hadoop cluster (HBase v.0.20.3), the RegionServer can't initialize
>> available tables. The region server's log contains this exception:
>>
>> 2010-08-26 18:56:49,073 INFO
>> org.apache.hadoop.hbase.regionserver.transactional.THLogRecoveryManager: Region
>> log has 9 unfinished transactions. Going to the transaction log to resolve
>> 2010-08-26 18:56:49,091 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit for
>> row<>  in tableName .META.: location server 10.2.146.41:60020, location region
>> name .META.,,1
>> 2010-08-26 18:56:49,178 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening
>> STAT_STARTUPS_TABLE,,1282738349665
>> java.lang.RuntimeException: Table not created. Call createTable() first
>>         at
>> org.apache.hadoop.hbase.client.transactional.HBaseBackedTransactionLogger.
>> initTable(HBaseBackedTransactionLogger.java:76)
>>         at
>> org.apache.hadoop.hbase.client.transactional.HBaseBackedTransactionLogger.<init>
>> (HBaseBackedTransactionLogger.java:69)
>>         at
>> org.apache.hadoop.hbase.regionserver.transactional.THLogRecoveryManager.
>> getGlobalTransactionLog(THLogRecoveryManager.java:256)
>>         at
>> org.apache.hadoop.hbase.regionserver.transactional.THLogRecoveryManager.
>> resolvePendingTransaction(THLogRecoveryManager.java:225)
>>         at
>> org.apache.hadoop.hbase.regionserver.transactional.THLogRecoveryManager.
>> getCommitsFromLog(THLogRecoveryManager.java:206)
>>         at
>> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion.
>> doReconstructionLog(TransactionalRegion.java:145)
>>         at org.apache.hadoop.hbase.regionserver.HRegion.initialize
>> (HRegion.java:326)
>>         at
>> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegionServer.
>> instantiateRegion(TransactionalRegionServer.java:121)
>>         at
>> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion
>> (HRegionServer.java:1531)
>>         at
>> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run
>> (HRegionServer.java:1451)
>>         at java.lang.Thread.run(Thread.java:619)
>>
>> After a look into HBase source code I found out, that the "Table not created.
>> Call createTable()" message appears, if the HBaseBackedTransactionLogger is
>> unable to find the __GLOBAL_TRX_LOG__ table. But I've got no idea, where the
>> table should be, whether it is critical and what should I do in this situation.
>>
>> Any help is appreciated.
>>
>> Andrey
>>
>>
>>      
>
>