Re: trunk code, RetriesExhaustedException

[Xinan Wu <wu...@gmail.com>]

follow up on this issue..

this is probably caused by an earlier Hmaster kill(possibly with -9
signal) so that a lot of entries are left in .META. table even if i
removed all tables and did a major_compact on .META. table...  sort of
related to HBASE-1383. With this weird .META. table, sometimes rpc
call fails at HConnectionManager.java locateRegionInMeta(...)
server.getClosestRowBefore(...) returning null with perfect legitimate
regionname and metakey.

solved by completely recreating hbase directory in hdfs.

Xinan


On Fri, May 29, 2009 at 7:11 PM, Xinan Wu <wu...@gmail.com> wrote:
> Hi,
>
> I started to experiment hbase-0.20 trunk code today and imported about
> 6GB data using native JAVA API. I occationally get
> RetriesExhaustedException exception during importing and it's usually
> fixable by restarting everything and resume importing from where it
> stopped last time. But now I am in a state that I can reproduce this
> exception repeatably, even after I completely restart hbase master and
> hdfs master (!)....
>
> After fresh new start, in hbase shell, i can scan '-ROOT-' and
> '.META.', but retrieving value (get) from any other table results
> RetriesExhaustedException exception:
>
> NativeException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> contact region server null for region , row '2', but failed after 5
> attempts.
> Exceptions:
> org.apache.hadoop.hbase.TableNotFoundException: myTable
> org.apache.hadoop.hbase.TableNotFoundException: myTable
> org.apache.hadoop.hbase.TableNotFoundException: myTable
> org.apache.hadoop.hbase.TableNotFoundException: myTable
> org.apache.hadoop.hbase.TableNotFoundException: myTable
>
>        from org/apache/hadoop/hbase/client/HConnectionManager.java:922:in
> `getRegionServerWithRetries'
>        from org/apache/hadoop/hbase/client/HTable.java:632:in `getRow'
> ...
>
> issuing 'scan' command of the table just hangs.
>
> Anyone has any idea where I should investigate? The hadoop
> namenode/datanode logs and hbase master/regionserver logs all look
> normal. My table scheme is very simple, single column, everything
> running on the same dev box.
>
> Thanks,
>