NoServerForRegionException, TableNotFoundException and WrongRegionException

[Marc Limotte <ms...@gmail.com>]

I'm seeing a nice variety of Exceptions from HBase and could use some
pointers about what to do next.

This is a new map/reduce program, updating about 550k rows with around a
dozen columns on a very small cluster (only 4 nodes... as we're still
testing and it doesn't have to support production yet).  Hbase Version
0.19.1.

I ran the job and it seems to make some progress, and then dies after
several hours, reporting "NoServerForRegionException: No server address
listed in .META. for region TABLEX,,1250526695078".  I retried it a few
times with the same result.  I also noticed that the load is not well
balanced, all requests seemed to be going to one node.  I adjust
hadoop-site.xml with the addition of these two entries:

    <name>hbase.hregion.max.filesize</name>
    <value>33554432</value>

   <name>hbase.client.retries.number</name>
    <value>5</value>

And restarted hbase (and hadoop to be safe).  Re-ran and got the same error
in the M/R job.

*I thought I'd try dropping the table, since it's a new table and I can
recreate it.  But that gives another exception:
*
hbase(main):002:0> disable 'TABLEX'
NativeException: org.apache.hadoop.hbase.TableNotFoundException:
org.apache.hadoop.hbase.TableNotFoundException: TABLEX
    at
org.apache.hadoop.hbase.master.TableOperation$ProcessTableOperation.call(TableOperation.java:129)
    at
org.apache.hadoop.hbase.master.TableOperation$ProcessTableOperation.call(TableOperation.java:70)
    at
org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:64)
    at
org.apache.hadoop.hbase.master.TableOperation.process(TableOperation.java:143)
    at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:691)
...


*And now I see this exception in the HBase logs:
*
org.apache.hadoop.hbase.regionserver.WrongRegionException:
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
of range for HRegion .META.,,1250280235390, startKey='',
getEndKey()='TABLEX,,1250219949252',
row='TABLEX,840.56098.0544,1250526661861'
    at
org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1788)
    at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1844)
    at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1912)
    at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1244)
    at
org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1216)
...


*As a test, tried a "count"...
*
hbase(main):007:0* count 'TABLEX'
NativeException: org.apache.hadoop.hbase.client.NoServerForRegionException:
No server address listed in .META. for region TABLEX,,1250526695078
    from org/apache/hadoop/hbase/client/HConnectionManager.java:548:in
`locateRegionInMeta'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:478:in
`locateRegion'
    from org/apache/hadoop/hbase/client/HConnectionManager.java:440:in
`locateRegion'
    from org/apache/hadoop/hbase/client/HTable.java:114:in `<init>'
    from org/apache/hadoop/hbase/client/HTable.java:97:in `<init>'
    from sun/reflect/NativeConstructorAccessorImpl.java:-2:in `newInstance0'
...


*Also saw a thread somewhere that suggested doing a major compaction.  Did
that.  It returns almost immediately.  Not sure if that's normal or not...
no perceivable impact from doing this, though.*

hbase(main):013:0> major_compact '.META.'
0 row(s) in 0.0220 seconds
hbase(main):014:0>

Not sure what else to try?  Is there a way to force removal of the table in
question?  Is there something else I should be looking at?

Marc

Re: NoServerForRegionException, TableNotFoundException and WrongRegionException

[stack <st...@duboce.net>]

Please update to the head of 0.19 trunk, or better update to 0.20 trunk --
espeically if you are testing.  Issues described below have been addressed.

How many regions do you have in your table?  Are all going to one
regionserver because you only have one region?

Yours,
St.Ack


On Mon, Aug 17, 2009 at 12:19 PM, Marc Limotte <ms...@gmail.com> wrote:

> I'm seeing a nice variety of Exceptions from HBase and could use some
> pointers about what to do next.
>
> This is a new map/reduce program, updating about 550k rows with around a
> dozen columns on a very small cluster (only 4 nodes... as we're still
> testing and it doesn't have to support production yet).  Hbase Version
> 0.19.1.
>
> I ran the job and it seems to make some progress, and then dies after
> several hours, reporting "NoServerForRegionException: No server address
> listed in .META. for region TABLEX,,1250526695078".  I retried it a few
> times with the same result.  I also noticed that the load is not well
> balanced, all requests seemed to be going to one node.  I adjust
> hadoop-site.xml with the addition of these two entries:
>
>    <name>hbase.hregion.max.filesize</name>
>    <value>33554432</value>
>
>   <name>hbase.client.retries.number</name>
>    <value>5</value>
>
> And restarted hbase (and hadoop to be safe).  Re-ran and got the same error
> in the M/R job.
>
> *I thought I'd try dropping the table, since it's a new table and I can
> recreate it.  But that gives another exception:
> *
> hbase(main):002:0> disable 'TABLEX'
> NativeException: org.apache.hadoop.hbase.TableNotFoundException:
> org.apache.hadoop.hbase.TableNotFoundException: TABLEX
>    at
>
> org.apache.hadoop.hbase.master.TableOperation$ProcessTableOperation.call(TableOperation.java:129)
>    at
>
> org.apache.hadoop.hbase.master.TableOperation$ProcessTableOperation.call(TableOperation.java:70)
>    at
>
> org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:64)
>    at
>
> org.apache.hadoop.hbase.master.TableOperation.process(TableOperation.java:143)
>    at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:691)
> ...
>
>
> *And now I see this exception in the HBase logs:
> *
> org.apache.hadoop.hbase.regionserver.WrongRegionException:
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row
> out
> of range for HRegion .META.,,1250280235390, startKey='',
> getEndKey()='TABLEX,,1250219949252',
> row='TABLEX,840.56098.0544,1250526661861'
>    at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1788)
>    at
>
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1844)
>    at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1912)
>    at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1244)
>    at
> org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1216)
> ...
>
>
> *As a test, tried a "count"...
> *
> hbase(main):007:0* count 'TABLEX'
> NativeException: org.apache.hadoop.hbase.client.NoServerForRegionException:
> No server address listed in .META. for region TABLEX,,1250526695078
>    from org/apache/hadoop/hbase/client/HConnectionManager.java:548:in
> `locateRegionInMeta'
>    from org/apache/hadoop/hbase/client/HConnectionManager.java:478:in
> `locateRegion'
>    from org/apache/hadoop/hbase/client/HConnectionManager.java:440:in
> `locateRegion'
>    from org/apache/hadoop/hbase/client/HTable.java:114:in `<init>'
>    from org/apache/hadoop/hbase/client/HTable.java:97:in `<init>'
>    from sun/reflect/NativeConstructorAccessorImpl.java:-2:in `newInstance0'
> ...
>
>
> *Also saw a thread somewhere that suggested doing a major compaction.  Did
> that.  It returns almost immediately.  Not sure if that's normal or not...
> no perceivable impact from doing this, though.*
>
> hbase(main):013:0> major_compact '.META.'
> 0 row(s) in 0.0220 seconds
> hbase(main):014:0>
>
> Not sure what else to try?  Is there a way to force removal of the table in
> question?  Is there something else I should be looking at?
>
> Marc
>