You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by 吕鹏 <lv...@gmail.com> on 2011/10/20 16:07:55 UTC
Two kinds of hbase client exceptions
Hello everyone:
My hbase&hadoop is cdh3u0. To test the cluster, i use 10 client to write
date to hbase concurrently. I implement the client by hbase java api. When
the test run for several hours, the 10 clients are all crashed, and two
kinds of exceptions is catched. How can i solve the problem? (I
set hbase.hregion.max.filesize to 2G, and do not pre-sharding hbase table.
Can this cause the problem?)
Exception in thread "main"
org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
lzo10DB,\x89\xB7\x17\xD2\x08\xA9\x87\xFFg\xE2\x9Fxy8\xD0\x8A\r\xA7\x96\xB9\x9D|D\x8D\xE9\xB1\x8B\x94\xDF5\xA3\xDD\x86O\x7F#f%S\x93\xE0F\x1E|\x07\xF9Q\xF3\xD4,1319046253246.dbc7a6366cdd86019af7d56284dc5d77.
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:741)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:593)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1117)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1237)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:660)
at
com.xx.spider.wdb.test.HbaseRandomKeyWriter._doTest(HbaseRandomKeyWriter.java:31)
at
com.xx.spider.wdb.test.HbaseTestBase.doTest(HbaseTestBase.java:43)
at com.xx.spider.wdb.test.HbaseTestMain.main(HbaseTestMain.java:48)
Exception in thread "main"
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
1 action: NotServingRegionException: 1 time, servers with issues:
sds-179-36:60020,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1223)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1237)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:660)
at
com.xx.spider.wdb.test.HbaseRandomKeyWriter._doTest(HbaseRandomKeyWriter.java:31)
at
com.xx.spider.wdb.test.HbaseTestBase.doTest(HbaseTestBase.java:43)
at com.xx.spider.wdb.test.HbaseTestMain.main(HbaseTestMain.java:48)
Re: Two kinds of hbase client exceptions
Posted by 吕鹏 <lv...@gmail.com>.
Thank you very much for you help.
Before i build the hbase cluster, i've already had a cdh3u0 hadoop, so the
u0 hbase is preferred.
I have lookup the master's log, when the client exception is occurred, the
rs is sending msg to master for splitting, such as
INFO org.apache.hadoop.hbase.master.ServerManager: Received REGION_SPLIT
In my hbase schema, the rowkey is like "company name + person name". I can
not estimate how many companys there are and how many persons in each
company. So it difficult to presplit, but it is easy to scan all the person
in one company.
If i do not presplit, how can i solve client exceptions. Retry several
times if client exception is catched? And how many times are advisable? How
can i solve the exception friendly?
On Fri, Oct 21, 2011 at 12:01 AM, Stack <st...@duboce.net> wrote:
> On Thu, Oct 20, 2011 at 7:07 AM, 吕鹏 <lv...@gmail.com> wrote:
> > Hello everyone:
> > My hbase&hadoop is cdh3u0.
>
> Why not run u1?
>
> > To test the cluster, i use 10 client to write
> > date to hbase concurrently. I implement the client by hbase java api.
> When
> > the test run for several hours, the 10 clients are all crashed, and two
> > kinds of exceptions is catched. How can i solve the problem? (I
> > set hbase.hregion.max.filesize to 2G, and do not pre-sharding hbase
> table.
> > Can this cause the problem?)
>
>
> You should presplit.
>
>
> > Exception in thread "main"
> > org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> >
> lzo10DB,\x89\xB7\x17\xD2\x08\xA9\x87\xFFg\xE2\x9Fxy8\xD0\x8A\r\xA7\x96\xB9\x9D|D\x8D\xE9\xB1\x8B\x94\xDF5\xA3\xDD\x86O\x7F#f%S\x93\xE0F\x1E|\x07\xF9Q\xF3\xD4,1319046253246.dbc7a6366cdd86019af7d56284dc5d77.
>
>
> The region is offline. Either it took too long for it to be put back
> on line after a server crash or a split or its not online at all.
> Grep around in your master log to see if you can figure what happened
> with this region.
>
>
> > Exception in thread "main"
> > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed
> > 1 action: NotServingRegionException: 1 time, servers with issues:
> > sds-179-36:60020,
>
>
> Client thinks region is on a server but server says it doesn't have
> it. Again grep around in master log to try see what happened.
> Likely, the accounting for this region got messed up.
>
> St.Ack
>
Re: Two kinds of hbase client exceptions
Posted by Stack <st...@duboce.net>.
On Thu, Oct 20, 2011 at 7:07 AM, 吕鹏 <lv...@gmail.com> wrote:
> Hello everyone:
> My hbase&hadoop is cdh3u0.
Why not run u1?
> To test the cluster, i use 10 client to write
> date to hbase concurrently. I implement the client by hbase java api. When
> the test run for several hours, the 10 clients are all crashed, and two
> kinds of exceptions is catched. How can i solve the problem? (I
> set hbase.hregion.max.filesize to 2G, and do not pre-sharding hbase table.
> Can this cause the problem?)
You should presplit.
> Exception in thread "main"
> org.apache.hadoop.hbase.client.RegionOfflineException: region offline:
> lzo10DB,\x89\xB7\x17\xD2\x08\xA9\x87\xFFg\xE2\x9Fxy8\xD0\x8A\r\xA7\x96\xB9\x9D|D\x8D\xE9\xB1\x8B\x94\xDF5\xA3\xDD\x86O\x7F#f%S\x93\xE0F\x1E|\x07\xF9Q\xF3\xD4,1319046253246.dbc7a6366cdd86019af7d56284dc5d77.
The region is offline. Either it took too long for it to be put back
on line after a server crash or a split or its not online at all.
Grep around in your master log to see if you can figure what happened
with this region.
> Exception in thread "main"
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
> 1 action: NotServingRegionException: 1 time, servers with issues:
> sds-179-36:60020,
Client thinks region is on a server but server says it doesn't have
it. Again grep around in master log to try see what happened.
Likely, the accounting for this region got messed up.
St.Ack