You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Lars George <la...@worldlingo.com> on 2008/01/04 20:19:54 UTC

WrongRegionException - How do I fix it?

Hi,

I have inserted about 3.5m documents in a single two column table in 
HBase running on 32 nodes. So far I was able to insert most data, but 
with the last million or so I am stuck with this error:

org.apache.hadoop.hbase.WrongRegionException: Requested row out of range 
for HRegion docs,DC20020099792_20020725,9149203683830573099, 
startKey='DC20020099792_20020725', endKey='DC20020099792_20020725', 
row='DC20020099841_20020725'

Querying for the document returns nothing, means it looks like the 
document does not exist - although I am sure I tried inserting it a few 
times. Deleting or trying to re-insert returns the above error, both 
through the API (using HTable) as well as through the HBase shell.

I tried a restart of Hadoop/HBase to no avail. How do fix this problem? 
Any help is appreciated.

Best regards,
Lars

----
Lars George, CTO
WorldLingo

Re: WrongRegionException - How do I fix it?

Posted by Lars George <la...@worldlingo.com>.
Hi Stack,

That also does not seem right?

   192.168.105.55:60020 -6321685367535019868 requests: 0 regions: 
64                                                               
   192.168.105.51:60020 -4174238310567886366 requests: 0 regions: 
51                                                               
   192.168.105.17:60020 5581663697624462721  requests: 0 regions: 
156                                                              
   192.168.105.57:60020 -5211607822000744542 requests: 0 regions: 
62                                                               
   192.168.105.31:60020 7888193037635443812  requests: 0 regions: 
287                                                              
   192.168.105.59:60020 -46386689243367454   requests: 0 regions: 
95                                                               
   192.168.105.45:60020 8533079767141988197  requests: 0 regions: 
42                                                               
   192.168.105.77:60020 7803898403387937725  requests: 0 regions: 
67                                                               
   192.168.105.49:60020 -163550390447606491  requests: 0 regions: 
217                                                              
   192.168.105.75:60020 -6446348712848682453 requests: 0 regions: 
44                                                               
   192.168.105.27:60020 4779581083944377871  requests: 0 regions: 
109                                                              
   192.168.105.65:60020 4901865573283507122  requests: 0 regions: 
54                                                               
   192.168.105.41:60020 4787302286434914341  requests: 0 regions: 
114                                                              
   192.168.105.33:60020 -2508294300014104969 requests: 0 regions: 
66                                                               
   192.168.105.73:60020 -2334633498376763895 requests: 0 regions: 
88                                                               
   192.168.105.63:60020 8674021859672682545  requests: 0 regions: 
48                                                               
   192.168.105.35:60020 1407450539566532332  requests: 4296 regions: 
1                                                             
   192.168.105.47:60020 -3402160042637529057 requests: 0 regions: 
35                                                               
   192.168.105.37:60020 8133142711395683731  requests: 0 regions: 
67                                                               
   192.168.105.71:60020 -9207688945963624521 requests: 0 regions: 
57                                                               
   192.168.105.61:60020 -7699687997693377026 requests: 0 regions: 
33                                                               
   192.168.105.23:60020 4216774296721487713  requests: 0 regions: 
89                                                               
   192.168.105.13:60020 -6032091814002308094 requests: 0 regions: 
44                                                               
   192.168.105.15:60020 -4136342318896158593 requests: 0 regions: 
320                                                              
   192.168.105.67:60020 5042396379505131550  requests: 0 regions: 
34                                                               
   192.168.105.21:60020 -7677313271932692345 requests: 5 regions: 
172                                                              
   192.168.105.53:60020 1635187659254945320  requests: 0 regions: 
0                                                                
   192.168.105.69:60020 5310455167394956263  requests: 0 regions: 
157                                                              
   192.168.105.25:60020 1421797545576665774  requests: 0 regions: 
97                                                               
   192.168.105.43:60020 -6942740213227675121 requests: 0 regions: 
79                                                               
   192.168.105.29:60020 3244292531316078025  requests: 0 regions: 
98                                                               
   192.168.105.39:60020 -2915937415126913637 requests: 0 regions: 
134                                                              
                                                                       
See that .35 server. Way different than the others. This is server 26 in 
the email I sent before, the one with all sorts of errors not being able 
to find regions, delete blocks etc.

Can I somehow wipe this clean?

Lars


stack wrote:
> Lars George wrote:
>> Hi,
>>
>> I have inserted about 3.5m documents in a single two column table in 
>> HBase running on 32 nodes. So far I was able to insert most data, but 
>> with the last million or so I am stuck with this error:
>>
>> org.apache.hadoop.hbase.WrongRegionException: Requested row out of 
>> range for HRegion docs,DC20020099792_20020725,9149203683830573099, 
>> startKey='DC20020099792_20020725', endKey='DC20020099792_20020725', 
>> row='DC20020099841_20020725'
>
> This happens every time you try to do an insert?
>
>> Querying for the document returns nothing, means it looks like the 
>> document does not exist - although I am sure I tried inserting it a 
>> few times. Deleting or trying to re-insert returns the above error, 
>> both through the API (using HTable) as well as through the HBase shell.
>>
>> I tried a restart of Hadoop/HBase to no avail. How do fix this 
>> problem? Any help is appreciated.
>
> You have DEBUG enabled for hbase (See 
> http://wiki.apache.org/lucene-hadoop/Hbase/FAQ#4).  Do the logs tell 
> you anything more: e.g. any interesting exceptions?
>
> Which hbase version?
>
> St.Ack
>

Re: WrongRegionException - How do I fix it?

Posted by Lars George <la...@worldlingo.com>.
Hi Stack,

Yes, I read that on Nabble today too, that is why I was asking actually. 
What is the timeline for this? I will reply to Jim's message too.

Thanks for pointing that out.

Lars

stack wrote:
> stack wrote:
>> Regards your having to do this again in the near future, hopefully 
>> not...
>
> Related, see 
> http://www.nabble.com/Question-for-HBase-users-tc14607732.html.
> St.Ack
>

Re: WrongRegionException - How do I fix it?

Posted by stack <st...@duboce.net>.
stack wrote:
> Regards your having to do this again in the near future, hopefully not...

Related, see http://www.nabble.com/Question-for-HBase-users-tc14607732.html.
St.Ack

Re: WrongRegionException - How do I fix it?

Posted by stack <st...@duboce.net>.
Lars George wrote:
> Eeek, means what? Replace jar files with the trunk version and the 
> reformat the whole Hadoop dfs (since I only have HBase on top of it) 
> and then reimporting all 4.5m documents? What are my chances that 
> there are more profound changes coming that require me to do that 
> again in the near future? Just asking.

Yes.  Start over especially if you are running hbase only.  That'd be 
cleanest and quickest path to hbase TRUNK nirvana.

Regards your having to do this again in the near future, hopefully not, 
but please allow for the fact that hbase is beta software.  There is at 
least one well-known means of losing data (HADOOP-1700) and probably 
more that we have yet to discover.  Because of this, we want to be clear 
that folks should be prepared for data loss in their hbase installations.

St.Ack

P.S. Lars, there is an #hbase channel up on irc.freenode.net if you need 
more immediate help or need suggestions for how you might faster 
populate hbase.

Re: WrongRegionException - How do I fix it?

Posted by Lars George <la...@worldlingo.com>.
Stack,

Eeek, means what? Replace jar files with the trunk version and the 
reformat the whole Hadoop dfs (since I only have HBase on top of it) and 
then reimporting all 4.5m documents? What are my chances that there are 
more profound changes coming that require me to do that again in the 
near future? Just asking.

Lars

stack wrote:
> Lars George wrote:
>> Hi Stack,
>>
>> Can and will do, but does that make the error go away, i.e. 
>> automagically fix it? Or is it broken and nothing can be done about 
>> it now?
> Your current install is broke.  We could try spending time getting it 
> back into a healthy state but TRUNK is more robust than the first 
> hbase release, 0.15.x.  It also lacks many of the bugs you reproduced 
> in your earlier mail.  I'd suggest that our time would be better spent 
> working with the latest hbase particularly as patches to hbase in the 
> 0.15 branch are unlikely.
>
> Thanks Lars,
> St.Ack
>

Re: WrongRegionException - How do I fix it?

Posted by stack <st...@duboce.net>.
Lars George wrote:
> Hi Stack,
>
> Can and will do, but does that make the error go away, i.e. 
> automagically fix it? Or is it broken and nothing can be done about it 
> now?
Your current install is broke.  We could try spending time getting it 
back into a healthy state but TRUNK is more robust than the first hbase 
release, 0.15.x.  It also lacks many of the bugs you reproduced in your 
earlier mail.  I'd suggest that our time would be better spent working 
with the latest hbase particularly as patches to hbase in the 0.15 
branch are unlikely.

Thanks Lars,
St.Ack

Re: WrongRegionException - How do I fix it?

Posted by Lars George <la...@worldlingo.com>.
Hi Stack,

Can and will do, but does that make the error go away, i.e. 
automagically fix it? Or is it broken and nothing can be done about it now?

Lars

stack wrote:
> If possible, please move to TRUNK instead.  Most of the below have 
> been addressed there (I can send you a patch if you want to run hbase 
> TRUNK on hadoop 0.15.x).
>
> Further comments inline below:
>
>
> Lars George wrote:
>> Hi Stack,
>>
>> Yes, it happens every time I insert particular rows. Before it would 
>> fail every now and so often, but since now all "good" rows are 
>> inserted I am stuck with the ones that do not insert. And I am sure 
>> they did once, with no error. So they are in there in limbo, but I 
>> cannot retrieve nor delete or insert them.
>>
>> It mentions in the FAQ that I can switch on debugging through the UI, 
>> but I cannot see where. I am using version 0.15.1, is that supposed 
>> to have that option or do I need to go the log4j.properties plus 
>> restart route?
>
> This is a post-0.15 release feature (It says post-0.15.x in the FAQ).
>
>
>> I have errors all the  time - which quite frankly worry me. Here a 
>> list of what I see so far:
>>
>> 1. At startup
>>
>> ==> 
>> /usr/local/hadoop/logs/hbase-pdc-regionserver-lv1-xen-pdc-40.worldlingo.com.log 
>> <==
>> 2008-01-03 14:11:22,512 WARN org.apache.hadoop.util.NativeCodeLoader: 
>> Unable to load native-hadoop library for your platform... using 
>> builtin-java classes where applicable
>> 2008-01-03 14:11:29,808 WARN org.apache.hadoop.hbase.HRegionServer: 
>> Processing message (Retry: 0)
>> java.io.IOException: java.io.IOException: 
>> java.util.ConcurrentModificationException 
> ....
>
>
> Fixed in TRUNK
>
>
>>
>>
>> 2. Sporadically
>>
>> 2008-01-03 21:32:00,639 WARN org.apache.hadoop.dfs.DataNode: 
>> Unexpected error trying to delete block blk_-8931657506153335343. 
>> Block not found in blockMap.
>> 2008-01-03 21:32:00,639 WARN org.apache.hadoop.dfs.DataNode: 
>> Unexpected error trying to delete block blk_3775459202881005176. 
>> Block not found in blockMap.
>> 2008-01-03 21:32:00,639 WARN org.apache.hadoop.dfs.DataNode: 
>> Unexpected error trying to delete block blk_-283089329129695997. 
>> Block not found in blockMap.
>> 2008-01-03 21:32:00,644 WARN org.apache.hadoop.dfs.DataNode: 
>> java.io.IOException: Error in deleting blocks.
>>        at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:719)
>>        at 
>> org.apache.hadoop.dfs.DataNode.processCommand(DataNode.java:625)
>>        at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:528)
>>        at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1494)
>>        at java.lang.Thread.run(Thread.java:595)
>>
>> These come with all sorts of blocks, they do not appear too often, 
>> but on a regular basis.
>>
> 0.15.x was doing updates against files that had been removed by 
> another thread.  Made for strange errors in hdfs.  That said, I don't 
> recall having seen the above.  Do you have namenode at DEBUG level?  
> If so, try tracing the above problematic blocks therein; see if you 
> can figure out a story as to what happened with these blocks.
>
> If HDFS is in an inconsistent state, hbase will be inconsistent too.
>
>> 3. Misc
>>
>> I see these too from this morning
>>
>> 2008-01-04 08:23:30,616 ERROR org.apache.hadoop.hbase.HRegionServer: 
>> unable to process message: MSG_REGION_OPEN : regionname: do
>> cs,DC20020096869_20020725,43610073395851568, startKey: 
>> <DC20020096869_20020725>, tableDesc: {name: docs, families: 
>> {contents:={n
>> ame: contents, max versions: 3, compression: NONE, in memory: false, 
>> max length: 2147483647, bloom filter: none}, language:={name: l
>> anguage, max versions: 3, compression: NONE, in memory: false, max 
>> length: 2147483647, bloom filter: none}, mimetype:={name: mimetyp
>> e, max versions: 3, compression: NONE, in memory: false, max length: 
>> 2147483647, bloom filter: none}}}
>> java.io.IOException: java.io.IOException: Cannot open filename 
>> /hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/co
>> ntents/info/1501965039462307633
>>        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
>>        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>>        at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>>
>>        at java.lang.reflect.Method.invoke(Method.java:585)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>>
>>        at 
>> sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
>>        at 
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) 
>>
>>        at 
>> java.lang.reflect.Constructor.newInstance(Constructor.java:494)
>>        at 
>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82) 
>>
>>        at 
>> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48) 
>>
>>        at 
>> org.apache.hadoop.hbase.HRegionServer$Worker.run(HRegionServer.java:903)
>>        at java.lang.Thread.run(Thread.java:595)
>>
>>
>
> I'd guess this file is made of the above cited problematic blocks.  
> Can you find it on hdfs?  Can you download it?  (Try doing a 
> './bin/hadoop fs fsck /HBASE_DIR' -- see what it says).
>
>
>
>> Another one is this:
>>
>> ==> 
>> /usr/local/hadoop/logs/hbase-pdc-regionserver-lv1-xen-pdc-62.worldlingo.com.log 
>> <==
>> 2008-01-04 08:16:32,001 WARN org.apache.hadoop.hbase.HRegion: Region 
>> docs,DC20020099792_20020725,9149203683830573099 is NOT spli
>> table though its aggregate size is 111.4m and desired size is 64.0m
>>
>> These come up with different region numbers.
>>
> This is ok.  We want to split the region because its > 64MB but this 
> region has outstanding references to another, parent region, so is not 
> yet splittable (It should have turned splittable a little later in 
> your log).
>
>
>> And another one:
>>
>> 2008-01-03 11:27:55,437 WARN org.apache.hadoop.hbase.HStore: Failed 
>> getting store size
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File does 
>> not exist
>>        at 
>> org.apache.hadoop.dfs.FSDirectory.getFileInfo(FSDirectory.java:489)
>>        at 
>> org.apache.hadoop.dfs.FSNamesystem.getFileInfo(FSNamesystem.java:1360)
>>        at org.apache.hadoop.dfs.NameNode.getFileInfo(NameNode.java:428)
>>        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>>        at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>>
>>        at java.lang.reflect.Method.invoke(Method.java:585)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>>
> Fixed in TRUNK
>
>
>>
>> And another one:
>>
>> 2008-01-03 15:43:39,590 WARN org.apache.hadoop.dfs.DataNode: Got 
>> exception while serving blk_3676251342939485484 to /192.168.105.21:
>> java.io.IOException: Block blk_3676251342939485484 is not valid.
>>        at 
>> org.apache.hadoop.dfs.FSDataset.getBlockFile(FSDataset.java:528)
>>        at 
>> org.apache.hadoop.dfs.DataNode$BlockSender.<init>(DataNode.java:1051)
>>        at 
>> org.apache.hadoop.dfs.DataNode$DataXceiver.readBlock(DataNode.java:843)
>>        at 
>> org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:801)
>>        at java.lang.Thread.run(Thread.java:595)
>>
>> Again, they come with different block numbers.
>>
> This hdfs exception may be recoverable IIRC; hdfs gets the block 
> elsewhere.
>
>
>> 4. Inserting document with errors
>>
>> If I try to add one of those documents, where I get an error back, 
>> this is what I see in the logs so far:
>>
>> Uploading -> DC20020099841_20020725
>>
>> Server 20:
>>
>>
> ....
>> 2007-12-30 17:26:42,392 INFO org.mortbay.util.Container: Started 
>> HttpContext[/static,/static]
>> 2007-12-30 17:26:42,395 INFO org.mortbay.http.SocketListener: Started 
>> SocketListener on 0.0.0.0:60030
>> 2007-12-30 17:26:42,395 INFO org.mortbay.util.Container: Started 
>> org.mortbay.jetty.Server@1fd0fafc
>> 2007-12-30 17:26:42,396 INFO org.apache.hadoop.ipc.Server: IPC Server 
>> listener on 60020: starting
>> 2007-12-30 17:26:42,397 INFO org.apache.hadoop.ipc.Server: IPC Server 
>> handler 0 on 60020: starting
>> 2007-12-30 17:26:42,402 INFO org.apache.hadoop.ipc.Server: IPC Server 
>> handler 1 on 60020: starting
>> 2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
>> handler 3 on 60020: starting
>> 2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
>> handler 4 on 60020: starting
>> 2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
>> handler 5 on 60020: starting
>> 2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
>> handler 6 on 60020: starting
>>
>
>
> Please set these to run at DEBUG level, at least while we are trying 
> to figure out whats going on.
>
>>
>> Server 26:
>>
>> 2008-01-04 12:18:28,125 WARN org.apache.hadoop.hbase.HRegionServer: 
>> java.io.IOException: java.io.IOException: Cannot open filename /
>> hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/contents/info/1501965039462307633 
>>
>>        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
>>        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>>        at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>>
>>        at java.lang.reflect.Method.invoke(Method.java:585)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>>
>> 2008-01-04 12:18:28,131 WARN org.apache.hadoop.hbase.HRegionServer: 
>> java.io.IOException: java.io.IOException: Cannot open filename /
>> hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/contents/info/1501965039462307633 
>>
>>        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
>>        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>>        at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>>
>>        at java.lang.reflect.Method.invoke(Method.java:585)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>>
>> 2008-01-04 12:18:28,147 WARN org.apache.hadoop.hbase.HRegionServer: 
>> java.io.IOException: java.io.IOException: Cannot open filename /
>> hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/contents/info/1501965039462307633 
>>
>>        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
>>        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>>        at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>>
>>        at java.lang.reflect.Method.invoke(Method.java:585)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>>
> The above are bad errors.  hdfs lost our files or, more likely, we 
> mangled their writing.
>
> ...
>>
>> Does this help?
>>
>
> Yes.
>
> St.Ack
>
>
>> Thanks,
>> Lars
>>
>>
>>
>> stack wrote:
>>> Lars George wrote:
>>>> Hi,
>>>>
>>>> I have inserted about 3.5m documents in a single two column table 
>>>> in HBase running on 32 nodes. So far I was able to insert most 
>>>> data, but with the last million or so I am stuck with this error:
>>>>
>>>> org.apache.hadoop.hbase.WrongRegionException: Requested row out of 
>>>> range for HRegion docs,DC20020099792_20020725,9149203683830573099, 
>>>> startKey='DC20020099792_20020725', endKey='DC20020099792_20020725', 
>>>> row='DC20020099841_20020725'
>>>
>>> This happens every time you try to do an insert?
>>>
>>>> Querying for the document returns nothing, means it looks like the 
>>>> document does not exist - although I am sure I tried inserting it a 
>>>> few times. Deleting or trying to re-insert returns the above error, 
>>>> both through the API (using HTable) as well as through the HBase 
>>>> shell.
>>>>
>>>> I tried a restart of Hadoop/HBase to no avail. How do fix this 
>>>> problem? Any help is appreciated.
>>>
>>> You have DEBUG enabled for hbase (See 
>>> http://wiki.apache.org/lucene-hadoop/Hbase/FAQ#4).  Do the logs tell 
>>> you anything more: e.g. any interesting exceptions?
>>>
>>> Which hbase version?
>>>
>>> St.Ack
>>>
>

Re: WrongRegionException - How do I fix it?

Posted by stack <st...@duboce.net>.
If possible, please move to TRUNK instead.  Most of the below have been 
addressed there (I can send you a patch if you want to run hbase TRUNK 
on hadoop 0.15.x).

Further comments inline below:


Lars George wrote:
> Hi Stack,
>
> Yes, it happens every time I insert particular rows. Before it would 
> fail every now and so often, but since now all "good" rows are 
> inserted I am stuck with the ones that do not insert. And I am sure 
> they did once, with no error. So they are in there in limbo, but I 
> cannot retrieve nor delete or insert them.
>
> It mentions in the FAQ that I can switch on debugging through the UI, 
> but I cannot see where. I am using version 0.15.1, is that supposed to 
> have that option or do I need to go the log4j.properties plus restart 
> route?

This is a post-0.15 release feature (It says post-0.15.x in the FAQ).


> I have errors all the  time - which quite frankly worry me. Here a 
> list of what I see so far:
>
> 1. At startup
>
> ==> 
> /usr/local/hadoop/logs/hbase-pdc-regionserver-lv1-xen-pdc-40.worldlingo.com.log 
> <==
> 2008-01-03 14:11:22,512 WARN org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using 
> builtin-java classes where applicable
> 2008-01-03 14:11:29,808 WARN org.apache.hadoop.hbase.HRegionServer: 
> Processing message (Retry: 0)
> java.io.IOException: java.io.IOException: 
> java.util.ConcurrentModificationException 
....


Fixed in TRUNK


>
>
> 2. Sporadically
>
> 2008-01-03 21:32:00,639 WARN org.apache.hadoop.dfs.DataNode: 
> Unexpected error trying to delete block blk_-8931657506153335343. 
> Block not found in blockMap.
> 2008-01-03 21:32:00,639 WARN org.apache.hadoop.dfs.DataNode: 
> Unexpected error trying to delete block blk_3775459202881005176. Block 
> not found in blockMap.
> 2008-01-03 21:32:00,639 WARN org.apache.hadoop.dfs.DataNode: 
> Unexpected error trying to delete block blk_-283089329129695997. Block 
> not found in blockMap.
> 2008-01-03 21:32:00,644 WARN org.apache.hadoop.dfs.DataNode: 
> java.io.IOException: Error in deleting blocks.
>        at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:719)
>        at 
> org.apache.hadoop.dfs.DataNode.processCommand(DataNode.java:625)
>        at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:528)
>        at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1494)
>        at java.lang.Thread.run(Thread.java:595)
>
> These come with all sorts of blocks, they do not appear too often, but 
> on a regular basis.
>
0.15.x was doing updates against files that had been removed by another 
thread.  Made for strange errors in hdfs.  That said, I don't recall 
having seen the above.  Do you have namenode at DEBUG level?  If so, try 
tracing the above problematic blocks therein; see if you can figure out 
a story as to what happened with these blocks.

If HDFS is in an inconsistent state, hbase will be inconsistent too.

> 3. Misc
>
> I see these too from this morning
>
> 2008-01-04 08:23:30,616 ERROR org.apache.hadoop.hbase.HRegionServer: 
> unable to process message: MSG_REGION_OPEN : regionname: do
> cs,DC20020096869_20020725,43610073395851568, startKey: 
> <DC20020096869_20020725>, tableDesc: {name: docs, families: {contents:={n
> ame: contents, max versions: 3, compression: NONE, in memory: false, 
> max length: 2147483647, bloom filter: none}, language:={name: l
> anguage, max versions: 3, compression: NONE, in memory: false, max 
> length: 2147483647, bloom filter: none}, mimetype:={name: mimetyp
> e, max versions: 3, compression: NONE, in memory: false, max length: 
> 2147483647, bloom filter: none}}}
> java.io.IOException: java.io.IOException: Cannot open filename 
> /hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/co
> ntents/info/1501965039462307633
>        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
>        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>
>        at java.lang.reflect.Method.invoke(Method.java:585)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>
>        at 
> sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
>        at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) 
>
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
>        at 
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82) 
>
>        at 
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48) 
>
>        at 
> org.apache.hadoop.hbase.HRegionServer$Worker.run(HRegionServer.java:903)
>        at java.lang.Thread.run(Thread.java:595)
>
>

I'd guess this file is made of the above cited problematic blocks.  Can 
you find it on hdfs?  Can you download it?  (Try doing a './bin/hadoop 
fs fsck /HBASE_DIR' -- see what it says).



> Another one is this:
>
> ==> 
> /usr/local/hadoop/logs/hbase-pdc-regionserver-lv1-xen-pdc-62.worldlingo.com.log 
> <==
> 2008-01-04 08:16:32,001 WARN org.apache.hadoop.hbase.HRegion: Region 
> docs,DC20020099792_20020725,9149203683830573099 is NOT spli
> table though its aggregate size is 111.4m and desired size is 64.0m
>
> These come up with different region numbers.
>
This is ok.  We want to split the region because its > 64MB but this 
region has outstanding references to another, parent region, so is not 
yet splittable (It should have turned splittable a little later in your 
log).


> And another one:
>
> 2008-01-03 11:27:55,437 WARN org.apache.hadoop.hbase.HStore: Failed 
> getting store size
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File does 
> not exist
>        at 
> org.apache.hadoop.dfs.FSDirectory.getFileInfo(FSDirectory.java:489)
>        at 
> org.apache.hadoop.dfs.FSNamesystem.getFileInfo(FSNamesystem.java:1360)
>        at org.apache.hadoop.dfs.NameNode.getFileInfo(NameNode.java:428)
>        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>
>        at java.lang.reflect.Method.invoke(Method.java:585)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>
Fixed in TRUNK


>
> And another one:
>
> 2008-01-03 15:43:39,590 WARN org.apache.hadoop.dfs.DataNode: Got 
> exception while serving blk_3676251342939485484 to /192.168.105.21:
> java.io.IOException: Block blk_3676251342939485484 is not valid.
>        at 
> org.apache.hadoop.dfs.FSDataset.getBlockFile(FSDataset.java:528)
>        at 
> org.apache.hadoop.dfs.DataNode$BlockSender.<init>(DataNode.java:1051)
>        at 
> org.apache.hadoop.dfs.DataNode$DataXceiver.readBlock(DataNode.java:843)
>        at 
> org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:801)
>        at java.lang.Thread.run(Thread.java:595)
>
> Again, they come with different block numbers.
>
This hdfs exception may be recoverable IIRC; hdfs gets the block elsewhere.


> 4. Inserting document with errors
>
> If I try to add one of those documents, where I get an error back, 
> this is what I see in the logs so far:
>
> Uploading -> DC20020099841_20020725
>
> Server 20:
>
>
....
> 2007-12-30 17:26:42,392 INFO org.mortbay.util.Container: Started 
> HttpContext[/static,/static]
> 2007-12-30 17:26:42,395 INFO org.mortbay.http.SocketListener: Started 
> SocketListener on 0.0.0.0:60030
> 2007-12-30 17:26:42,395 INFO org.mortbay.util.Container: Started 
> org.mortbay.jetty.Server@1fd0fafc
> 2007-12-30 17:26:42,396 INFO org.apache.hadoop.ipc.Server: IPC Server 
> listener on 60020: starting
> 2007-12-30 17:26:42,397 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 0 on 60020: starting
> 2007-12-30 17:26:42,402 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 1 on 60020: starting
> 2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 3 on 60020: starting
> 2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 4 on 60020: starting
> 2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 5 on 60020: starting
> 2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 6 on 60020: starting
>


Please set these to run at DEBUG level, at least while we are trying to 
figure out whats going on.

>
> Server 26:
>
> 2008-01-04 12:18:28,125 WARN org.apache.hadoop.hbase.HRegionServer: 
> java.io.IOException: java.io.IOException: Cannot open filename /
> hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/contents/info/1501965039462307633 
>
>        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
>        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>
>        at java.lang.reflect.Method.invoke(Method.java:585)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>
> 2008-01-04 12:18:28,131 WARN org.apache.hadoop.hbase.HRegionServer: 
> java.io.IOException: java.io.IOException: Cannot open filename /
> hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/contents/info/1501965039462307633 
>
>        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
>        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>
>        at java.lang.reflect.Method.invoke(Method.java:585)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>
> 2008-01-04 12:18:28,147 WARN org.apache.hadoop.hbase.HRegionServer: 
> java.io.IOException: java.io.IOException: Cannot open filename /
> hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/contents/info/1501965039462307633 
>
>        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
>        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
>
>        at java.lang.reflect.Method.invoke(Method.java:585)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>
The above are bad errors.  hdfs lost our files or, more likely, we 
mangled their writing.

...
>
> Does this help?
>

Yes.

St.Ack


> Thanks,
> Lars
>
>
>
> stack wrote:
>> Lars George wrote:
>>> Hi,
>>>
>>> I have inserted about 3.5m documents in a single two column table in 
>>> HBase running on 32 nodes. So far I was able to insert most data, 
>>> but with the last million or so I am stuck with this error:
>>>
>>> org.apache.hadoop.hbase.WrongRegionException: Requested row out of 
>>> range for HRegion docs,DC20020099792_20020725,9149203683830573099, 
>>> startKey='DC20020099792_20020725', endKey='DC20020099792_20020725', 
>>> row='DC20020099841_20020725'
>>
>> This happens every time you try to do an insert?
>>
>>> Querying for the document returns nothing, means it looks like the 
>>> document does not exist - although I am sure I tried inserting it a 
>>> few times. Deleting or trying to re-insert returns the above error, 
>>> both through the API (using HTable) as well as through the HBase shell.
>>>
>>> I tried a restart of Hadoop/HBase to no avail. How do fix this 
>>> problem? Any help is appreciated.
>>
>> You have DEBUG enabled for hbase (See 
>> http://wiki.apache.org/lucene-hadoop/Hbase/FAQ#4).  Do the logs tell 
>> you anything more: e.g. any interesting exceptions?
>>
>> Which hbase version?
>>
>> St.Ack
>>

Re: WrongRegionException - How do I fix it?

Posted by Lars George <la...@worldlingo.com>.
Hi Stack,

Yes, it happens every time I insert particular rows. Before it would 
fail every now and so often, but since now all "good" rows are inserted 
I am stuck with the ones that do not insert. And I am sure they did 
once, with no error. So they are in there in limbo, but I cannot 
retrieve nor delete or insert them.

It mentions in the FAQ that I can switch on debugging through the UI, 
but I cannot see where. I am using version 0.15.1, is that supposed to 
have that option or do I need to go the log4j.properties plus restart route?

I have errors all the  time - which quite frankly worry me. Here a list 
of what I see so far:

1. At startup

==> 
/usr/local/hadoop/logs/hbase-pdc-regionserver-lv1-xen-pdc-40.worldlingo.com.log 
<==
2008-01-03 14:11:22,512 WARN org.apache.hadoop.util.NativeCodeLoader: 
Unable to load native-hadoop library for your platform... using 
builtin-java classes where applicable
2008-01-03 14:11:29,808 WARN org.apache.hadoop.hbase.HRegionServer: 
Processing message (Retry: 0)
java.io.IOException: java.io.IOException: 
java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:841)
        at java.util.HashMap$EntryIterator.next(HashMap.java:883)
        at java.util.HashMap$EntryIterator.next(HashMap.java:881)
        at 
org.apache.hadoop.hbase.HMaster.getRegionsToAssign(HMaster.java:1739)
        at org.apache.hadoop.hbase.HMaster.assignRegions(HMaster.java:1597)
        at org.apache.hadoop.hbase.HMaster.processMsgs(HMaster.java:1582)
        at 
org.apache.hadoop.hbase.HMaster.regionServerReport(HMaster.java:1400)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method)
        at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
        at 
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
        at 
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
        at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:532)
        at java.lang.Thread.run(Thread.java:595)

2. Sporadically

2008-01-03 21:32:00,639 WARN org.apache.hadoop.dfs.DataNode: Unexpected 
error trying to delete block blk_-8931657506153335343. Block not found 
in blockMap.
2008-01-03 21:32:00,639 WARN org.apache.hadoop.dfs.DataNode: Unexpected 
error trying to delete block blk_3775459202881005176. Block not found in 
blockMap.
2008-01-03 21:32:00,639 WARN org.apache.hadoop.dfs.DataNode: Unexpected 
error trying to delete block blk_-283089329129695997. Block not found in 
blockMap.
2008-01-03 21:32:00,644 WARN org.apache.hadoop.dfs.DataNode: 
java.io.IOException: Error in deleting blocks.
        at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:719)
        at org.apache.hadoop.dfs.DataNode.processCommand(DataNode.java:625)
        at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:528)
        at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1494)
        at java.lang.Thread.run(Thread.java:595)

These come with all sorts of blocks, they do not appear too often, but 
on a regular basis.

3. Misc

I see these too from this morning

2008-01-04 08:23:30,616 ERROR org.apache.hadoop.hbase.HRegionServer: 
unable to process message: MSG_REGION_OPEN : regionname: do
cs,DC20020096869_20020725,43610073395851568, startKey: 
<DC20020096869_20020725>, tableDesc: {name: docs, families: {contents:={n
ame: contents, max versions: 3, compression: NONE, in memory: false, max 
length: 2147483647, bloom filter: none}, language:={name: l
anguage, max versions: 3, compression: NONE, in memory: false, max 
length: 2147483647, bloom filter: none}, mimetype:={name: mimetyp
e, max versions: 3, compression: NONE, in memory: false, max length: 
2147483647, bloom filter: none}}}
java.io.IOException: java.io.IOException: Cannot open filename 
/hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/co
ntents/info/1501965039462307633
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

        at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown 
Source)
        at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
        at 
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
        at 
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
        at 
org.apache.hadoop.hbase.HRegionServer$Worker.run(HRegionServer.java:903)
        at java.lang.Thread.run(Thread.java:595)


Another one is this:

==> 
/usr/local/hadoop/logs/hbase-pdc-regionserver-lv1-xen-pdc-62.worldlingo.com.log 
<==
2008-01-04 08:16:32,001 WARN org.apache.hadoop.hbase.HRegion: Region 
docs,DC20020099792_20020725,9149203683830573099 is NOT spli
table though its aggregate size is 111.4m and desired size is 64.0m

These come up with different region numbers.

And another one:

2008-01-03 11:27:55,437 WARN org.apache.hadoop.hbase.HStore: Failed 
getting store size
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File does 
not exist
        at 
org.apache.hadoop.dfs.FSDirectory.getFileInfo(FSDirectory.java:489)
        at 
org.apache.hadoop.dfs.FSNamesystem.getFileInfo(FSNamesystem.java:1360)
        at org.apache.hadoop.dfs.NameNode.getFileInfo(NameNode.java:428)
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

        at org.apache.hadoop.ipc.Client.call(Client.java:482)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
        at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
        at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:438)
        at 
org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:334)
        at org.apache.hadoop.hbase.HStoreFile.length(HStoreFile.java:971)
        at org.apache.hadoop.hbase.HStore.size(HStore.java:1322)
        at org.apache.hadoop.hbase.HRegion.largestHStore(HRegion.java:630)
        at org.apache.hadoop.hbase.HRegion.needsSplit(HRegion.java:590)
        at 
org.apache.hadoop.hbase.HRegionServer$SplitOrCompactChecker.checkForSplitsOrCompactions(HRegionServer.java:201)
        at 
org.apache.hadoop.hbase.HRegionServer$SplitOrCompactChecker.chore(HRegionServer.java:186)
        at org.apache.hadoop.hbase.Chore.run(Chore.java:58)


And another one:

2008-01-03 15:43:39,590 WARN org.apache.hadoop.dfs.DataNode: Got 
exception while serving blk_3676251342939485484 to /192.168.105.21:
java.io.IOException: Block blk_3676251342939485484 is not valid.
        at org.apache.hadoop.dfs.FSDataset.getBlockFile(FSDataset.java:528)
        at 
org.apache.hadoop.dfs.DataNode$BlockSender.<init>(DataNode.java:1051)
        at 
org.apache.hadoop.dfs.DataNode$DataXceiver.readBlock(DataNode.java:843)
        at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:801)
        at java.lang.Thread.run(Thread.java:595)

Again, they come with different block numbers.

4. Inserting document with errors

If I try to add one of those documents, where I get an error back, this 
is what I see in the logs so far:

Uploading -> DC20020099841_20020725

Server 20:

2008-01-04 12:18:01,987 WARN org.apache.hadoop.hbase.HRegion: Region 
docs,DC20020099792_20020725,9149203683830573099 is NOT spli
table though its aggregate size is 111.4m and desired size is 64.0m
2008-01-04 12:18:16,983 WARN org.apache.hadoop.hbase.HRegion: Region 
docs,DC20020099792_20020725,3506350021119702948 is NOT spli
table though its aggregate size is 111.4m and desired size is 64.0m
2008-01-04 12:18:16,988 WARN org.apache.hadoop.hbase.HRegion: Region 
docs,DC20020099792_20020725,9149203683830573099 is NOT spli
table though its aggregate size is 111.4m and desired size is 64.0m
2008-01-04 12:18:31,985 WARN org.apache.hadoop.hbase.HRegion: Region 
docs,DC20020099792_20020725,3506350021119702948 is NOT spli
table though its aggregate size is 111.4m and desired size is 64.0m
2008-01-04 12:18:31,990 WARN org.apache.hadoop.hbase.HRegion: Region 
docs,DC20020099792_20020725,9149203683830573099 is NOT spli
table though its aggregate size is 111.4m and desired size is 64.0m

==> 
/usr/local/hadoop/logs/hbase-pdc-regionserver-lv1-xen-pdc-20.worldlingo.com.log.2007-12-30 
<==
2007-12-30 17:26:40,932 INFO org.apache.hadoop.hbase.HRegionServer: Root 
dir: /hbase
2007-12-30 17:26:41,091 WARN org.apache.hadoop.util.NativeCodeLoader: 
Unable to load native-hadoop library for your platform... usin
g builtin-java classes where applicable
2007-12-30 17:26:41,093 INFO org.apache.hadoop.hbase.HLog: new log 
writer created at /hbase/log_192.168.105.29_2760247535651987904_6
0020/hlog.dat.000
2007-12-30 17:26:41,166 INFO org.mortbay.util.Credential: Checking 
Resource aliases
2007-12-30 17:26:41,328 INFO org.mortbay.http.HttpServer: Version 
Jetty/5.1.4
2007-12-30 17:26:42,276 INFO org.mortbay.util.Container: Started 
org.mortbay.jetty.servlet.WebApplicationHandler@69d6065
2007-12-30 17:26:42,392 INFO org.mortbay.util.Container: Started 
WebApplicationContext[/,/]
2007-12-30 17:26:42,392 INFO org.mortbay.util.Container: Started 
HttpContext[/logs,/logs]
2007-12-30 17:26:42,392 INFO org.mortbay.util.Container: Started 
HttpContext[/static,/static]
2007-12-30 17:26:42,395 INFO org.mortbay.http.SocketListener: Started 
SocketListener on 0.0.0.0:60030
2007-12-30 17:26:42,395 INFO org.mortbay.util.Container: Started 
org.mortbay.jetty.Server@1fd0fafc
2007-12-30 17:26:42,396 INFO org.apache.hadoop.ipc.Server: IPC Server 
listener on 60020: starting
2007-12-30 17:26:42,397 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 0 on 60020: starting
2007-12-30 17:26:42,402 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 1 on 60020: starting
2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 3 on 60020: starting
2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 4 on 60020: starting
2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 5 on 60020: starting
2007-12-30 17:26:42,403 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 6 on 60020: starting


Server 26:

2008-01-04 12:18:28,125 WARN org.apache.hadoop.hbase.HRegionServer: 
java.io.IOException: java.io.IOException: Cannot open filename /
hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/contents/info/1501965039462307633
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

2008-01-04 12:18:28,131 WARN org.apache.hadoop.hbase.HRegionServer: 
java.io.IOException: java.io.IOException: Cannot open filename /
hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/contents/info/1501965039462307633
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

2008-01-04 12:18:28,147 WARN org.apache.hadoop.hbase.HRegionServer: 
java.io.IOException: java.io.IOException: Cannot open filename /
hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/contents/info/1501965039462307633
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

...

2008-01-04 12:18:28,181 ERROR org.apache.hadoop.hbase.HRegionServer: 
unable to process message: MSG_REGION_OPEN : regionname: do
cs,DC20020095856_20020725,2673659877920938250, startKey: 
<DC20020095856_20020725>, tableDesc: {name: docs, families: {contents:=
{name: contents, max versions: 3, compression: NONE, in memory: false, 
max length: 2147483647, bloom filter: none}, language:={name:
 language, max versions: 3, compression: NONE, in memory: false, max 
length: 2147483647, bloom filter: none}, mimetype:={name: mimet
ype, max versions: 3, compression: NONE, in memory: false, max length: 
2147483647, bloom filter: none}}}
java.io.IOException: java.io.IOException: Cannot open filename 
/hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/co
ntents/info/1501965039462307633
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

        at 
sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source)
        at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
        at 
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
        at 
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
        at 
org.apache.hadoop.hbase.HRegionServer$Worker.run(HRegionServer.java:903)
        at java.lang.Thread.run(Thread.java:595)
2008-01-04 12:18:28,187 ERROR org.apache.hadoop.hbase.HRegionServer: 
unable to process message: MSG_REGION_OPEN : regionname: do
cs,DC20020096869_20020725,43610073395851568, startKey: 
<DC20020096869_20020725>, tableDesc: {name: docs, families: {contents:={n
ame: contents, max versions: 3, compression: NONE, in memory: false, max 
length: 2147483647, bloom filter: none}, language:={name: l
anguage, max versions: 3, compression: NONE, in memory: false, max 
length: 2147483647, bloom filter: none}, mimetype:={name: mimetyp
e, max versions: 3, compression: NONE, in memory: false, max length: 
2147483647, bloom filter: none}}}
java.io.IOException: java.io.IOException: Cannot open filename 
/hbase/hregion_docs,DC20020095856_20020725,7894263634108415584/co
ntents/info/1501965039462307633
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:238)
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

        at 
sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source)
        at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
        at 
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
        at 
org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
        at 
org.apache.hadoop.hbase.HRegionServer$Worker.run(HRegionServer.java:903)
        at java.lang.Thread.run(Thread.java:595)


Does this help?

Thanks,
Lars



stack wrote:
> Lars George wrote:
>> Hi,
>>
>> I have inserted about 3.5m documents in a single two column table in 
>> HBase running on 32 nodes. So far I was able to insert most data, but 
>> with the last million or so I am stuck with this error:
>>
>> org.apache.hadoop.hbase.WrongRegionException: Requested row out of 
>> range for HRegion docs,DC20020099792_20020725,9149203683830573099, 
>> startKey='DC20020099792_20020725', endKey='DC20020099792_20020725', 
>> row='DC20020099841_20020725'
>
> This happens every time you try to do an insert?
>
>> Querying for the document returns nothing, means it looks like the 
>> document does not exist - although I am sure I tried inserting it a 
>> few times. Deleting or trying to re-insert returns the above error, 
>> both through the API (using HTable) as well as through the HBase shell.
>>
>> I tried a restart of Hadoop/HBase to no avail. How do fix this 
>> problem? Any help is appreciated.
>
> You have DEBUG enabled for hbase (See 
> http://wiki.apache.org/lucene-hadoop/Hbase/FAQ#4).  Do the logs tell 
> you anything more: e.g. any interesting exceptions?
>
> Which hbase version?
>
> St.Ack
>

Re: WrongRegionException - How do I fix it?

Posted by stack <st...@duboce.net>.
Lars George wrote:
> Hi,
>
> I have inserted about 3.5m documents in a single two column table in 
> HBase running on 32 nodes. So far I was able to insert most data, but 
> with the last million or so I am stuck with this error:
>
> org.apache.hadoop.hbase.WrongRegionException: Requested row out of 
> range for HRegion docs,DC20020099792_20020725,9149203683830573099, 
> startKey='DC20020099792_20020725', endKey='DC20020099792_20020725', 
> row='DC20020099841_20020725'

This happens every time you try to do an insert?

> Querying for the document returns nothing, means it looks like the 
> document does not exist - although I am sure I tried inserting it a 
> few times. Deleting or trying to re-insert returns the above error, 
> both through the API (using HTable) as well as through the HBase shell.
>
> I tried a restart of Hadoop/HBase to no avail. How do fix this 
> problem? Any help is appreciated.

You have DEBUG enabled for hbase (See 
http://wiki.apache.org/lucene-hadoop/Hbase/FAQ#4).  Do the logs tell you 
anything more: e.g. any interesting exceptions?

Which hbase version?

St.Ack