You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jack Levin <ma...@gmail.com> on 2010/09/28 19:53:29 UTC

getting region PUT errors.

I am using rest and getting following errors on some insertions:


2010-09-28 10:51:07,215 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
row out of range for HRegion
img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029.,
startKey='samp754.png', getEndKey()='screenshot205.png',
row='sf6c.jpg'
	at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967)
	at org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030)
	at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120)
	at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367)
	at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)

And from REST log:


2010-09-28 10:50:26,308 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Failed
execution of all on mtab5.prod.imageshack.com:60020
org.apache.hadoop.hbase.regionserver.WrongRegionException:
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
row out of range for HRegion
img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029.,
startKey='samp754.png', getEndKey()='screenshot205.png',
row='sf6c.jpg'
	at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967)
	at org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030)
	at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120)
	at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367)
	at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)

	at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.translateException(HConnectionManager.java:1552)
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithoutRetries(HConnectionManager.java:1162)
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1530)
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1528)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)

It looks like even after restarting REST I get the same error, could
this been that zookeepers cached the wrong location of the region and
its not on the same regionserver?   Are there other methods to debug?

-Jack

Re: getting region PUT errors.

Posted by Jack Levin <ma...@gmail.com>.
I was able to fix it by running:

./hbase org.jruby.Main add_table.rb /hbase/img833

then in shell, disable, enable 'img833'

I am checking to see if I lost any files.

-Jack

On Tue, Sep 28, 2010 at 10:53 AM, Jack Levin <ma...@gmail.com> wrote:
> I am using rest and getting following errors on some insertions:
>
>
> 2010-09-28 10:51:07,215 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
> row out of range for HRegion
> img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029.,
> startKey='samp754.png', getEndKey()='screenshot205.png',
> row='sf6c.jpg'
<snip>

Re: getting region PUT errors.

Posted by Jack Levin <ma...@gmail.com>.
http://pastebin.com/mU8tpuiD there it is.

On Tue, Sep 28, 2010 at 12:03 PM, Stack <st...@duboce.net> wrote:
> Can you get history of this region from master log and pastebin it
> (paste result of a grep of 84162fcdf083fe4736c39571223cb029 to
> pastebin)?
> St.Ack
>
> On Tue, Sep 28, 2010 at 10:53 AM, Jack Levin <ma...@gmail.com> wrote:
>> I am using rest and getting following errors on some insertions:
>>
>>
>> 2010-09-28 10:51:07,215 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
>> row out of range for HRegion
>> img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029.,
>> startKey='samp754.png', getEndKey()='screenshot205.png',
>> row='sf6c.jpg'
>>        at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967)
>>        at org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030)
>>        at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120)
>>        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367)
>>        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321)
>>        at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
>>        at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
>>        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
>>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
>>
>> And from REST log:
>>
>>
>> 2010-09-28 10:50:26,308 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Failed
>> execution of all on mtab5.prod.imageshack.com:60020
>> org.apache.hadoop.hbase.regionserver.WrongRegionException:
>> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
>> row out of range for HRegion
>> img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029.,
>> startKey='samp754.png', getEndKey()='screenshot205.png',
>> row='sf6c.jpg'
>>        at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967)
>>        at org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030)
>>        at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120)
>>        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367)
>>        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321)
>>        at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
>>        at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
>>        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
>>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
>>
>>        at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown Source)
>>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>        at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.translateException(HConnectionManager.java:1552)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithoutRetries(HConnectionManager.java:1162)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1530)
>>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1528)
>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>> It looks like even after restarting REST I get the same error, could
>> this been that zookeepers cached the wrong location of the region and
>> its not on the same regionserver?   Are there other methods to debug?
>>
>> -Jack
>>
>

Re: getting region PUT errors.

Posted by Stack <st...@duboce.net>.
Can you get history of this region from master log and pastebin it
(paste result of a grep of 84162fcdf083fe4736c39571223cb029 to
pastebin)?
St.Ack

On Tue, Sep 28, 2010 at 10:53 AM, Jack Levin <ma...@gmail.com> wrote:
> I am using rest and getting following errors on some insertions:
>
>
> 2010-09-28 10:51:07,215 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
> row out of range for HRegion
> img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029.,
> startKey='samp754.png', getEndKey()='screenshot205.png',
> row='sf6c.jpg'
>        at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967)
>        at org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030)
>        at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120)
>        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367)
>        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
>        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
>
> And from REST log:
>
>
> 2010-09-28 10:50:26,308 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Failed
> execution of all on mtab5.prod.imageshack.com:60020
> org.apache.hadoop.hbase.regionserver.WrongRegionException:
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested
> row out of range for HRegion
> img833,samp754.png,1285522187411.84162fcdf083fe4736c39571223cb029.,
> startKey='samp754.png', getEndKey()='screenshot205.png',
> row='sf6c.jpg'
>        at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1967)
>        at org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:2030)
>        at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:2120)
>        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1367)
>        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1321)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1803)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.multiPut(HRegionServer.java:2464)
>        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:560)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1027)
>
>        at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown Source)
>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.translateException(HConnectionManager.java:1552)
>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithoutRetries(HConnectionManager.java:1162)
>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1530)
>        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$6.call(HConnectionManager.java:1528)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:619)
>
> It looks like even after restarting REST I get the same error, could
> this been that zookeepers cached the wrong location of the region and
> its not on the same regionserver?   Are there other methods to debug?
>
> -Jack
>