You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Shady Xu <sh...@gmail.com> on 2015/04/14 15:03:34 UTC

Requested row out of range for row lock while batch put operation

Our RegionServers kept getting this exception and were therefore unable to
response to other requests:

org.apache.hadoop.hbase.regionserver.HRegion: Failed getting lock in batch
put, row=...
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row
out of range for row lock on HRegion
test,1752868451_785774043,1422169937336.cb3ab092981ff22d51be923bcf7fa81c.,
startKey='1752868451_785774043', getEndKey()='186061631_989857395',
row='1716676661_1231903641'
        at
org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:3260)
        at
org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:3278)
        at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2213)
        at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2085)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4050)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3361)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3265)
        at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
        at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)

I guess maybe the batch put operation happened during the table split, Any
other possible causes and how to prevent it from happening again?

Re: Requested row out of range for row lock while batch put operation

Posted by Shady Xu <sh...@gmail.com>.
I didn't pre-split the table.

I checked the master log, there were no splits before the exception, but
during the exception, which lasted for dozens of minutes before I restarted
the region server, there was a split:

INFO org.apache.hadoop.hbase.master.SplitLogManager: dead splitlog workers
...
INFO org.apache.hadoop.hbase.master.SplitLogManager: started splitting 0
logs in []
INFO org.apache.hadoop.hbase.master.SplitLogManager: finished splitting
(more than or equal to) 0 bytes in 0 log files in [] in 0ms

However, the split seems to have done nothing. So maybe it's not caused by
a split, what else may be the cause, was it a bug?

P.S. I am using  hbase-0.96.1.1+cdh5.0.1+67 with CDH 5.0.1.

2015-04-14 21:40 GMT+08:00 Ted Yu <yu...@gmail.com>:

> Can you check master log to see if test,1752868451_785774043,1422169937336.
> cb3ab092981ff22d51be923bcf7fa81c. split around the time the exception was
> thrown ?
>
> Have you pre-split table test ?
>
> Which hbase release are you using ?
>
> Cheers
>
> On Tue, Apr 14, 2015 at 6:03 AM, Shady Xu <sh...@gmail.com> wrote:
>
> > Our RegionServers kept getting this exception and were therefore unable
> to
> > response to other requests:
> >
> > org.apache.hadoop.hbase.regionserver.HRegion: Failed getting lock in
> batch
> > put, row=...
> > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row
> > out of range for row lock on HRegion
> >
> test,1752868451_785774043,1422169937336.cb3ab092981ff22d51be923bcf7fa81c.,
> > startKey='1752868451_785774043', getEndKey()='186061631_989857395',
> > row='1716676661_1231903641'
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:3260)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:3278)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2213)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2085)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4050)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3361)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3265)
> >         at
> >
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935)
> >         at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
> >         at
> > org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
> >
> > I guess maybe the batch put operation happened during the table split,
> Any
> > other possible causes and how to prevent it from happening again?
> >
>

Re: Requested row out of range for row lock while batch put operation

Posted by Ted Yu <yu...@gmail.com>.
Can you check master log to see if test,1752868451_785774043,1422169937336.
cb3ab092981ff22d51be923bcf7fa81c. split around the time the exception was
thrown ?

Have you pre-split table test ?

Which hbase release are you using ?

Cheers

On Tue, Apr 14, 2015 at 6:03 AM, Shady Xu <sh...@gmail.com> wrote:

> Our RegionServers kept getting this exception and were therefore unable to
> response to other requests:
>
> org.apache.hadoop.hbase.regionserver.HRegion: Failed getting lock in batch
> put, row=...
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row
> out of range for row lock on HRegion
> test,1752868451_785774043,1422169937336.cb3ab092981ff22d51be923bcf7fa81c.,
> startKey='1752868451_785774043', getEndKey()='186061631_989857395',
> row='1716676661_1231903641'
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:3260)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:3278)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2213)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2085)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4050)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3361)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3265)
>         at
>
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935)
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
>         at
> org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
>
> I guess maybe the batch put operation happened during the table split, Any
> other possible causes and how to prevent it from happening again?
>