You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Graceline Abigail Prem Kumar <pg...@gmail.com> on 2017/07/06 05:38:53 UTC
Exception in HBase get
Hi
We have got an IllegalStateException during HBase get operation. And we're
not sure about the cause. Here is the exception trace. What could the
problem be?
Caused by: java.lang.IllegalStateException: isDelete failed:
deleteBuffer=22, qualifier=21, timestamp=1487055525513, comparison result:
1
at
org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:147)
at
org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:395)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:529)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5731)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5894)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5668)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5645)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5631)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2049)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
... 4 more
Mon Jul 03 04:34:07 PDT 2017,
RpcRetryingCaller{globalStartTime=1499081647183, pause=100, retries=35},
java.io.IOException: java.io.IOException: isDelete failed: deleteBuffer=22,
qualifier=21, timestamp=1487055525513, comparison result: 1
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2239)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
Regards,
Graceline Abigail P
Re: Exception in HBase get
Posted by Ted Yu <yu...@gmail.com>.
Which hbase release are you using ?
Looking at ScanDeleteTracker.java from branch-1.2, the line number seems to
match the trace:
int ret = Bytes.compareTo(deleteBuffer, deleteOffset, deleteLength,
cell.getQualifierArray(), qualifierOffset, qualifierLength);
...
} else {
throw new IllegalStateException("isDelete failed: deleteBuffer="
The given timestamp (1487055525513) is for Tuesday, February 14, 2017.
Do you see any other occurrence of this exception ?
Cheers
On Wed, Jul 5, 2017 at 10:38 PM, Graceline Abigail Prem Kumar <
pgabigail95@gmail.com> wrote:
> Hi
>
> We have got an IllegalStateException during HBase get operation. And we're
> not sure about the cause. Here is the exception trace. What could the
> problem be?
>
> Caused by: java.lang.IllegalStateException: isDelete failed:
> deleteBuffer=22, qualifier=21, timestamp=1487055525513, comparison result:
> 1
> at
> org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(
> ScanDeleteTracker.java:147)
> at
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
> match(ScanQueryMatcher.java:395)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.
> next(StoreScanner.java:529)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> next(KeyValueHeap.java:150)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> populateResult(HRegion.java:5731)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> nextInternal(HRegion.java:5894)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(
> HRegion.java:5668)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(
> HRegion.java:5645)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(
> HRegion.java:5631)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> get(RSRpcServices.java:2049)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.
> callBlockingMethod(ClientProtos.java:33644)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> ... 4 more
>
> Mon Jul 03 04:34:07 PDT 2017,
> RpcRetryingCaller{globalStartTime=1499081647183, pause=100, retries=35},
> java.io.IOException: java.io.IOException: isDelete failed: deleteBuffer=22,
> qualifier=21, timestamp=1487055525513, comparison result: 1
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2239)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> Regards,
> Graceline Abigail P
>
Re: Exception in HBase get
Posted by mukund murrali <mu...@gmail.com>.
Hi
Few more pointers in the issue. Let's say there are two cells with column
keys as A and B for same row. These two cells are present in two different
files. Cell B is deleted recently. While scanning StoreFileScanner has read
B first then reads A. Though the lexographical sorting is preserved in
individual files why does the scanner reads deleted cell B first? This is
the issue causing problem in reads. Because of this, we have commented the
else part which throws IllegalStateException. Will this commenting cause
any issues(though no issues till now). Please someone enlighten more on
this issue please. We are using 1.2.5 stable version
On Mon, 10 Jul 2017 at 7:29 PM, mukund murrali <mu...@gmail.com>
wrote:
> +dev
>
> Regards,
>
> Mukund Murrali
>
> On Mon, Jul 10, 2017 at 1:17 PM, mukund murrali <mu...@gmail.com>
> wrote:
>
>> I know the row key. So in what way it will be helpful in analyzing this
>> issue?
>>
>> Regards,
>>
>> Mukund Murrali
>>
>> On Mon, Jul 10, 2017 at 5:32 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Can you find the hfile where this exception happens when doing get /
>>> scan ?
>>>
>>> Unfortunately the log didn't contain row key.
>>> Here is small change which would log row key:
>>>
>>> https://pastebin.com/XU5hCLXq
>>>
>>> Cheers
>>>
>>> On Thu, Jul 6, 2017 at 10:17 PM, Graceline Abigail Prem Kumar <
>>> pgabigail95@gmail.com> wrote:
>>>
>>> > Hi
>>> >
>>> > We are currently used Hbase 1.2.5. This exception occurs frequently,
>>> both
>>> > on the server and on the client side.
>>> >
>>> > Regards,
>>> > Graceline Abigail P
>>> >
>>> > On Thu, Jul 6, 2017 at 11:08 AM, Graceline Abigail Prem Kumar <
>>> > pgabigail95@gmail.com> wrote:
>>> >
>>> > > Hi
>>> > >
>>> > > We have got an IllegalStateException during HBase get operation. And
>>> > we're
>>> > > not sure about the cause. Here is the exception trace. What could the
>>> > > problem be?
>>> > >
>>> > > Caused by: java.lang.IllegalStateException: isDelete failed:
>>> > > deleteBuffer=22, qualifier=21, timestamp=1487055525513, comparison
>>> > result:
>>> > > 1
>>> > > at
>>> org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(
>>> > > ScanDeleteTracker.java:147)
>>> > > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
>>> > > match(ScanQueryMatcher.java:395)
>>> > > at org.apache.hadoop.hbase.regionserver.StoreScanner.
>>> > > next(StoreScanner.java:529)
>>> > > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
>>> > > next(KeyValueHeap.java:150)
>>> > > at
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
>>> > > populateResult(HRegion.java:5731)
>>> > > at
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
>>> > > nextInternal(HRegion.java:5894)
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion$
>>> > > RegionScannerImpl.nextRaw(HRegion.java:5668)
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion$
>>> > > RegionScannerImpl.next(HRegion.java:5645)
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion$
>>> > > RegionScannerImpl.next(HRegion.java:5631)
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion.get(
>>> > > HRegion.java:6829)
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion.get(
>>> > > HRegion.java:6807)
>>> > > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
>>> > > get(RSRpcServices.java:2049)
>>> > > at org.apache.hadoop.hbase.protobuf.generated.
>>> > >
>>> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
>>> > > at
>>> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
>>> > > ... 4 more
>>> > >
>>> > > Mon Jul 03 04:34:07 PDT 2017, RpcRetryingCaller{
>>> > globalStartTime=1499081647183,
>>> > > pause=100, retries=35}, java.io.IOException: java.io.IOException:
>>> > isDelete
>>> > > failed: deleteBuffer=22, qualifier=21, timestamp=1487055525513,
>>> > comparison
>>> > > result: 1
>>> > > at
>>> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2239)
>>> > > at org.apache.hadoop.hbase.ipc.Ca
>>> llRunner.run(CallRunner.java:112)
>>> > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
>>> > > RpcExecutor.java:133)
>>> > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
>>> > > java:108)
>>> > > at java.lang.Thread.run(Thread.java:745)
>>> > >
>>> > > Regards,
>>> > > Graceline Abigail P
>>> > >
>>> >
>>>
>>
>>
> --
Regards,
Mukund Murrali
Re: Exception in HBase get
Posted by mukund murrali <mu...@gmail.com>.
Hi
Few more pointers in the issue. Let's say there are two cells with column
keys as A and B for same row. These two cells are present in two different
files. Cell B is deleted recently. While scanning StoreFileScanner has read
B first then reads A. Though the lexographical sorting is preserved in
individual files why does the scanner reads deleted cell B first? This is
the issue causing problem in reads. Because of this, we have commented the
else part which throws IllegalStateException. Will this commenting cause
any issues(though no issues till now). Please someone enlighten more on
this issue please. We are using 1.2.5 stable version
On Mon, 10 Jul 2017 at 7:29 PM, mukund murrali <mu...@gmail.com>
wrote:
> +dev
>
> Regards,
>
> Mukund Murrali
>
> On Mon, Jul 10, 2017 at 1:17 PM, mukund murrali <mu...@gmail.com>
> wrote:
>
>> I know the row key. So in what way it will be helpful in analyzing this
>> issue?
>>
>> Regards,
>>
>> Mukund Murrali
>>
>> On Mon, Jul 10, 2017 at 5:32 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Can you find the hfile where this exception happens when doing get /
>>> scan ?
>>>
>>> Unfortunately the log didn't contain row key.
>>> Here is small change which would log row key:
>>>
>>> https://pastebin.com/XU5hCLXq
>>>
>>> Cheers
>>>
>>> On Thu, Jul 6, 2017 at 10:17 PM, Graceline Abigail Prem Kumar <
>>> pgabigail95@gmail.com> wrote:
>>>
>>> > Hi
>>> >
>>> > We are currently used Hbase 1.2.5. This exception occurs frequently,
>>> both
>>> > on the server and on the client side.
>>> >
>>> > Regards,
>>> > Graceline Abigail P
>>> >
>>> > On Thu, Jul 6, 2017 at 11:08 AM, Graceline Abigail Prem Kumar <
>>> > pgabigail95@gmail.com> wrote:
>>> >
>>> > > Hi
>>> > >
>>> > > We have got an IllegalStateException during HBase get operation. And
>>> > we're
>>> > > not sure about the cause. Here is the exception trace. What could the
>>> > > problem be?
>>> > >
>>> > > Caused by: java.lang.IllegalStateException: isDelete failed:
>>> > > deleteBuffer=22, qualifier=21, timestamp=1487055525513, comparison
>>> > result:
>>> > > 1
>>> > > at
>>> org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(
>>> > > ScanDeleteTracker.java:147)
>>> > > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
>>> > > match(ScanQueryMatcher.java:395)
>>> > > at org.apache.hadoop.hbase.regionserver.StoreScanner.
>>> > > next(StoreScanner.java:529)
>>> > > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
>>> > > next(KeyValueHeap.java:150)
>>> > > at
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
>>> > > populateResult(HRegion.java:5731)
>>> > > at
>>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
>>> > > nextInternal(HRegion.java:5894)
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion$
>>> > > RegionScannerImpl.nextRaw(HRegion.java:5668)
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion$
>>> > > RegionScannerImpl.next(HRegion.java:5645)
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion$
>>> > > RegionScannerImpl.next(HRegion.java:5631)
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion.get(
>>> > > HRegion.java:6829)
>>> > > at org.apache.hadoop.hbase.regionserver.HRegion.get(
>>> > > HRegion.java:6807)
>>> > > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
>>> > > get(RSRpcServices.java:2049)
>>> > > at org.apache.hadoop.hbase.protobuf.generated.
>>> > >
>>> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
>>> > > at
>>> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
>>> > > ... 4 more
>>> > >
>>> > > Mon Jul 03 04:34:07 PDT 2017, RpcRetryingCaller{
>>> > globalStartTime=1499081647183,
>>> > > pause=100, retries=35}, java.io.IOException: java.io.IOException:
>>> > isDelete
>>> > > failed: deleteBuffer=22, qualifier=21, timestamp=1487055525513,
>>> > comparison
>>> > > result: 1
>>> > > at
>>> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2239)
>>> > > at org.apache.hadoop.hbase.ipc.Ca
>>> llRunner.run(CallRunner.java:112)
>>> > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
>>> > > RpcExecutor.java:133)
>>> > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
>>> > > java:108)
>>> > > at java.lang.Thread.run(Thread.java:745)
>>> > >
>>> > > Regards,
>>> > > Graceline Abigail P
>>> > >
>>> >
>>>
>>
>>
> --
Regards,
Mukund Murrali
Re: Exception in HBase get
Posted by mukund murrali <mu...@gmail.com>.
+dev
Regards,
Mukund Murrali
On Mon, Jul 10, 2017 at 1:17 PM, mukund murrali <mu...@gmail.com>
wrote:
> I know the row key. So in what way it will be helpful in analyzing this
> issue?
>
> Regards,
>
> Mukund Murrali
>
> On Mon, Jul 10, 2017 at 5:32 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Can you find the hfile where this exception happens when doing get / scan
>> ?
>>
>> Unfortunately the log didn't contain row key.
>> Here is small change which would log row key:
>>
>> https://pastebin.com/XU5hCLXq
>>
>> Cheers
>>
>> On Thu, Jul 6, 2017 at 10:17 PM, Graceline Abigail Prem Kumar <
>> pgabigail95@gmail.com> wrote:
>>
>> > Hi
>> >
>> > We are currently used Hbase 1.2.5. This exception occurs frequently,
>> both
>> > on the server and on the client side.
>> >
>> > Regards,
>> > Graceline Abigail P
>> >
>> > On Thu, Jul 6, 2017 at 11:08 AM, Graceline Abigail Prem Kumar <
>> > pgabigail95@gmail.com> wrote:
>> >
>> > > Hi
>> > >
>> > > We have got an IllegalStateException during HBase get operation. And
>> > we're
>> > > not sure about the cause. Here is the exception trace. What could the
>> > > problem be?
>> > >
>> > > Caused by: java.lang.IllegalStateException: isDelete failed:
>> > > deleteBuffer=22, qualifier=21, timestamp=1487055525513, comparison
>> > result:
>> > > 1
>> > > at org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.
>> isDeleted(
>> > > ScanDeleteTracker.java:147)
>> > > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
>> > > match(ScanQueryMatcher.java:395)
>> > > at org.apache.hadoop.hbase.regionserver.StoreScanner.
>> > > next(StoreScanner.java:529)
>> > > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
>> > > next(KeyValueHeap.java:150)
>> > > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerIm
>> pl.
>> > > populateResult(HRegion.java:5731)
>> > > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerIm
>> pl.
>> > > nextInternal(HRegion.java:5894)
>> > > at org.apache.hadoop.hbase.regionserver.HRegion$
>> > > RegionScannerImpl.nextRaw(HRegion.java:5668)
>> > > at org.apache.hadoop.hbase.regionserver.HRegion$
>> > > RegionScannerImpl.next(HRegion.java:5645)
>> > > at org.apache.hadoop.hbase.regionserver.HRegion$
>> > > RegionScannerImpl.next(HRegion.java:5631)
>> > > at org.apache.hadoop.hbase.regionserver.HRegion.get(
>> > > HRegion.java:6829)
>> > > at org.apache.hadoop.hbase.regionserver.HRegion.get(
>> > > HRegion.java:6807)
>> > > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
>> > > get(RSRpcServices.java:2049)
>> > > at org.apache.hadoop.hbase.protobuf.generated.
>> > > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos
>> .java:33644)
>> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:21
>> 88)
>> > > ... 4 more
>> > >
>> > > Mon Jul 03 04:34:07 PDT 2017, RpcRetryingCaller{
>> > globalStartTime=1499081647183,
>> > > pause=100, retries=35}, java.io.IOException: java.io.IOException:
>> > isDelete
>> > > failed: deleteBuffer=22, qualifier=21, timestamp=1487055525513,
>> > comparison
>> > > result: 1
>> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:22
>> 39)
>> > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:
>> 112)
>> > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
>> > > RpcExecutor.java:133)
>> > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
>> > > java:108)
>> > > at java.lang.Thread.run(Thread.java:745)
>> > >
>> > > Regards,
>> > > Graceline Abigail P
>> > >
>> >
>>
>
>
Re: Exception in HBase get
Posted by mukund murrali <mu...@gmail.com>.
I know the row key. So in what way it will be helpful in analyzing this
issue?
Regards,
Mukund Murrali
On Mon, Jul 10, 2017 at 5:32 AM, Ted Yu <yu...@gmail.com> wrote:
> Can you find the hfile where this exception happens when doing get / scan ?
>
> Unfortunately the log didn't contain row key.
> Here is small change which would log row key:
>
> https://pastebin.com/XU5hCLXq
>
> Cheers
>
> On Thu, Jul 6, 2017 at 10:17 PM, Graceline Abigail Prem Kumar <
> pgabigail95@gmail.com> wrote:
>
> > Hi
> >
> > We are currently used Hbase 1.2.5. This exception occurs frequently, both
> > on the server and on the client side.
> >
> > Regards,
> > Graceline Abigail P
> >
> > On Thu, Jul 6, 2017 at 11:08 AM, Graceline Abigail Prem Kumar <
> > pgabigail95@gmail.com> wrote:
> >
> > > Hi
> > >
> > > We have got an IllegalStateException during HBase get operation. And
> > we're
> > > not sure about the cause. Here is the exception trace. What could the
> > > problem be?
> > >
> > > Caused by: java.lang.IllegalStateException: isDelete failed:
> > > deleteBuffer=22, qualifier=21, timestamp=1487055525513, comparison
> > result:
> > > 1
> > > at org.apache.hadoop.hbase.regionserver.
> ScanDeleteTracker.isDeleted(
> > > ScanDeleteTracker.java:147)
> > > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
> > > match(ScanQueryMatcher.java:395)
> > > at org.apache.hadoop.hbase.regionserver.StoreScanner.
> > > next(StoreScanner.java:529)
> > > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > > next(KeyValueHeap.java:150)
> > > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > > populateResult(HRegion.java:5731)
> > > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > > nextInternal(HRegion.java:5894)
> > > at org.apache.hadoop.hbase.regionserver.HRegion$
> > > RegionScannerImpl.nextRaw(HRegion.java:5668)
> > > at org.apache.hadoop.hbase.regionserver.HRegion$
> > > RegionScannerImpl.next(HRegion.java:5645)
> > > at org.apache.hadoop.hbase.regionserver.HRegion$
> > > RegionScannerImpl.next(HRegion.java:5631)
> > > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> > > HRegion.java:6829)
> > > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> > > HRegion.java:6807)
> > > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > > get(RSRpcServices.java:2049)
> > > at org.apache.hadoop.hbase.protobuf.generated.
> > > ClientProtos$ClientService$2.callBlockingMethod(
> ClientProtos.java:33644)
> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> > > ... 4 more
> > >
> > > Mon Jul 03 04:34:07 PDT 2017, RpcRetryingCaller{
> > globalStartTime=1499081647183,
> > > pause=100, retries=35}, java.io.IOException: java.io.IOException:
> > isDelete
> > > failed: deleteBuffer=22, qualifier=21, timestamp=1487055525513,
> > comparison
> > > result: 1
> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2239)
> > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> > > RpcExecutor.java:133)
> > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> > > java:108)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > Regards,
> > > Graceline Abigail P
> > >
> >
>
Re: Exception in HBase get
Posted by Ted Yu <yu...@gmail.com>.
Can you find the hfile where this exception happens when doing get / scan ?
Unfortunately the log didn't contain row key.
Here is small change which would log row key:
https://pastebin.com/XU5hCLXq
Cheers
On Thu, Jul 6, 2017 at 10:17 PM, Graceline Abigail Prem Kumar <
pgabigail95@gmail.com> wrote:
> Hi
>
> We are currently used Hbase 1.2.5. This exception occurs frequently, both
> on the server and on the client side.
>
> Regards,
> Graceline Abigail P
>
> On Thu, Jul 6, 2017 at 11:08 AM, Graceline Abigail Prem Kumar <
> pgabigail95@gmail.com> wrote:
>
> > Hi
> >
> > We have got an IllegalStateException during HBase get operation. And
> we're
> > not sure about the cause. Here is the exception trace. What could the
> > problem be?
> >
> > Caused by: java.lang.IllegalStateException: isDelete failed:
> > deleteBuffer=22, qualifier=21, timestamp=1487055525513, comparison
> result:
> > 1
> > at org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(
> > ScanDeleteTracker.java:147)
> > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
> > match(ScanQueryMatcher.java:395)
> > at org.apache.hadoop.hbase.regionserver.StoreScanner.
> > next(StoreScanner.java:529)
> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > next(KeyValueHeap.java:150)
> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > populateResult(HRegion.java:5731)
> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > nextInternal(HRegion.java:5894)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.nextRaw(HRegion.java:5668)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.next(HRegion.java:5645)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.next(HRegion.java:5631)
> > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> > HRegion.java:6829)
> > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> > HRegion.java:6807)
> > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > get(RSRpcServices.java:2049)
> > at org.apache.hadoop.hbase.protobuf.generated.
> > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> > ... 4 more
> >
> > Mon Jul 03 04:34:07 PDT 2017, RpcRetryingCaller{
> globalStartTime=1499081647183,
> > pause=100, retries=35}, java.io.IOException: java.io.IOException:
> isDelete
> > failed: deleteBuffer=22, qualifier=21, timestamp=1487055525513,
> comparison
> > result: 1
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2239)
> > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> > RpcExecutor.java:133)
> > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> > java:108)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Regards,
> > Graceline Abigail P
> >
>
Re: Exception in HBase get
Posted by mukund murrali <mu...@gmail.com>.
Even we faced this issue recently. Not sure of the cause. And retries are
having things force for this case. Anyone has any idea of why this happens
Regards,
Mukund Murrali
On Fri, Jul 7, 2017 at 10:47 AM, Graceline Abigail Prem Kumar <
pgabigail95@gmail.com> wrote:
> Hi
>
> We are currently used Hbase 1.2.5. This exception occurs frequently, both
> on the server and on the client side.
>
> Regards,
> Graceline Abigail P
>
> On Thu, Jul 6, 2017 at 11:08 AM, Graceline Abigail Prem Kumar <
> pgabigail95@gmail.com> wrote:
>
> > Hi
> >
> > We have got an IllegalStateException during HBase get operation. And
> we're
> > not sure about the cause. Here is the exception trace. What could the
> > problem be?
> >
> > Caused by: java.lang.IllegalStateException: isDelete failed:
> > deleteBuffer=22, qualifier=21, timestamp=1487055525513, comparison
> result:
> > 1
> > at org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(
> > ScanDeleteTracker.java:147)
> > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
> > match(ScanQueryMatcher.java:395)
> > at org.apache.hadoop.hbase.regionserver.StoreScanner.
> > next(StoreScanner.java:529)
> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > next(KeyValueHeap.java:150)
> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > populateResult(HRegion.java:5731)
> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > nextInternal(HRegion.java:5894)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.nextRaw(HRegion.java:5668)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.next(HRegion.java:5645)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.next(HRegion.java:5631)
> > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> > HRegion.java:6829)
> > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> > HRegion.java:6807)
> > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > get(RSRpcServices.java:2049)
> > at org.apache.hadoop.hbase.protobuf.generated.
> > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> > ... 4 more
> >
> > Mon Jul 03 04:34:07 PDT 2017, RpcRetryingCaller{
> globalStartTime=1499081647183,
> > pause=100, retries=35}, java.io.IOException: java.io.IOException:
> isDelete
> > failed: deleteBuffer=22, qualifier=21, timestamp=1487055525513,
> comparison
> > result: 1
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2239)
> > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> > RpcExecutor.java:133)
> > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> > java:108)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Regards,
> > Graceline Abigail P
> >
>
Re: Exception in HBase get
Posted by Graceline Abigail Prem Kumar <pg...@gmail.com>.
Hi
We are currently used Hbase 1.2.5. This exception occurs frequently, both
on the server and on the client side.
Regards,
Graceline Abigail P
On Thu, Jul 6, 2017 at 11:08 AM, Graceline Abigail Prem Kumar <
pgabigail95@gmail.com> wrote:
> Hi
>
> We have got an IllegalStateException during HBase get operation. And we're
> not sure about the cause. Here is the exception trace. What could the
> problem be?
>
> Caused by: java.lang.IllegalStateException: isDelete failed:
> deleteBuffer=22, qualifier=21, timestamp=1487055525513, comparison result:
> 1
> at org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(
> ScanDeleteTracker.java:147)
> at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
> match(ScanQueryMatcher.java:395)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.
> next(StoreScanner.java:529)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> next(KeyValueHeap.java:150)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> populateResult(HRegion.java:5731)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> nextInternal(HRegion.java:5894)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.nextRaw(HRegion.java:5668)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.next(HRegion.java:5645)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.next(HRegion.java:5631)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(
> HRegion.java:6829)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(
> HRegion.java:6807)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> get(RSRpcServices.java:2049)
> at org.apache.hadoop.hbase.protobuf.generated.
> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> ... 4 more
>
> Mon Jul 03 04:34:07 PDT 2017, RpcRetryingCaller{globalStartTime=1499081647183,
> pause=100, retries=35}, java.io.IOException: java.io.IOException: isDelete
> failed: deleteBuffer=22, qualifier=21, timestamp=1487055525513, comparison
> result: 1
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2239)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> Regards,
> Graceline Abigail P
>