You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by mukund murrali <mu...@gmail.com> on 2017/07/05 08:00:50 UTC
Problem during hbase read
Hi,
Recently we faced an issue in our HBase Cluster and following is the
exception trace in client. Any idea what could be the cause and how to
rectify this?
RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
java.io.IOException: java.io.IOException: Could not reseek
StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreFileReader$1@2e508d2b,
cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
1498885978888/Maximum/vlen=0/seqid=0] to key xxxxxxx/dataCF:5f1e00cd-cbea-
45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseek(StoreFileScanner.java:217)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
enforceSeek(StoreFileScanner.java:408)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
pollRealKV(KeyValueHeap.java:363)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
next(KeyValueHeap.java:123)
at org.apache.hadoop.hbase.regionserver.StoreScanner.
next(StoreScanner.java:588)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
next(KeyValueHeap.java:150)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
populateResult(HRegion.java:5731)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
nextInternal(HRegion.java:5894)
at org.apache.hadoop.hbase.regionserver.HRegion$
RegionScannerImpl.nextRaw(HRegion.java:5668)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(
HRegion.java:5645)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(
HRegion.java:5631)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.
get(RSRpcServices.java:2049)
at org.apache.hadoop.hbase.protobuf.generated.
ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: On-disk size without header provided is
36901, but block header contains 859321901. Block offset: 440164791, data
starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
at org.apache.hadoop.hbase.io.hfile.HFileBlock.
validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
HFileBlock.java:88)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
readBlockDataInternal(HFileBlock.java:1665)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
readBlockData(HFileBlock.java:1532)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
HFileReaderV2.java:452)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.
loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
AbstractScannerV2.seekTo(HFileReaderV2.java:649)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
reseekTo(HalfStoreFileReader.java:293)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseekAtOrAfter(StoreFileScanner.java:288)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
reseek(StoreFileScanner.java:200)
... 19 more
ThanksU
Regards,
Mukund Murrali
Re: Problem during hbase read
Posted by mukund murrali <mu...@gmail.com>.
We are again facing the issue for a different region. Any suggestions will
be highly helpful
Thanks
On Thu, 6 Jul 2017 at 11:00 AM, mukund murrali <mu...@gmail.com>
wrote:
> Hi Stack
>
> We are currently using hbase-1.2.5(Stable). I was not able to read the
> file using HFile tool. It throws the same error. But I downloaded the file
> using dfs commands. The file had very less data. Ya I saw that comment.
> More details on the issue and fix will be helpful. Adding him to this
> thread.
>
> Regards,
>
> Mukund Murrali
>
> On Wed, Jul 5, 2017 at 8:30 PM, Stack <st...@duboce.net> wrote:
>
>> (What version of hbase?)
>>
>> Unfortunately the exception does not print the file name. Can you figure
>> which file or region it is? Could try reading content w/ hfile tool to see
>> if it complains also.
>>
>> Problem is:
>>
>> On-disk size without header provided is
>> 36901, but block header contains 859321901. Block offset: 440164791, data
>> starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
>> at org.apache.hadoop.hbase.io.hfile.HFileBlock.
>> validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
>>
>> I see the above exception in HBASE-13830 where Nils perhaps fixes it by
>> upgrading hbase (see end comment).
>>
>> St.Ack
>>
>>
>>
>> On Wed, Jul 5, 2017 at 1:00 AM, mukund murrali <mu...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > Recently we faced an issue in our HBase Cluster and following is the
>> > exception trace in client. Any idea what could be the cause and how to
>> > rectify this?
>> >
>> > RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
>> > java.io.IOException: java.io.IOException: Could not reseek
>> > StoreFileScanner[org.apache.hadoop.hbase.io
>> .HalfStoreFileReader$1@2e508d2b
>> > ,
>> > cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
>> > 1498885978888/Maximum/vlen=0/seqid=0] to key
>> xxxxxxx/dataCF:5f1e00cd-cbea-
>> > 45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
>> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
>> > reseek(StoreFileScanner.java:217)
>> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
>> > enforceSeek(StoreFileScanner.java:408)
>> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
>> > pollRealKV(KeyValueHeap.java:363)
>> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
>> > next(KeyValueHeap.java:123)
>> > at org.apache.hadoop.hbase.regionserver.StoreScanner.
>> > next(StoreScanner.java:588)
>> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
>> > next(KeyValueHeap.java:150)
>> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
>> > populateResult(HRegion.java:5731)
>> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
>> > nextInternal(HRegion.java:5894)
>> > at org.apache.hadoop.hbase.regionserver.HRegion$
>> > RegionScannerImpl.nextRaw(HRegion.java:5668)
>> > at org.apache.hadoop.hbase.regionserver.HRegion$
>> > RegionScannerImpl.next(
>> > HRegion.java:5645)
>> > at org.apache.hadoop.hbase.regionserver.HRegion$
>> > RegionScannerImpl.next(
>> > HRegion.java:5631)
>> > at
>> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
>> > at
>> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
>> > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
>> > get(RSRpcServices.java:2049)
>> > at org.apache.hadoop.hbase.protobuf.generated.
>> > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
>> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
>> > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
>> > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
>> > RpcExecutor.java:133)
>> > at
>> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Caused by: java.io.IOException: On-disk size without header provided is
>> > 36901, but block header contains 859321901. Block offset: 440164791,
>> data
>> > starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
>> > at org.apache.hadoop.hbase.io.hfile.HFileBlock.
>> > validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
>> > at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
>> > HFileBlock.java:88)
>> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
>> > readBlockDataInternal(HFileBlock.java:1665)
>> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
>> > readBlockData(HFileBlock.java:1532)
>> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
>> > HFileReaderV2.java:452)
>> > at org.apache.hadoop.hbase.io
>> .hfile.HFileBlockIndex$BlockIndexReader.
>> > loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
>> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
>> > AbstractScannerV2.seekTo(HFileReaderV2.java:649)
>> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
>> > AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
>> > at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
>> > reseekTo(HalfStoreFileReader.java:293)
>> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
>> > reseekAtOrAfter(StoreFileScanner.java:288)
>> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
>> > reseek(StoreFileScanner.java:200)
>> > ... 19 more
>> >
>> > ThanksU
>> > Regards,
>> >
>> > Mukund Murrali
>> >
>>
>
> --
Regards,
Mukund Murrali
Re: Problem during hbase read
Posted by mukund murrali <mu...@gmail.com>.
Hi Stack
We are currently using hbase-1.2.5(Stable). I was not able to read the file
using HFile tool. It throws the same error. But I downloaded the file using
dfs commands. The file had very less data. Ya I saw that comment. More
details on the issue and fix will be helpful. Adding him to this thread.
Regards,
Mukund Murrali
On Wed, Jul 5, 2017 at 8:30 PM, Stack <st...@duboce.net> wrote:
> (What version of hbase?)
>
> Unfortunately the exception does not print the file name. Can you figure
> which file or region it is? Could try reading content w/ hfile tool to see
> if it complains also.
>
> Problem is:
>
> On-disk size without header provided is
> 36901, but block header contains 859321901. Block offset: 440164791, data
> starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
> at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
>
> I see the above exception in HBASE-13830 where Nils perhaps fixes it by
> upgrading hbase (see end comment).
>
> St.Ack
>
>
>
> On Wed, Jul 5, 2017 at 1:00 AM, mukund murrali <mu...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Recently we faced an issue in our HBase Cluster and following is the
> > exception trace in client. Any idea what could be the cause and how to
> > rectify this?
> >
> > RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
> > java.io.IOException: java.io.IOException: Could not reseek
> > StoreFileScanner[org.apache.hadoop.hbase.io.
> HalfStoreFileReader$1@2e508d2b
> > ,
> > cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
> > 1498885978888/Maximum/vlen=0/seqid=0] to key
> xxxxxxx/dataCF:5f1e00cd-cbea-
> > 45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > reseek(StoreFileScanner.java:217)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > enforceSeek(StoreFileScanner.java:408)
> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > pollRealKV(KeyValueHeap.java:363)
> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > next(KeyValueHeap.java:123)
> > at org.apache.hadoop.hbase.regionserver.StoreScanner.
> > next(StoreScanner.java:588)
> > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> > next(KeyValueHeap.java:150)
> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > populateResult(HRegion.java:5731)
> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> > nextInternal(HRegion.java:5894)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.nextRaw(HRegion.java:5668)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.next(
> > HRegion.java:5645)
> > at org.apache.hadoop.hbase.regionserver.HRegion$
> > RegionScannerImpl.next(
> > HRegion.java:5631)
> > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> HRegion.java:6829)
> > at org.apache.hadoop.hbase.regionserver.HRegion.get(
> HRegion.java:6807)
> > at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > get(RSRpcServices.java:2049)
> > at org.apache.hadoop.hbase.protobuf.generated.
> > ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> > RpcExecutor.java:133)
> > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> java:108)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.io.IOException: On-disk size without header provided is
> > 36901, but block header contains 859321901. Block offset: 440164791, data
> > starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> > validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
> > HFileBlock.java:88)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> > readBlockDataInternal(HFileBlock.java:1665)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> > readBlockData(HFileBlock.java:1532)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
> > HFileReaderV2.java:452)
> > at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$
> BlockIndexReader.
> > loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> > AbstractScannerV2.seekTo(HFileReaderV2.java:649)
> > at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> > AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
> > at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
> > reseekTo(HalfStoreFileReader.java:293)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > reseekAtOrAfter(StoreFileScanner.java:288)
> > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> > reseek(StoreFileScanner.java:200)
> > ... 19 more
> >
> > ThanksU
> > Regards,
> >
> > Mukund Murrali
> >
>
Re: Problem during hbase read
Posted by Stack <st...@duboce.net>.
(What version of hbase?)
Unfortunately the exception does not print the file name. Can you figure
which file or region it is? Could try reading content w/ hfile tool to see
if it complains also.
Problem is:
On-disk size without header provided is
36901, but block header contains 859321901. Block offset: 440164791, data
starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
at org.apache.hadoop.hbase.io.hfile.HFileBlock.
validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
I see the above exception in HBASE-13830 where Nils perhaps fixes it by
upgrading hbase (see end comment).
St.Ack
On Wed, Jul 5, 2017 at 1:00 AM, mukund murrali <mu...@gmail.com>
wrote:
> Hi,
>
> Recently we faced an issue in our HBase Cluster and following is the
> exception trace in client. Any idea what could be the cause and how to
> rectify this?
>
> RpcRetryingCaller{globalStartTime=1499165478610, pause=100, retries=35},
> java.io.IOException: java.io.IOException: Could not reseek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoreFileReader$1@2e508d2b
> ,
> cur=xxxxxxx/dataCF:5f1e00cd-cbea-45d6-8542-72b8b00960ff/
> 1498885978888/Maximum/vlen=0/seqid=0] to key xxxxxxx/dataCF:5f1e00cd-cbea-
> 45d6-8542-72b8b00960ff/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseek(StoreFileScanner.java:217)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> enforceSeek(StoreFileScanner.java:408)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> pollRealKV(KeyValueHeap.java:363)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> next(KeyValueHeap.java:123)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.
> next(StoreScanner.java:588)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.
> next(KeyValueHeap.java:150)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> populateResult(HRegion.java:5731)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.
> nextInternal(HRegion.java:5894)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.nextRaw(HRegion.java:5668)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.next(
> HRegion.java:5645)
> at org.apache.hadoop.hbase.regionserver.HRegion$
> RegionScannerImpl.next(
> HRegion.java:5631)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6829)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6807)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> get(RSRpcServices.java:2049)
> at org.apache.hadoop.hbase.protobuf.generated.
> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: On-disk size without header provided is
> 36901, but block header contains 859321901. Block offset: 440164791, data
> starts with: 1-9d66-4386-83bc-48b335630d0e\x88\x01\x01
> at org.apache.hadoop.hbase.io.hfile.HFileBlock.
> validateOnDiskSizeWithoutHeader(HFileBlock.java:521)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock.access$700(
> HFileBlock.java:88)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> readBlockDataInternal(HFileBlock.java:1665)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.
> readBlockData(HFileBlock.java:1532)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(
> HFileReaderV2.java:452)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.
> loadDataBlockWithScanInfo(HFileBlockIndex.java:271)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> AbstractScannerV2.seekTo(HFileReaderV2.java:649)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$
> AbstractScannerV2.reseekTo(HFileReaderV2.java:629)
> at org.apache.hadoop.hbase.io.HalfStoreFileReader$1.
> reseekTo(HalfStoreFileReader.java:293)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseekAtOrAfter(StoreFileScanner.java:288)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.
> reseek(StoreFileScanner.java:200)
> ... 19 more
>
> ThanksU
> Regards,
>
> Mukund Murrali
>