You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by William Shen <wi...@marinsoftware.com> on 2018/11/21 02:20:33 UTC
HBase Compaction Fails for Phoenix Table
Hi there,
We've encountered the following compaction failure(#1) for a Phoenix table,
and are not sure how to make sense of it. Using the HBase row key from the
error, we are able to query data directly from hbase shell, and by
examining the data, there arent anything immediately obvious about the data
as they seem to be stored consistent with the Phoenix data type (#2). When
querying Phoenix for the given row through sqline, the row would return if
only primary key columns are selected, and the query would not return if
non primary key columns are selected (#3).
Few questions hoping to find some help on:
a. Are we correct in understanding the error message to indicate an issue
with data for the row key (
\x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13)?
We are not sure what to make sense of the string "
1539019716378.3dcf2b1e057915feb74395d9711ba4ad." that is included with the
row key...
b. What is out of bound here? It's not apparently clear here what
StatisticsScanner and FastDiffDeltaEncoder are tripping over...
c. Is it normal for hbase shell to return some parts of the hex string as
ascii characters? We are seeing that in the row key as well as column name
encoding and value. We are not sure if that is causing any issues, or if
that was just a display issue that we can safely ignore.
*#1 Compaction Failure*
Compaction failed Request =
regionName=qa2.ADGROUPS,\x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13,1539019716378.3dcf2b1e057915feb74395d9711ba4ad.,
storeName=AG, fileCount=4, fileSize=316.0 M (315.8 M, 188.7 K, 6.8 K,
14.2 K), priority=1, time=40613533856170784
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkBounds(Buffer.java:567)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:458)
at org.apache.phoenix.schema.stats.StatisticsScanner.next(StatisticsScanner.java:69)
at org.apache.phoenix.schema.stats.StatisticsScanner.next(StatisticsScanner.java:76)
at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:334)
at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:106)
at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:131)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1245)
at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1852)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:529)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:566)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
*#2 Data on the HBase level*
hbase(main):002:0> get 'qa2.ADGROUPS',
"\x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13"
COLUMN CELL
AG:\x00\x00\x00\x00 timestamp=1539019457506, value=x
AG:\x80\x0F timestamp=1539019457506,
value=D:USA_AK:Nightmute:1456903:hotel
AG:\x80\x12 timestamp=1539019457506,
value=ACTIVE
AG:\x80\x13 timestamp=1539019457506, value=ADD
AG:\x80# timestamp=1539019457506,
value=\x80\x00\x00\x00\x00\x00'\x10
AG:\x80' timestamp=1539019457506,
value=\x80\x00\x01[\x97\x02\x02X\x00\x00\x00\x00
AG:\x80( timestamp=1539019457506,
value=\x80\x00\x00\x00\x00\x1D\xEC\xA4
AG:\x808 timestamp=1539019457506,
value=\x00
8 row(s) in 0.0510 seconds
*#3 Data in Phoenix via sqline*
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "cstId",
"cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
"id" = 30496531;
+----------+----------+--------+----------+----------+-----------+
| cstId | cltId | pubId | accId | cpgnId | id |
+----------+----------+--------+----------+----------+-----------+
| 2053276 | 1867077 | 4 | 1904409 | 4811421 | 30496531 |
+----------+----------+--------+----------+----------+-----------+
1 row selected (0.095 seconds)
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "cstId",
"cltId", "pubId", "accId","cpgnId", "id", "stts" from "qa2".adgroups
where "id" = 30496531;
*[hangs]*
Thanks in advance for your help!
Re: HBase Compaction Fails for Phoenix Table
Posted by William Shen <wi...@marinsoftware.com>.
Sorry for the delay (US Thanksgiving Holidays). We are on 1.2.0-cdh5.9.2
On Thu, Nov 22, 2018 at 6:54 PM Jaanai Zhang <cl...@gmail.com> wrote:
> It seems that is a bug about FastDiff in HBase, what' the version number
> of your HBase? I saw a similar question:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201504.mbox/%3C44434B16-823D-43EF-AAB1-337BFBA6F19A@5dlab.com%3E
>
>
> ----------------------------------------
> Jaanai Zhang
> Best regards!
>
>
>
> William Shen <wi...@marinsoftware.com> 于2018年11月22日周四 上午8:28写道：
>
>> Further narrowed down the HBase row, and was able to reproduce the
>> Exception in HBase Shell. The error also shows up in the region server log.
>> Would appreciate any pointers on next step to investigate the issue on the
>> HBase level. Thank you!
>>
>> hbase(main):007:0> get 'qa2.ADGROUPS', "\x05\x80\x00\x00\x00\x00\x1F\x54\x9C\x80\x00\x00\x00\x00\x1C\x7D\x45\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00\x4A\x64\x6F\x80\x00\x00\x00\x01\xD9\xDB\xCE"
>> COLUMN CELL
>>
>> ERROR: java.io.IOException
>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
>> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
>> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
>> Caused by: java.lang.IndexOutOfBoundsException
>> at java.nio.Buffer.checkBounds(Buffer.java:567)
>> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
>> at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
>> at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
>> at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
>> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
>> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
>> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
>> at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
>> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
>> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5706)
>> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5865)
>> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5643)
>> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5620)
>> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5606)
>> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6801)
>> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6779)
>> at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2029)
>> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
>> ... 3 more
>>
>>
>> On Wed, Nov 21, 2018 at 11:19 AM William Shen <wi...@marinsoftware.com>
>> wrote:
>>
> Hi Jaanai,
>>>
>>> We are running Phoenix 4.13, and actually upon investigating the issue
>>> more, we are not so sure if it is that row that's having issue. It seems
>>> like the data issue might be on a different row than the one HBase
>>> compaction is complaining about because when we do a point look up, the
>>> data comes back, but it does not return if it is a range scan (#1). Not
>>> totally sure why HBase complained about that row when the failure is
>>> somewhere else...
>>>
>>> I don't have the specific Phoenix DDL handy, but here is describe in
>>> HBase (#2), and also selective info from SYSTEMS.CATALOGS (#3). Hopefully
>>> that's helpful to make sense of the data types / keys, etc. Is there
>>> anything specific that you want to see?
>>>
>>>
>>>
>>> *#1: Range scan vs point look up*
>>>
>>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
>>> "cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
>>> "cstId" = 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4
>>> and "accId" = 1904409 and "cpgnId" = 4811421;
>>>
>>>
>>> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>>
>>> | PLAN
>>> | EST_BYTES_READ | EST_ROWS_READ |
>>> EST_INFO_TS |
>>>
>>>
>>> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>>
>>> | CLIENT 1-CHUNK 1 ROWS 8052 BYTES PARALLEL 1-WAY ROUND ROBIN POINT
>>> LOOKUP ON 1 KEY OVER qa2.ADGROUPS | 8052 | 1 |
>>> 1542827075237 |
>>>
>>>
>>> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>>
>>> 1 row selected (0.048 seconds)
>>>
>>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
>>> "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "cstId"
>>> = 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4 and
>>> "accId" = 1904409 and "cpgnId" = 4811421;
>>>
>>>
>>> +---------+----------+----------+--------+----------+----------+-----------+
>>>
>>> | stts | cstId | cltId | pubId | accId | cpgnId | id
>>> |
>>>
>>>
>>> +---------+----------+----------+--------+----------+----------+-----------+
>>>
>>> | ACTIVE | 2053276 | 1867077 | 4 | 1904409 | 4811421 |
>>> 30496531 |
>>>
>>>
>>> +---------+----------+----------+--------+----------+----------+-----------+
>>>
>>> 1 row selected (0.072 seconds)
>>>
>>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
>>> "cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
>>> "id" = 30496531;
>>>
>>>
>>> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>>
>>> | PLAN
>>> | EST_BYTES_READ |
>>> EST_ROWS_READ | EST_INFO_TS |
>>>
>>>
>>> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>>
>>> | CLIENT 364-CHUNK 36227938 ROWS 34037631025 BYTES PARALLEL 10-WAY
>>> ROUND ROBIN RANGE SCAN OVER qa2.ADGROUPS [0] - [9] | 34037631025 | 36227938
>>> | 1542823263107 |
>>>
>>> | SERVER FILTER BY "id" = 30496531
>>> | 34037631025 | 36227938
>>> | 1542823263107 |
>>>
>>>
>>> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>>
>>> 2 rows selected (0.063 seconds)
>>>
>>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
>>> "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "id" =
>>> 30496531;
>>>
>>> *[Hangs]*
>>>
>>> #2
>>>
>>> hbase(main):001:0> describe "qa2.ADGROUPS"
>>>
>>> Table qa2.ADGROUPS is ENABLED
>>>
>>>
>>>
>>>
>>> qa2.ADGROUPS, {TABLE_ATTRIBUTES => {coprocessor$1 =>
>>> '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|',
>>> coprocessor$2 =>
>>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
>>> coprocessor$
>>>
>>> 3 =>
>>> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
>>> coprocessor$4 =>
>>> '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
>>> coprocessor$5 => '|org.apache.phoenix.hbase.index.Index
>>>
>>> er|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'}
>>>
>>>
>>> COLUMN FAMILIES DESCRIPTION
>>>
>>>
>>>
>>>
>>> {NAME => 'AG', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY =>
>>> 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIFF',
>>> TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE =>
>>> 'true'
>>>
>>> , BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
>>>
>>>
>>>
>>>
>>> 1 row(s) in 0.4210 seconds
>>>
>>> #3
>>>
>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select KEY_SEQ,
>>> ORDINAL_POSITION, COLUMN_QUALIFIER, DATA_TYPE from SYSTEM.CATALOG
>>> where TABLE_SCHEM = 'qa2' and TABLE_NAME = 'ADGROUPS' order by
>>> ORDINAL_POSITION ;
>>>
>>> +----------+-------------------+-------------------+------------+
>>>
>>> | KEY_SEQ | ORDINAL_POSITION | COLUMN_QUALIFIER | DATA_TYPE |
>>>
>>> +----------+-------------------+-------------------+------------+
>>>
>>> | null | null | | null |
>>>
>>> | null | null | | null |
>>>
>>> | null | null | | null |
>>>
>>> | null | null | | null |
>>>
>>> | 1 | 1 | | -5 |
>>>
>>> | 2 | 2 | | -5 |
>>>
>>> | 3 | 3 | | 13 |
>>>
>>> | 4 | 4 | | -5 |
>>>
>>> | 5 | 5 | | -5 |
>>>
>>> | 6 | 6 | | -5 |
>>>
>>> | null | 7 | 800b | 12 |
>>>
>>> | null | 8 | 800c | -5 |
>>>
>>> | null | 9 | 800d | 2995 |
>>>
>>> | null | 10 | 800e | -5 |
>>>
>>> | null | 11 | 800f | 12 |
>>>
>>> | null | 12 | 8010 | 1 |
>>>
>>> | null | 13 | 8011 | 12 |
>>>
>>> | null | 14 | 8012 | 12 |
>>>
>>> | null | 15 | 8013 | 12 |
>>>
>>> | null | 16 | 8014 | 12 |
>>>
>>> | null | 17 | 8015 | 93 |
>>>
>>> | null | 18 | 8016 | 93 |
>>>
>>> | null | 19 | 8017 | 12 |
>>>
>>> | null | 20 | 8018 | -5 |
>>>
>>> | null | 21 | 8019 | 12 |
>>>
>>> | null | 22 | 801a | 12 |
>>>
>>> | null | 23 | 801b | -3 |
>>>
>>> | null | 24 | 801c | 12 |
>>>
>>> | null | 25 | 801d | 12 |
>>>
>>> | null | 26 | 801e | 12 |
>>>
>>> | null | 27 | 801f | 12 |
>>>
>>> | null | 28 | 8020 | -5 |
>>>
>>> | null | 29 | 8021 | 12 |
>>>
>>> | null | 30 | 8022 | 12 |
>>>
>>> | null | 31 | 8023 | -5 |
>>>
>>> | null | 32 | 8024 | 16 |
>>>
>>> | null | 33 | 8025 | 12 |
>>>
>>> | null | 34 | 8026 | -3 |
>>>
>>> | null | 35 | 8027 | 93 |
>>>
>>> | null | 36 | 8028 | -5 |
>>>
>>> | null | 37 | 8029 | 93 |
>>>
>>> | null | 38 | 802a | -5 |
>>>
>>> | null | 39 | 802b | 16 |
>>>
>>> | null | 40 | 802c | 12 |
>>>
>>> | null | 41 | 802d | 16 |
>>>
>> | null | 42 <span
>>> class="m_-4528002918779368419m_1004149540963671125m_-6947968
>>>
>>
Re: HBase Compaction Fails for Phoenix Table
Posted by Jaanai Zhang <cl...@gmail.com>.
It seems that is a bug about FastDiff in HBase, what' the version number
of your HBase? I saw a similar question:
http://mail-archives.apache.org/mod_mbox/hbase-user/201504.mbox/%3C44434B16-823D-43EF-AAB1-337BFBA6F19A@5dlab.com%3E
----------------------------------------
Jaanai Zhang
Best regards!
William Shen <wi...@marinsoftware.com> 于2018年11月22日周四 上午8:28写道：
> Further narrowed down the HBase row, and was able to reproduce the
> Exception in HBase Shell. The error also shows up in the region server log.
> Would appreciate any pointers on next step to investigate the issue on the
> HBase level. Thank you!
>
> hbase(main):007:0> get 'qa2.ADGROUPS', "\x05\x80\x00\x00\x00\x00\x1F\x54\x9C\x80\x00\x00\x00\x00\x1C\x7D\x45\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00\x4A\x64\x6F\x80\x00\x00\x00\x01\xD9\xDB\xCE"
> COLUMN CELL
>
> ERROR: java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkBounds(Buffer.java:567)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
> at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
> at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
> at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5706)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5865)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5643)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5620)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5606)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6801)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6779)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2029)
> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> ... 3 more
>
>
> On Wed, Nov 21, 2018 at 11:19 AM William Shen <wi...@marinsoftware.com>
> wrote:
>
>> Hi Jaanai,
>>
>> We are running Phoenix 4.13, and actually upon investigating the issue
>> more, we are not so sure if it is that row that's having issue. It seems
>> like the data issue might be on a different row than the one HBase
>> compaction is complaining about because when we do a point look up, the
>> data comes back, but it does not return if it is a range scan (#1). Not
>> totally sure why HBase complained about that row when the failure is
>> somewhere else...
>>
>> I don't have the specific Phoenix DDL handy, but here is describe in
>> HBase (#2), and also selective info from SYSTEMS.CATALOGS (#3). Hopefully
>> that's helpful to make sense of the data types / keys, etc. Is there
>> anything specific that you want to see?
>>
>>
>>
>> *#1: Range scan vs point look up*
>>
>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
>> "cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
>> "cstId" = 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4
>> and "accId" = 1904409 and "cpgnId" = 4811421;
>>
>>
>> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>
>> | PLAN
>> | EST_BYTES_READ | EST_ROWS_READ |
>> EST_INFO_TS |
>>
>>
>> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>
>> | CLIENT 1-CHUNK 1 ROWS 8052 BYTES PARALLEL 1-WAY ROUND ROBIN POINT
>> LOOKUP ON 1 KEY OVER qa2.ADGROUPS | 8052 | 1 |
>> 1542827075237 |
>>
>>
>> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>
>> 1 row selected (0.048 seconds)
>>
>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
>> "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "cstId"
>> = 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4 and
>> "accId" = 1904409 and "cpgnId" = 4811421;
>>
>>
>> +---------+----------+----------+--------+----------+----------+-----------+
>>
>> | stts | cstId | cltId | pubId | accId | cpgnId | id
>> |
>>
>>
>> +---------+----------+----------+--------+----------+----------+-----------+
>>
>> | ACTIVE | 2053276 | 1867077 | 4 | 1904409 | 4811421 | 30496531
>> |
>>
>>
>> +---------+----------+----------+--------+----------+----------+-----------+
>>
>> 1 row selected (0.072 seconds)
>>
>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
>> "cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
>> "id" = 30496531;
>>
>>
>> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>
>> | PLAN
>> | EST_BYTES_READ |
>> EST_ROWS_READ | EST_INFO_TS |
>>
>>
>> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>
>> | CLIENT 364-CHUNK 36227938 ROWS 34037631025 BYTES PARALLEL 10-WAY ROUND
>> ROBIN RANGE SCAN OVER qa2.ADGROUPS [0] - [9] | 34037631025 | 36227938
>> | 1542823263107 |
>>
>> | SERVER FILTER BY "id" = 30496531
>> | 34037631025 | 36227938
>> | 1542823263107 |
>>
>>
>> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>>
>> 2 rows selected (0.063 seconds)
>>
>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
>> "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "id" =
>> 30496531;
>>
>> *[Hangs]*
>>
>> #2
>>
>> hbase(main):001:0> describe "qa2.ADGROUPS"
>>
>> Table qa2.ADGROUPS is ENABLED
>>
>>
>>
>>
>> qa2.ADGROUPS, {TABLE_ATTRIBUTES => {coprocessor$1 =>
>> '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|',
>> coprocessor$2 =>
>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
>> coprocessor$
>>
>> 3 =>
>> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
>> coprocessor$4 =>
>> '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
>> coprocessor$5 => '|org.apache.phoenix.hbase.index.Index
>>
>> er|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'}
>>
>>
>> COLUMN FAMILIES DESCRIPTION
>>
>>
>>
>>
>> {NAME => 'AG', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY =>
>> 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIFF',
>> TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE =>
>> 'true'
>>
>> , BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
>>
>>
>>
>>
>> 1 row(s) in 0.4210 seconds
>>
>> #3
>>
>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select KEY_SEQ,
>> ORDINAL_POSITION, COLUMN_QUALIFIER, DATA_TYPE from SYSTEM.CATALOG where
>> TABLE_SCHEM = 'qa2' and TABLE_NAME = 'ADGROUPS' order by ORDINAL_POSITION ;
>>
>> +----------+-------------------+-------------------+------------+
>>
>> | KEY_SEQ | ORDINAL_POSITION | COLUMN_QUALIFIER | DATA_TYPE |
>>
>> +----------+-------------------+-------------------+------------+
>>
>> | null | null | | null |
>>
>> | null | null | | null |
>>
>> | null | null | | null |
>>
>> | null | null | | null |
>>
>> | 1 | 1 | | -5 |
>>
>> | 2 | 2 | | -5 |
>>
>> | 3 | 3 | | 13 |
>>
>> | 4 | 4 | | -5 |
>>
>> | 5 | 5 | | -5 |
>>
>> | 6 | 6 | | -5 |
>>
>> | null | 7 | 800b | 12 |
>>
>> | null | 8 | 800c | -5 |
>>
>> | null | 9 | 800d | 2995 |
>>
>> | null | 10 | 800e | -5 |
>>
>> | null | 11 | 800f | 12 |
>>
>> | null | 12 | 8010 | 1 |
>>
>> | null | 13 | 8011 | 12 |
>>
>> | null | 14 | 8012 | 12 |
>>
>> | null | 15 | 8013 | 12 |
>>
>> | null | 16 | 8014 | 12 |
>>
>> | null | 17 | 8015 | 93 |
>>
>> | null | 18 | 8016 | 93 |
>>
>> | null | 19 | 8017 | 12 |
>>
>> | null | 20 | 8018 | -5 |
>>
>> | null | 21 | 8019 | 12 |
>>
>> | null | 22 | 801a | 12 |
>>
>> | null | 23 | 801b | -3 |
>>
>> | null | 24 | 801c | 12 |
>>
>> | null | 25 | 801d | 12 |
>>
>> | null | 26 | 801e | 12 |
>>
>> | null | 27 | 801f | 12 |
>>
>> | null | 28 | 8020 | -5 |
>>
>> | null | 29 | 8021 | 12 |
>>
>> | null | 30 | 8022 | 12 |
>>
>> | null | 31 | 8023 | -5 |
>>
>> | null | 32 | 8024 | 16 |
>>
>> | null | 33 | 8025 | 12 |
>>
>> | null | 34 | 8026 | -3 |
>>
>> | null | 35 | 8027 | 93 |
>>
>> | null | 36 | 8028 | -5 |
>>
>> | null | 37 | 8029 | 93 |
>>
>> | null | 38 | 802a | -5 |
>>
>> | null | 39 | 802b | 16 |
>>
>> | null | 40 | 802c | 12 |
>>
>> | null | 41 | 802d | 16 |
>>
>> | null | 42 | 802e | 93 |
>>
>> | null | 43 | 802f | 12 |
>>
>> | null | 44 | 8030 | -5 |
>>
>> | null | 45 | 8031 | 12 |
>>
>> | null | 46 | 8032 | 4 |
>>
>> | null | 47 | 8033 | 6 |
>>
>> | null | 48 | 8034 | 12 |
>>
>> | null | 49 | 8035 | 16 |
>>
>> | null | 50 | 8036 | 3012 |
>>
>> | null | 51 | 8037 | 93 |
>>
>> | null | 52 | 8038 | 16 |
>>
>> | null | 53 | 8039 | 4 |
>>
>> | null | 54 | 803a | 12 |
>>
>> | null | 55 | 803b | 12 |
>>
>> | null | 56 | 803c | 12 |
>>
>> | null | 57 | 803d | 3012 |
>>
>> | null | 58 | 803e | -3 |
>>
>> | null | 59 | 803f | 12 |
>>
>> | null | 60 | 8040 | -5 |
>>
>> | null | 61 | 8041 | 8 |
>>
>> | null | 62 | 8042 | 8 |
>>
>> | null | 63 | 8043 | 12 |
>>
>> | null | 64 | 8044 | -5 |
>>
>> | null | 65 | 8045 | 12 |
>>
>> | null | 66 | 8046 | 4 |
>>
>> | null | 67 | 8047 | 4 |
>>
>> | null | 68 | 8048 | -5 |
>>
>> | null | 69 | 8049 | -6 |
>>
>> | null | 70 | 804a | 12 |
>>
>> | null | 71 | 804b | 12 |
>>
>> | null | 72 | 804c | 4 |
>>
>> | null | 73 | 804d | -5 |
>>
>> | null | 74 | 804e | -5 |
>>
>> | null | 75 | 804f | 12 |
>>
>> | null | 76 | 8050 | 12 |
>>
>> | null | 77 | 8051 | 4 |
>>
>> | null | 78 | 8052 | 16 |
>>
>> | null | 79 | 8053 | -5 |
>>
>> | null | 80 | 8054 | 12 |
>>
>> | null | 81 | 8055 | 12 |
>>
>> | null | 82 | 8056 | 12 |
>>
>> | null | 83 | 8057 | 12 |
>>
>> | null | 84 | 8058 | 12 |
>>
>> +----------+-------------------+-------------------+------------+
>>
>
Re: HBase Compaction Fails for Phoenix Table
Posted by William Shen <wi...@marinsoftware.com>.
Further narrowed down the HBase row, and was able to reproduce the
Exception in HBase Shell. The error also shows up in the region server log.
Would appreciate any pointers on next step to investigate the issue on the
HBase level. Thank you!
hbase(main):007:0> get 'qa2.ADGROUPS',
"\x05\x80\x00\x00\x00\x00\x1F\x54\x9C\x80\x00\x00\x00\x00\x1C\x7D\x45\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00\x4A\x64\x6F\x80\x00\x00\x00\x01\xD9\xDB\xCE"
COLUMN
CELL
ERROR: java.io.IOException
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
Caused by: java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkBounds(Buffer.java:567)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5706)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5865)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5643)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5620)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5606)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6801)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6779)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2029)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
... 3 more
On Wed, Nov 21, 2018 at 11:19 AM William Shen <wi...@marinsoftware.com>
wrote:
> Hi Jaanai,
>
> We are running Phoenix 4.13, and actually upon investigating the issue
> more, we are not so sure if it is that row that's having issue. It seems
> like the data issue might be on a different row than the one HBase
> compaction is complaining about because when we do a point look up, the
> data comes back, but it does not return if it is a range scan (#1). Not
> totally sure why HBase complained about that row when the failure is
> somewhere else...
>
> I don't have the specific Phoenix DDL handy, but here is describe in HBase
> (#2), and also selective info from SYSTEMS.CATALOGS (#3). Hopefully that's
> helpful to make sense of the data types / keys, etc. Is there anything
> specific that you want to see?
>
>
>
> *#1: Range scan vs point look up*
>
> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
> "cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
> "cstId" = 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4
> and "accId" = 1904409 and "cpgnId" = 4811421;
>
>
> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>
> | PLAN
> | EST_BYTES_READ | EST_ROWS_READ |
> EST_INFO_TS |
>
>
> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>
> | CLIENT 1-CHUNK 1 ROWS 8052 BYTES PARALLEL 1-WAY ROUND ROBIN POINT
> LOOKUP ON 1 KEY OVER qa2.ADGROUPS | 8052 | 1 |
> 1542827075237 |
>
>
> +------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>
> 1 row selected (0.048 seconds)
>
> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
> "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "cstId"
> = 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4 and
> "accId" = 1904409 and "cpgnId" = 4811421;
>
>
> +---------+----------+----------+--------+----------+----------+-----------+
>
> | stts | cstId | cltId | pubId | accId | cpgnId | id
> |
>
>
> +---------+----------+----------+--------+----------+----------+-----------+
>
> | ACTIVE | 2053276 | 1867077 | 4 | 1904409 | 4811421 | 30496531
> |
>
>
> +---------+----------+----------+--------+----------+----------+-----------+
>
> 1 row selected (0.072 seconds)
>
> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
> "cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
> "id" = 30496531;
>
>
> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>
> | PLAN
> | EST_BYTES_READ |
> EST_ROWS_READ | EST_INFO_TS |
>
>
> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>
> | CLIENT 364-CHUNK 36227938 ROWS 34037631025 BYTES PARALLEL 10-WAY ROUND
> ROBIN RANGE SCAN OVER qa2.ADGROUPS [0] - [9] | 34037631025 | 36227938
> | 1542823263107 |
>
> | SERVER FILTER BY "id" = 30496531
> | 34037631025 | 36227938
> | 1542823263107 |
>
>
> +----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
>
> 2 rows selected (0.063 seconds)
>
> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
> "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "id" =
> 30496531;
>
> *[Hangs]*
>
> #2
>
> hbase(main):001:0> describe "qa2.ADGROUPS"
>
> Table qa2.ADGROUPS is ENABLED
>
>
>
>
> qa2.ADGROUPS, {TABLE_ATTRIBUTES => {coprocessor$1 =>
> '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|',
> coprocessor$2 =>
> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
> coprocessor$
>
> 3 =>
> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
> coprocessor$4 =>
> '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
> coprocessor$5 => '|org.apache.phoenix.hbase.index.Index
>
> er|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'}
>
>
> COLUMN FAMILIES DESCRIPTION
>
>
>
>
> {NAME => 'AG', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY =>
> 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIFF',
> TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE =>
> 'true'
>
> , BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
>
>
>
>
> 1 row(s) in 0.4210 seconds
>
> #3
>
> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select KEY_SEQ,
> ORDINAL_POSITION, COLUMN_QUALIFIER, DATA_TYPE from SYSTEM.CATALOG where
> TABLE_SCHEM = 'qa2' and TABLE_NAME = 'ADGROUPS' order by ORDINAL_POSITION ;
>
> +----------+-------------------+-------------------+------------+
>
> | KEY_SEQ | ORDINAL_POSITION | COLUMN_QUALIFIER | DATA_TYPE |
>
> +----------+-------------------+-------------------+------------+
>
> | null | null | | null |
>
> | null | null | | null |
>
> | null | null | | null |
>
> | null | null | | null |
>
> | 1 | 1 | | -5 |
>
> | 2 | 2 | | -5 |
>
> | 3 | 3 | | 13 |
>
> | 4 | 4 | | -5 |
>
> | 5 | 5 | | -5 |
>
> | 6 | 6 | | -5 |
>
> | null | 7 | 800b | 12 |
>
> | null | 8 | 800c | -5 |
>
> | null | 9 | 800d | 2995 |
>
> | null | 10 | 800e | -5 |
>
> | null | 11 | 800f | 12 |
>
> | null | 12 | 8010 | 1 |
>
> | null | 13 | 8011 | 12 |
>
> | null | 14 | 8012 | 12 |
>
> | null | 15 | 8013 | 12 |
>
> | null | 16 | 8014 | 12 |
>
> | null | 17 | 8015 | 93 |
>
> | null | 18 | 8016 | 93 |
>
> | null | 19 | 8017 | 12 |
>
> | null | 20 | 8018 | -5 |
>
> | null | 21 | 8019 | 12 |
>
> | null | 22 | 801a | 12 |
>
> | null | 23 | 801b | -3 |
>
> | null | 24 | 801c | 12 |
>
> | null | 25 | 801d | 12 |
>
> | null | 26 | 801e | 12 |
>
> | null | 27 | 801f | 12 |
>
> | null | 28 | 8020 | -5 |
>
> | null | 29 | 8021 | 12 |
>
> | null | 30 | 8022 | 12 |
>
> | null | 31 | 8023 | -5 |
>
> | null | 32 | 8024 | 16 |
>
> | null | 33 | 8025 | 12 |
>
> | null | 34 | 8026 | -3 |
>
> | null | 35 | 8027 | 93 |
>
> | null | 36 | 8028 | -5 |
>
> | null | 37 | 8029 | 93 |
>
> | null | 38 | 802a | -5 |
>
> | null | 39 | 802b | 16 |
>
> | null | 40 | 802c | 12 |
>
> | null | 41 | 802d | 16 |
>
> | null | 42 | 802e | 93 |
>
> | null | 43 | 802f | 12 |
>
> | null | 44 | 8030 | -5 |
>
> | null | 45 | 8031 | 12 |
>
> | null | 46 | 8032 | 4 |
>
> | null | 47 | 8033 | 6 |
>
> | null | 48 | 8034 | 12 |
>
> | null | 49 | 8035 | 16 |
>
> | null | 50 | 8036 | 3012 |
>
> | null | 51 | 8037 | 93 |
>
> | null | 52 | 8038 | 16 |
>
> | null | 53 | 8039 | 4 |
>
> | null | 54 | 803a | 12 |
>
> | null | 55 | 803b | 12 |
>
> | null | 56 | 803c | 12 |
>
> | null | 57 | 803d | 3012 |
>
> | null | 58 | 803e | -3 |
>
> | null | 59 | 803f | 12 |
>
> | null | 60 | 8040 | -5 |
>
> | null | 61 | 8041 | 8 |
>
> | null | 62 | 8042 | 8 |
>
> | null | 63 | 8043 | 12 |
>
> | null | 64 | 8044 | -5 |
>
> | null | 65 | 8045 | 12 |
>
> | null | 66 | 8046 | 4 |
>
> | null | 67 | 8047 | 4 |
>
> | null | 68 | 8048 | -5 |
>
> | null | 69 | 8049 | -6 |
>
> | null | 70 | 804a | 12 |
>
> | null | 71 | 804b | 12 |
>
> | null | 72 | 804c | 4 |
>
> | null | 73 | 804d | -5 |
>
> | null | 74 | 804e | -5 |
>
> | null | 75 | 804f | 12 |
>
> | null | 76 | 8050 | 12 |
>
> | null | 77 | 8051 | 4 |
>
> | null | 78 | 8052 | 16 |
>
> | null | 79 | 8053 | -5 |
>
> | null | 80 | 8054 | 12 |
>
> | null | 81 | 8055 | 12 |
>
> | null | 82 | 8056 | 12 |
>
> | null | 83 | 8057 | 12 |
>
> | null | 84 | 8058 | 12 |
>
> +----------+-------------------+-------------------+------------+
>
Re: HBase Compaction Fails for Phoenix Table
Posted by William Shen <wi...@marinsoftware.com>.
Hi Jaanai,
We are running Phoenix 4.13, and actually upon investigating the issue
more, we are not so sure if it is that row that's having issue. It seems
like the data issue might be on a different row than the one HBase
compaction is complaining about because when we do a point look up, the
data comes back, but it does not return if it is a range scan (#1). Not
totally sure why HBase complained about that row when the failure is
somewhere else...
I don't have the specific Phoenix DDL handy, but here is describe in HBase
(#2), and also selective info from SYSTEMS.CATALOGS (#3). Hopefully that's
helpful to make sense of the data types / keys, etc. Is there anything
specific that you want to see?
*#1: Range scan vs point look up*
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
"cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
"cstId" = 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4
and "accId" = 1904409 and "cpgnId" = 4811421;
+------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
| PLAN
| EST_BYTES_READ | EST_ROWS_READ |
EST_INFO_TS |
+------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
| CLIENT 1-CHUNK 1 ROWS 8052 BYTES PARALLEL 1-WAY ROUND ROBIN POINT LOOKUP
ON 1 KEY OVER qa2.ADGROUPS | 8052 | 1 |
1542827075237 |
+------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
1 row selected (0.048 seconds)
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
"cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "cstId"
= 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4 and
"accId" = 1904409 and "cpgnId" = 4811421;
+---------+----------+----------+--------+----------+----------+-----------+
| stts | cstId | cltId | pubId | accId | cpgnId | id |
+---------+----------+----------+--------+----------+----------+-----------+
| ACTIVE | 2053276 | 1867077 | 4 | 1904409 | 4811421 | 30496531 |
+---------+----------+----------+--------+----------+----------+-----------+
1 row selected (0.072 seconds)
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
"cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
"id" = 30496531;
+----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
| PLAN
| EST_BYTES_READ |
EST_ROWS_READ | EST_INFO_TS |
+----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
| CLIENT 364-CHUNK 36227938 ROWS 34037631025 BYTES PARALLEL 10-WAY ROUND
ROBIN RANGE SCAN OVER qa2.ADGROUPS [0] - [9] | 34037631025 | 36227938
| 1542823263107 |
| SERVER FILTER BY "id" = 30496531
| 34037631025 | 36227938
| 1542823263107 |
+----------------------------------------------------------------------------------------------------------------------+-----------------+----------------+----------------+
2 rows selected (0.063 seconds)
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
"cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "id" =
30496531;
*[Hangs]*
#2
hbase(main):001:0> describe "qa2.ADGROUPS"
Table qa2.ADGROUPS is ENABLED
qa2.ADGROUPS, {TABLE_ATTRIBUTES => {coprocessor$1 =>
'|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|',
coprocessor$2 =>
'|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
coprocessor$
3 =>
'|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
coprocessor$4 =>
'|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
coprocessor$5 => '|org.apache.phoenix.hbase.index.Index
er|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'}
COLUMN FAMILIES DESCRIPTION
{NAME => 'AG', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY =>
'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIFF',
TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE =>
'true'
, BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
1 row(s) in 0.4210 seconds
#3
0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select KEY_SEQ,
ORDINAL_POSITION, COLUMN_QUALIFIER, DATA_TYPE from SYSTEM.CATALOG where
TABLE_SCHEM = 'qa2' and TABLE_NAME = 'ADGROUPS' order by ORDINAL_POSITION ;
+----------+-------------------+-------------------+------------+
| KEY_SEQ | ORDINAL_POSITION | COLUMN_QUALIFIER | DATA_TYPE |
+----------+-------------------+-------------------+------------+
| null | null | | null |
| null | null | | null |
| null | null | | null |
| null | null | | null |
| 1 | 1 | | -5 |
| 2 | 2 | | -5 |
| 3 | 3 | | 13 |
| 4 | 4 | | -5 |
| 5 | 5 | | -5 |
| 6 | 6 | | -5 |
| null | 7 | 800b | 12 |
| null | 8 | 800c | -5 |
| null | 9 | 800d | 2995 |
| null | 10 | 800e | -5 |
| null | 11 | 800f | 12 |
| null | 12 | 8010 | 1 |
| null | 13 | 8011 | 12 |
| null | 14 | 8012 | 12 |
| null | 15 | 8013 | 12 |
| null | 16 | 8014 | 12 |
| null | 17 | 8015 | 93 |
| null | 18 | 8016 | 93 |
| null | 19 | 8017 | 12 |
| null | 20 | 8018 | -5 |
| null | 21 | 8019 | 12 |
| null | 22 | 801a | 12 |
| null | 23 | 801b | -3 |
| null | 24 | 801c | 12 |
| null | 25 | 801d | 12 |
| null | 26 | 801e | 12 |
| null | 27 | 801f | 12 |
| null | 28 | 8020 | -5 |
| null | 29 | 8021 | 12 |
| null | 30 | 8022 | 12 |
| null | 31 | 8023 | -5 |
| null | 32 | 8024 | 16 |
| null | 33 | 8025 | 12 |
| null | 34 | 8026 | -3 |
| null | 35 | 8027 | 93 |
| null | 36 | 8028 | -5 |
| null | 37 | 8029 | 93 |
| null | 38 | 802a | -5 |
| null | 39 | 802b | 16 |
| null | 40 | 802c | 12 |
| null | 41 | 802d | 16 |
| null | 42 | 802e | 93 |
| null | 43 | 802f | 12 |
| null | 44 | 8030 | -5 |
| null | 45 | 8031 | 12 |
| null | 46 | 8032 | 4 |
| null | 47 | 8033 | 6 |
| null | 48 | 8034 | 12 |
| null | 49 | 8035 | 16 |
| null | 50 | 8036 | 3012 |
| null | 51 | 8037 | 93 |
| null | 52 | 8038 | 16 |
| null | 53 | 8039 | 4 |
| null | 54 | 803a | 12 |
| null | 55 | 803b | 12 |
| null | 56 | 803c | 12 |
| null | 57 | 803d | 3012 |
| null | 58 | 803e | -3 |
| null | 59 | 803f | 12 |
| null | 60 | 8040 | -5 |
| null | 61 | 8041 | 8 |
| null | 62 | 8042 | 8 |
| null | 63 | 8043 | 12 |
| null | 64 | 8044 | -5 |
| null | 65 | 8045 | 12 |
| null | 66 | 8046 | 4 |
| null | 67 | 8047 | 4 |
| null | 68 | 8048 | -5 |
| null | 69 | 8049 | -6 |
| null | 70 | 804a | 12 |
| null | 71 | 804b | 12 |
| null | 72 | 804c | 4 |
| null | 73 | 804d | -5 |
| null | 74 | 804e | -5 |
| null | 75 | 804f | 12 |
| null | 76 | 8050 | 12 |
| null | 77 | 8051 | 4 |
| null | 78 | 8052 | 16 |
| null | 79 | 8053 | -5 |
| null | 80 | 8054 | 12 |
| null | 81 | 8055 | 12 |
| null | 82 | 8056 | 12 |
| null | 83 | 8057 | 12 |
| null | 84 | 8058 | 12 |
+----------+-------------------+-------------------+------------+
Thanks
On Wed, Nov 21, 2018 at 1:23 AM Jaanai Zhang <cl...@gmail.com> wrote:
> Can you please attach Create the table/indexes SQL？ is this Phoneix's
> version? Are you sure many rows data had been corrupted or only this one
> row?
>
> ----------------------------------------
> Jaanai Zhang
> Best regards!
>
>
>
> William Shen <wi...@marinsoftware.com> 于2018年11月21日周三 上午10:20写道：
>
>> Hi there,
>>
>> We've encountered the following compaction failure(#1) for a Phoenix
>> table, and are not sure how to make sense of it. Using the HBase row key
>> from the error, we are able to query data directly from hbase shell, and by
>> examining the data, there arent anything immediately obvious about the data
>> as they seem to be stored consistent with the Phoenix data type (#2). When
>> querying Phoenix for the given row through sqline, the row would return if
>> only primary key columns are selected, and the query would not return if
>> non primary key columns are selected (#3).
>>
>> Few questions hoping to find some help on:
>>
>> a. Are we correct in understanding the error message to indicate an issue
>> with data for the row key (
>> \x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13)?
>> We are not sure what to make sense of the string "
>> 1539019716378.3dcf2b1e057915feb74395d9711ba4ad." that is included with
>> the row key...
>>
>> b. What is out of bound here? It's not apparently clear here what
>> StatisticsScanner and FastDiffDeltaEncoder are tripping over...
>>
>> c. Is it normal for hbase shell to return some parts of the hex string as
>> ascii characters? We are seeing that in the row key as well as column name
>> encoding and value. We are not sure if that is causing any issues, or if
>> that was just a display issue that we can safely ignore.
>>
>> *#1 Compaction Failure*
>>
>> Compaction failed Request = regionName=qa2.ADGROUPS,\x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13,1539019716378.3dcf2b1e057915feb74395d9711ba4ad., storeName=AG, fileCount=4, fileSize=316.0 M (315.8 M, 188.7 K, 6.8 K, 14.2 K), priority=1, time=40613533856170784
>> java.lang.IndexOutOfBoundsException
>> at java.nio.Buffer.checkBounds(Buffer.java:567)
>> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
>> at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
>> at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
>> at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
>> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
>> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
>> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
>> at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
>> at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:458)
>> at org.apache.phoenix.schema.stats.StatisticsScanner.next(StatisticsScanner.java:69)
>> at org.apache.phoenix.schema.stats.StatisticsScanner.next(StatisticsScanner.java:76)
>> at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:334)
>> at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:106)
>> at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:131)
>> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1245)
>> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1852)
>> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:529)
>> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:566)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> *#2 Data on the HBase level*
>>
>> hbase(main):002:0> get 'qa2.ADGROUPS', "\x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13"
>> COLUMN CELL
>> AG:\x00\x00\x00\x00 timestamp=1539019457506, value=x
>> AG:\x80\x0F timestamp=1539019457506, value=D:USA_AK:Nightmute:1456903:hotel
>> AG:\x80\x12 timestamp=1539019457506, value=ACTIVE
>> AG:\x80\x13 timestamp=1539019457506, value=ADD
>> AG:\x80# timestamp=1539019457506, value=\x80\x00\x00\x00\x00\x00'\x10
>> AG:\x80' timestamp=1539019457506, value=\x80\x00\x01[\x97\x02\x02X\x00\x00\x00\x00
>> AG:\x80( timestamp=1539019457506, value=\x80\x00\x00\x00\x00\x1D\xEC\xA4
>> AG:\x808 timestamp=1539019457506, value=\x00
>> 8 row(s) in 0.0510 seconds
>>
>>
>> *#3 Data in Phoenix via sqline*
>>
>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "id" = 30496531;
>> +----------+----------+--------+----------+----------+-----------+
>> | cstId | cltId | pubId | accId | cpgnId | id |
>> +----------+----------+--------+----------+----------+-----------+
>> | 2053276 | 1867077 | 4 | 1904409 | 4811421 | 30496531 |
>> +----------+----------+--------+----------+----------+-----------+
>> 1 row selected (0.095 seconds)
>>
>> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "cstId", "cltId", "pubId", "accId","cpgnId", "id", "stts" from "qa2".adgroups where "id" = 30496531;
>>
>> *[hangs]*
>>
>>
>> Thanks in advance for your help!
>>
>
Re: HBase Compaction Fails for Phoenix Table
Posted by Jaanai Zhang <cl...@gmail.com>.
Can you please attach Create the table/indexes SQL？ is this Phoneix's
version? Are you sure many rows data had been corrupted or only this one
row?
----------------------------------------
Jaanai Zhang
Best regards!
William Shen <wi...@marinsoftware.com> 于2018年11月21日周三 上午10:20写道：
> Hi there,
>
> We've encountered the following compaction failure(#1) for a Phoenix
> table, and are not sure how to make sense of it. Using the HBase row key
> from the error, we are able to query data directly from hbase shell, and by
> examining the data, there arent anything immediately obvious about the data
> as they seem to be stored consistent with the Phoenix data type (#2). When
> querying Phoenix for the given row through sqline, the row would return if
> only primary key columns are selected, and the query would not return if
> non primary key columns are selected (#3).
>
> Few questions hoping to find some help on:
>
> a. Are we correct in understanding the error message to indicate an issue
> with data for the row key (
> \x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13)?
> We are not sure what to make sense of the string "
> 1539019716378.3dcf2b1e057915feb74395d9711ba4ad." that is included with
> the row key...
>
> b. What is out of bound here? It's not apparently clear here what
> StatisticsScanner and FastDiffDeltaEncoder are tripping over...
>
> c. Is it normal for hbase shell to return some parts of the hex string as
> ascii characters? We are seeing that in the row key as well as column name
> encoding and value. We are not sure if that is causing any issues, or if
> that was just a display issue that we can safely ignore.
>
> *#1 Compaction Failure*
>
> Compaction failed Request = regionName=qa2.ADGROUPS,\x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13,1539019716378.3dcf2b1e057915feb74395d9711ba4ad., storeName=AG, fileCount=4, fileSize=316.0 M (315.8 M, 188.7 K, 6.8 K, 14.2 K), priority=1, time=40613533856170784
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkBounds(Buffer.java:567)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
> at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
> at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
> at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:458)
> at org.apache.phoenix.schema.stats.StatisticsScanner.next(StatisticsScanner.java:69)
> at org.apache.phoenix.schema.stats.StatisticsScanner.next(StatisticsScanner.java:76)
> at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:334)
> at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:106)
> at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:131)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1245)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1852)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:529)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:566)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
>
>
> *#2 Data on the HBase level*
>
> hbase(main):002:0> get 'qa2.ADGROUPS', "\x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13"
> COLUMN CELL
> AG:\x00\x00\x00\x00 timestamp=1539019457506, value=x
> AG:\x80\x0F timestamp=1539019457506, value=D:USA_AK:Nightmute:1456903:hotel
> AG:\x80\x12 timestamp=1539019457506, value=ACTIVE
> AG:\x80\x13 timestamp=1539019457506, value=ADD
> AG:\x80# timestamp=1539019457506, value=\x80\x00\x00\x00\x00\x00'\x10
> AG:\x80' timestamp=1539019457506, value=\x80\x00\x01[\x97\x02\x02X\x00\x00\x00\x00
> AG:\x80( timestamp=1539019457506, value=\x80\x00\x00\x00\x00\x1D\xEC\xA4
> AG:\x808 timestamp=1539019457506, value=\x00
> 8 row(s) in 0.0510 seconds
>
>
> *#3 Data in Phoenix via sqline*
>
> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "id" = 30496531;
> +----------+----------+--------+----------+----------+-----------+
> | cstId | cltId | pubId | accId | cpgnId | id |
> +----------+----------+--------+----------+----------+-----------+
> | 2053276 | 1867077 | 4 | 1904409 | 4811421 | 30496531 |
> +----------+----------+--------+----------+----------+-----------+
> 1 row selected (0.095 seconds)
>
> 0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "cstId", "cltId", "pubId", "accId","cpgnId", "id", "stts" from "qa2".adgroups where "id" = 30496531;
>
> *[hangs]*
>
>
> Thanks in advance for your help!
>