You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Duo Zhang (Jira)" <ji...@apache.org> on 2021/12/29 02:01:00 UTC
[jira] [Commented] (HBASE-26633) When getting rowkey, the program enters the org.apache.hadoop.hbase.regionserver.KeyValueHeap#generalizedSeek and an endless loop occurs
[ https://issues.apache.org/jira/browse/HBASE-26633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17466283#comment-17466283 ]
Duo Zhang commented on HBASE-26633:
-----------------------------------
Talking with [~liuzhangqing1] offline through wechat, the strange thing is that, in KeyValueHeap.generalizedSeek, the topKey is always the same while we keep seeking the top scanner. This should not happen.
She can reproduce this locally, but since the data are a bit sensitive, she can not attach them here. Will help her debugging more and see if we could provide a simple UT to reproduce the problem.
Thanks.
> When getting rowkey, the program enters the org.apache.hadoop.hbase.regionserver.KeyValueHeap#generalizedSeek and an endless loop occurs
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-26633
> URL: https://issues.apache.org/jira/browse/HBASE-26633
> Project: HBase
> Issue Type: Bug
> Reporter: liuzhangqing
> Priority: Major
> Attachments: image-2021-12-28-22-17-36-088.png
>
>
> 1. Phenomenon description:
> Get rowkey: 350ef10764bc4af960b6b468479aff1e_ The data of 10032279726411 is the master of the program card and enters the dead cycle
> 2. Data information:
> hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/250853d176294585ae0339d11f9714c1
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:directory/1637459509892/Put/vlen=0/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637459509892/Put/vlen=0/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wareid/1637459509892/Put/vlen=14/seqid=0
> hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/85ab3a19f409476aada64ab5bf906653
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:directory/1637373810660/Put/vlen=0/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637373810660/Put/vlen=0/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wareid/1637373810660/Put/vlen=14/seqid=0
> hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/93cc194d62fc419792991613332b990d
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:directory/1637112759915/Put/vlen=0/seqid=1555467268
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:product_ext_nick/1637057318457/Put/vlen=27/seqid=1554882684
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:store_info_ex_pc/1637125253154/Put/vlen=10329/seqid=1555632639
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637112759915/Put/vlen=0/seqid=1555467268
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:warehouse/1637125253154/Put/vlen=17843/seqid=1555632639
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wareid/1637112759915/Put/vlen=14/seqid=1555467268
> hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/a001e8a2dd454029ae38554b88018109
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:directory/1637027051359/Put/vlen=0/seqid=1554430204
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:dredisprice/1636877442107/Put/vlen=5/seqid=1552727948
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:lastupdatetime/1636829185081/Put/vlen=10/seqid=1552173184
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:store_info_ex_pc/1636979778239/Put/vlen=10297/seqid=1553901027
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637027051359/Put/vlen=0/seqid=1554430204
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:warehouse/1636979778239/Put/vlen=17299/seqid=1553901027
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wareid/1637027051359/Put/vlen=14/seqid=1554430204
> hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/a62976c426574d43af4b5d3fef8af921
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:directory/1637545456304/Put/vlen=0/seqid=1556484345
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:product_ext_nick/1637629087047/Put/vlen=18/seqid=1556602639
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:union_spec_ids/1637582728941/Put/vlen=213/seqid=1556506804
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637545456304/Put/vlen=0/seqid=1556484345
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wareid/1637545456304/Put/vlen=14/seqid=1556484345
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wstate/1637549762970/Put/vlen=2/seqid=1556484528
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wyn/1637549762970/Put/vlen=1/seqid=1556484528
> hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/ff8f38c495ba46398ae9c02207950f7d
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:directory/1636594256197/Put/vlen=0/seqid=1549353741
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:dredisprice/1636646590929/Put/vlen=5/seqid=1550021079
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:lastupdatetime/1635940338088/Put/vlen=10/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:product_ext_nick/1634995561817/Put/vlen=41/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:product_id/1623293068319/Put/vlen=14/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:sku_union_spec_ids/1623293068319/Put/vlen=213/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:store_info_ex_pc/1636191109189/Put/vlen=10297/seqid=1544129517
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:union_expand_ids/1623293068319/Put/vlen=0/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:union_spec_ids/1635935424287/Put/vlen=213/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:unlimit_catid/1623293068319/Put/vlen=4/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_id/1623293068319/Put/vlen=6/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1636594256197/Put/vlen=0/seqid=1549353741
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:ware_type/1623293068319/Put/vlen=1/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:ware_weight/1623293068319/Put/vlen=3/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:warehouse/1636405561106/Put/vlen=17321/seqid=1546667844
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wareid/1636594256197/Put/vlen=14/seqid=1549353741
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:warename/1623293068319/Put/vlen=99/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:warepid/1623293068319/Put/vlen=14/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:waretitle/1627360205219/Put/vlen=54/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:winsdate/1623293068319/Put/vlen=10/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wpid2/1623293068319/Put/vlen=1/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wredisprice/1623293074488/Put/vlen=5/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wstate/1635418719233/Put/vlen=1/seqid=0
> K: 350ef10764bc4af960b6b468479aff1e_10032279726411/d:wyn/1623293070700/Put/vlen=1/seqid=0
> 3. Abnormal process:
> When executed to: d:vender_ name column ScanQueryMatcher.MatchCode qcode = matcher.match(cell); The return value is: seek_ NEXT_ COL，The method calling process is：
> seekOrSkipToNextColumn-》seekAsDirection-》reseek-》heap.resee-》generalizedSeek
> However, when the program executes the generalizedseek method, it cannot exit，At this time, the data information of heap is：
> 2021-12-28 19:02:26,801 INFO [main] regionserver.StoreScanner(844): current:StoreFileScanner[HFileScanner for reader reader=[hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/93cc194d62fc419792991613332b990d], compression=lz4, cacheConf=cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, firstKey=Optional[34f5c0cb717779d9ede9a8b80e46d4d4_10039682866610/d:product_ext_nick/1637109574351/Put/seqid=0], lastKey=Optional[351eb49327ecd6099a11efe0d1347724_10020558744609/d:commentcount/1637134716442/Put/seqid=0], avgKeyLen=70, avgValueLen=76, entries=3831303, length=251907468, cur=350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637112759915/Put/vlen=0/seqid=1555467268]
> 2021-12-28 19:02:26,801 INFO [main] regionserver.StoreScanner(847): keyValueScanner:StoreFileScanner[HFileScanner for reader reader=[hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/a001e8a2dd454029ae38554b88018109], compression=lz4, cacheConf=cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, firstKey=Optional[34f5c0c9f049e2fa0b0b18b50a4f8b82_10039161263881/d:product_ext_nick/1637024214199/Put/seqid=0], lastKey=Optional[351eb49327ecd6099a11efe0d1347724_10020558744609/d:commentcount/1637044327190/Put/seqid=0], avgKeyLen=71, avgValueLen=48, entries=9028492, length=456744125, cur=350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1637027051359/Put/vlen=0/seqid=1554430204]
> 2021-12-28 19:02:26,801 INFO [main] regionserver.StoreScanner(847): keyValueScanner:StoreFileScanner[HFileScanner for reader reader=[hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/ff8f38c495ba46398ae9c02207950f7d], compression=lz4, cacheConf=cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, firstKey=Optional[34f5c0c91665cb0a1a3360076a6294f2_10037000082814/d:averagescore/1631432545837/Put/seqid=0], lastKey=Optional[351eb49327ecd6099a11efe0d1347724_10020558744609/d:wyn/1626850561189/Put/seqid=0], avgKeyLen=69, avgValueLen=26, entries=539934002, length=17765279323, cur=350ef10764bc4af960b6b468479aff1e_10032279726411/d:vender_name/1636594256197/Put/vlen=0/seqid=1549353741]
> 2021-12-28 19:02:26,801 INFO [main] regionserver.StoreScanner(847): keyValueScanner:StoreFileScanner[HFileScanner for reader reader=[hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/250853d176294585ae0339d11f9714c1], compression=lz4, cacheConf=cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, firstKey=Optional[34f5c0dcb467c26bd5355982cc01a095_43099352773/d:directory/1637459509892/Put/seqid=0], lastKey=Optional[351eb48a36179802a0f8b38452e26b7a_10039387550202/d:wareid/1637459509892/Put/seqid=0], avgKeyLen=68, avgValueLen=12, entries=336963, length=9094030, cur=350ef10764bc4af960b6b468479aff1e_10032279726411/d:wareid/1637459509892/Put/vlen=14/seqid=-1]
> 2021-12-28 19:02:26,801 INFO [main] regionserver.StoreScanner(847): keyValueScanner:StoreFileScanner[HFileScanner for reader reader=[hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/a62976c426574d43af4b5d3fef8af921], compression=lz4, cacheConf=cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, firstKey=Optional[34f5c0cca42ad68aa73b72953336d214_10038013927618/d:catname/1637623282535/Put/seqid=0], lastKey=Optional[351eb49327ecd6099a11efe0d1347724_10020558744609/d:union_spec_ids/1637582728941/Put/seqid=0], avgKeyLen=71, avgValueLen=42, entries=18764748, length=899013217, cur=350ef10764bc4af960b6b468479aff1e_10032279726411/d:wareid/1637545456304/Put/vlen=14/seqid=1556484345]
> 2021-12-28 19:02:26,801 INFO [main] regionserver.StoreScanner(847): keyValueScanner:StoreFileScanner[HFileScanner for reader reader=[hdfs://snake/hbase/data/realtime_search/WD_MERGED_DATA/2fc7bd24c7d6d8d500f04b2480660de9/d/85ab3a19f409476aada64ab5bf906653], compression=lz4, cacheConf=cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, firstKey=Optional[34f5c0dcb467c26bd5355982cc01a095_43099352773/d:directory/1637373810660/Put/seqid=0], lastKey=Optional[351eb48a36179802a0f8b38452e26b7a_10039387550202/d:wareid/1637373810660/Put/seqid=0], avgKeyLen=68, avgValueLen=12, entries=334692, length=9047191, cur=350ef10764bc4af960b6b468479aff1e_10032279726411/d:wareid/1637373810660/Put/vlen=14/seqid=-1]
> Because cur of multiple storefilescanners in heap are the d:vender_name columns,So in the while loop of the generalizedSeek method，After entering the while loop, the comparator will be compared infinitely getComparator(). Compare (seekkey, topkey), the next cycle will not exit,and The value of the scan variable is not changing
> !image-2021-12-28-22-17-36-088.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)