You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Anoop Sam John (Jira)" <ji...@apache.org> on 2021/04/30 12:45:00 UTC
[jira] [Commented] (HBASE-25698) Persistent
IllegalReferenceCountException at scanner open
[ https://issues.apache.org/jira/browse/HBASE-25698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337365#comment-17337365 ]
Anoop Sam John commented on HBASE-25698:
----------------------------------------
So the trace is coming in Bloom check and its the release of a Bloom block created this exception. That means no way it would have come from Off heap BC. When offheap BC is enabled, non DATA blocks will be always in LRU on heap cache. (hbase.blockcache.use.external is not set true right. Then only we create CombinedBC with L2 as VictimCache for L1)
So this block under issue, would have come either from L1 LRU cache or the block is created by a read from FS. Doing more checks in those areas. Am not targeting BucketCache check. Pls raise concern if any.
> Persistent IllegalReferenceCountException at scanner open
> ---------------------------------------------------------
>
> Key: HBASE-25698
> URL: https://issues.apache.org/jira/browse/HBASE-25698
> Project: HBase
> Issue Type: Bug
> Components: HFile, Scanners
> Affects Versions: 2.4.2
> Reporter: Andrew Kyle Purtell
> Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> Persistent scanner open failure with offheap read path enabled.
> Not sure how it happened. Test scenario was HBase 1 cluster replicating to HBase 2 cluster. ITBLL as data generator at source, calm policy only. Scanner open errors on sink HBase 2 cluster later during ITBLL verify phase. Sink schema settings bloom=ROW encoding=FAST_DIFF compression=NONE.
> {noformat}
> Caused by: org.apache.hbase.thirdparty.io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
> at org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
> at org.apache.hbase.thirdparty.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
> at org.apache.hbase.thirdparty.io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
> at org.apache.hadoop.hbase.nio.ByteBuff.release(ByteBuff.java:79)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock.release(HFileBlock.java:429)
> at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:109)
> at org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
> at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
> at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:471)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:249)
> at org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2177)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2168)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:7172)
> {noformat}
> Bloom filter type on all files here is ROW, block encoding is FAST_DIFF:
> {noformat}
> hbase:017:0> describe "IntegrationTestBigLinkedList"
> Table IntegrationTestBigLinkedList is ENABLED
> IntegrationTestBigLinkedList
> COLUMN FAMILIES DESCRIPTION
> {NAME => 'big', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIF
> F', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'meta', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {NAME => 'tiny', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DI
> FF', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)