You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Kyle Purtell (Jira)" <ji...@apache.org> on 2021/11/10 17:18:00 UTC
[jira] [Updated] (HBASE-26440) Region opens sometimes fail when
bucket cache is in use
[ https://issues.apache.org/jira/browse/HBASE-26440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Kyle Purtell updated HBASE-26440:
----------------------------------------
Description:
After a split, I think, the region is reopened, and:
2021-11-10 16:47:48,929 ERROR [RS_OPEN_REGION-regionserver/ip-172-31-63-83:8120-1] regionserver.HRegion: Could not initialize all stores for the region=IntegrationTestLoadCommonCrawl,,1636562865609.e2df2061bcdc037070041de734a187ff.
Caused by: java.io.IOException: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:577)
at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:534)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:298)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6546)
Caused by: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.validateBlockAddition(BlockCacheUtil.java:205)
at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCacheUtil.java:237)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.shouldReplaceExistingCacheBlock(BucketCache.java:446)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlockWithWait(BucketCache.java:431)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlock(BucketCache.java:417)
at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.cacheBlock(CombinedBlockCache.java:64)
at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.lambda$readBlock$2(HFileReaderImpl.java:1346)
The open fails but another attempt somewhere else succeeds.
was:
After a split, I think, the region is reopened, and:
2021-11-10 16:47:48,929 ERROR [RS_OPEN_REGION-regionserver/ip-172-31-63-83:8120-1] regionserver.HRegion: Could not initialize all stores for the region=IntegrationTestLoadCommonCrawl,,1636562865609.e2df2061bcdc037070041de734a187ff.
Caused by: java.io.IOException: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:577)
at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:534)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:298)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6546)
Caused by: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.validateBlockAddition(BlockCacheUtil.java:205)
at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCacheUtil.java:237)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.shouldReplaceExistingCacheBlock(BucketCache.java:446)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlockWithWait(BucketCache.java:431)
at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlock(BucketCache.java:417)
at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.cacheBlock(CombinedBlockCache.java:64)
at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.lambda$readBlock$2(HFileReaderImpl.java:1346)
The open fails but another attempt somewhere else succeeds.
> Region opens sometimes fail when bucket cache is in use
> -------------------------------------------------------
>
> Key: HBASE-26440
> URL: https://issues.apache.org/jira/browse/HBASE-26440
> Project: HBase
> Issue Type: Bug
> Components: BlockCache, BucketCache, regionserver
> Affects Versions: 2.4.8
> Reporter: Andrew Kyle Purtell
> Priority: Major
>
> After a split, I think, the region is reopened, and:
> 2021-11-10 16:47:48,929 ERROR [RS_OPEN_REGION-regionserver/ip-172-31-63-83:8120-1] regionserver.HRegion: Could not initialize all stores for the region=IntegrationTestLoadCommonCrawl,,1636562865609.e2df2061bcdc037070041de734a187ff.
> Caused by: java.io.IOException: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
> cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
> at org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:577)
> at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:534)
> at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:298)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6546)
> Caused by: java.lang.RuntimeException: Cached block contents differ, which should not have happened.
> cacheKey:1e395fa6f0584ebc9fd831f1451dc4ba.cbb4e3a74e703805898a2a3bf0af94e7_48543272
> at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.validateBlockAddition(BlockCacheUtil.java:205)
> at org.apache.hadoop.hbase.io.hfile.BlockCacheUtil.shouldReplaceExistingCacheBlock(BlockCacheUtil.java:237)
> at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.shouldReplaceExistingCacheBlock(BucketCache.java:446)
> at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlockWithWait(BucketCache.java:431)
> at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.cacheBlock(BucketCache.java:417)
> at org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.cacheBlock(CombinedBlockCache.java:64)
> at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.lambda$readBlock$2(HFileReaderImpl.java:1346)
> The open fails but another attempt somewhere else succeeds.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)