You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Peter Somogyi (Jira)" <ji...@apache.org> on 2021/07/22 08:17:00 UTC
[jira] [Resolved] (HBASE-26107) MOB compaction with missing files
catches incorrect exception
[ https://issues.apache.org/jira/browse/HBASE-26107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Somogyi resolved HBASE-26107.
-----------------------------------
Fix Version/s: 3.0.0-alpha-2
Resolution: Fixed
Thanks for the reviews [~zhangduo] and [~pankajkumar]! Pushed to master.
> MOB compaction with missing files catches incorrect exception
> -------------------------------------------------------------
>
> Key: HBASE-26107
> URL: https://issues.apache.org/jira/browse/HBASE-26107
> Project: HBase
> Issue Type: Bug
> Components: mob
> Affects Versions: 3.0.0-alpha-1
> Reporter: Peter Somogyi
> Assignee: Peter Somogyi
> Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> The MOB compaction catches FileNotFoundException when {{hbase.unsafe.mob.discard.miss}} is true to handle missing MOB cells. The FNFE is wrapped in DoNotRetryIOException so the compaction fails for the given region.
> {noformat}
> 2021-07-21 13:51:05,880 WARN org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only when first upgrading to a version with the distributed mob compaction feature on a cluster that has experienced MOB data corruption.
> 2021-07-21 13:51:05,880 WARN org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only when first upgrading to a version with the distributed mob compaction feature on a cluster that has experienced MOB data corruption.
> 2021-07-21 13:51:05,880 INFO org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true optimized configured=false optimized enabled=false maximum MOB file size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB family=test_cf region=3a2ee81f9244c39ba61d694e616c1a89]
> 2021-07-21 13:51:05,880 INFO org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true optimized configured=false optimized enabled=false maximum MOB file size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB family=test_cf region=7a96f55bb9ae04500a06cbaef02da6a3]
> 2021-07-21 13:51:05,888 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
> 2021-07-21 13:51:05,891 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,88888888,1626787996628.8fd002bda07755decda67b7084d1e0f6.
> 2021-07-21 13:51:05,895 ERROR org.apache.hadoop.hbase.regionserver.HMobStore: The mob file 1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3 could not be found in the locations [hdfs://example.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce5
> 3c2/test_cf, hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf] or it is corrupt
> 2021-07-21 13:51:05,895 INFO org.apache.hadoop.hbase.regionserver.throttle.PressureAwareThroughputController: 7a96f55bb9ae04500a06cbaef02da6a3#test_cf#compaction#1 average throughput is 0.07 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 10.00 MB/second
> 2021-07-21 13:51:05,908 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,dddddddd,1626787996628.53186ca5008e3a964eee5f96ee3f1b26.
> 2021-07-21 13:51:05,997 ERROR org.apache.hadoop.hbase.regionserver.CompactSplit: Compaction failed Request=regionName=IntegrationTestIngestWithMOB,11111111,1626787996628.7a96f55bb9ae04500a06cbaef02da6a3., storeName=test_cf, fileCount=1, fileSize=110.6 M (110.6 M), priority=1, time=1626875465819
> java.io.IOException: Mob compaction failed for region: 7a96f55bb9ae04500a06cbaef02da6a3
> at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:575)
> at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
> at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.compact(DefaultMobStoreCompactor.java:227)
> at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2183)
> at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:633)
> at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:675)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.io.FileNotFoundException: File does not exist: hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
> at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:438)
> at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:328)
> at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:307)
> at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:366)
> ... 10 more
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
> at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1500)
> at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1493)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1508)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:140)
> at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:104)
> at org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:220)
> at org.apache.hadoop.hbase.mob.CachedMobFile.create(CachedMobFile.java:49)
> at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:220)
> at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:399)
> ... 13 more
> 2021-07-21 13:51:06,013 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction of test_cf in IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)