You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Peter Somogyi (Jira)" <ji...@apache.org> on 2021/07/21 15:39:00 UTC
[jira] [Created] (HBASE-26107) MOB compaction with missing files
catches incorrect exception
Peter Somogyi created HBASE-26107:
-------------------------------------
Summary: MOB compaction with missing files catches incorrect exception
Key: HBASE-26107
URL: https://issues.apache.org/jira/browse/HBASE-26107
Project: HBase
Issue Type: Bug
Components: mob
Affects Versions: 3.0.0-alpha-1
Reporter: Peter Somogyi
Assignee: Peter Somogyi
The MOB compaction catches FileNotFoundException when {{hbase.unsafe.mob.discard.miss}} is true to handle missing MOB cells. The FNFE is wrapped in DoNotRetryIOException so the compaction fails for the given region.
{noformat}
2021-07-21 13:51:05,880 WARN org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only when first upgrading to a version with the distributed mob compaction feature on a cluster that has experienced MOB data corruption.
2021-07-21 13:51:05,880 WARN org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: hbase.unsafe.mob.discard.miss=true. This is unsafe setting recommended only when first upgrading to a version with the distributed mob compaction feature on a cluster that has experienced MOB data corruption.
2021-07-21 13:51:05,880 INFO org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true optimized configured=false optimized enabled=false maximum MOB file size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB family=test_cf region=3a2ee81f9244c39ba61d694e616c1a89]
2021-07-21 13:51:05,880 INFO org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor: Compact MOB=true optimized configured=false optimized enabled=false maximum MOB file size=1073741824 major=true store=[table=IntegrationTestIngestWithMOB family=test_cf region=7a96f55bb9ae04500a06cbaef02da6a3]
2021-07-21 13:51:05,888 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
2021-07-21 13:51:05,891 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,88888888,1626787996628.8fd002bda07755decda67b7084d1e0f6.
2021-07-21 13:51:05,895 ERROR org.apache.hadoop.hbase.regionserver.HMobStore: The mob file 1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3 could not be found in the locations [hdfs://example.com:8020/hbase/mobdir/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce5
3c2/test_cf, hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf] or it is corrupt
2021-07-21 13:51:05,895 INFO org.apache.hadoop.hbase.regionserver.throttle.PressureAwareThroughputController: 7a96f55bb9ae04500a06cbaef02da6a3#test_cf#compaction#1 average throughput is 0.07 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 10.00 MB/second
2021-07-21 13:51:05,908 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Compacting IntegrationTestIngestWithMOB,dddddddd,1626787996628.53186ca5008e3a964eee5f96ee3f1b26.
2021-07-21 13:51:05,997 ERROR org.apache.hadoop.hbase.regionserver.CompactSplit: Compaction failed Request=regionName=IntegrationTestIngestWithMOB,11111111,1626787996628.7a96f55bb9ae04500a06cbaef02da6a3., storeName=test_cf, fileCount=1, fileSize=110.6 M (110.6 M), priority=1, time=1626875465819
java.io.IOException: Mob compaction failed for region: 7a96f55bb9ae04500a06cbaef02da6a3
at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:575)
at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327)
at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.compact(DefaultMobStoreCompactor.java:227)
at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407)
at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2183)
at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:633)
at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:675)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.io.FileNotFoundException: File does not exist: hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:438)
at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:328)
at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:307)
at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:366)
... 10 more
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://example.com:8020/hbase/archive/data/default/IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53c2/test_cf/1bbd886460827015e5d605ed44252251202107200e5065290b424e38992f5556d9943b6a_7a96f55bb9ae04500a06cbaef02da6a3
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1500)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1493)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1508)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:140)
at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:104)
at org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:220)
at org.apache.hadoop.hbase.mob.CachedMobFile.create(CachedMobFile.java:49)
at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:220)
at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:399)
... 13 more
2021-07-21 13:51:06,013 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction of test_cf in IntegrationTestIngestWithMOB,77777777,1626787996628.c71cad04514b17ee86a407490bd27424.
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)