You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Shrinidhi Shridhar Talpankar (Jira)" <ji...@apache.org> on 2026/06/04 11:19:00 UTC
[jira] [Commented] (HBASE-30154) HFiles Deleted While Still Referenced by HFileLinks in Cloned Table
[ https://issues.apache.org/jira/browse/HBASE-30154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18086070#comment-18086070 ]
Shrinidhi Shridhar Talpankar commented on HBASE-30154:
------------------------------------------------------
Bump
> HFiles Deleted While Still Referenced by HFileLinks in Cloned Table
> -------------------------------------------------------------------
>
> Key: HBASE-30154
> URL: https://issues.apache.org/jira/browse/HBASE-30154
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 2.6.2
> Reporter: Shrinidhi Shridhar Talpankar
> Priority: Major
> Labels: pull-request-available
>
> There is a race condition where Hfiles referenced by cloned table can be permanently deleted by HFileLinkCleaner causing data loss. We've seen this happening on HBase 2.6.2, but looking at the latest code it seems that the race can still happen there.
> The sequence of operations that leads to this issue:
> # Regions merged in the parent table.
> # Snapshot taken while a merged region still holds reference files, and that snapshot is then cloned. RestoreSnapshotHelper.restoreReferenceFile() creates HFileLink Reference files in the clone (e.g. srcTable=srcRegion-hfile.cloneRegion) but does not write back-references to the archive directory.
> # Parent table compacted. CatalogJanitor GCs the pre-merge regions, archiving their HFiles.
> # HFileLinkCleaner sees no back-references for those archived HFiles and deletes them.
> # Every subsequent region open/scan on the cloned table fails with FileNotFoundException.
>
> Logs evidence supporting the above flow:
> h4. 07:41:16Z — Merge 1 Initiated (RegionNormalizer)
> The {{RegionNormalizerWorker}} identified regions {{e95408fd}} (4511 MB) and {{565ae77f}} (128 MB) as candidates for merging due to size imbalance. It submitted {{MergeTableRegionsProcedure}} (pid=2307105).
> INFO [normalizer-worker-0] master.HMaster: merge regions [e95408fd3383e18df6f3a171a6bfc05d, 565ae77f2c2e16d6ee539dd95540a240] INFO [PEWorker-9] procedure.MasterProcedureScheduler: Took xlock for pid=2307105, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE
> Source regions unassigned from regionserver-11 and regionserver-13.
> h4. 07:41:18Z — Merge 1 Completes → {{43bb39bf}} created
> INFO [PEWorker-14] procedure2.ProcedureExecutor: Finished pid=2307105, state=SUCCESS; MergeTableRegionsProcedure table=aeris_v2, regions=[e95408fd3383e18df6f3a171a6bfc05d, 565ae77f2c2e16d6ee539dd95540a240], force=false in 2.0710 sec
> Merged region {{43bb39bf414f2c2bb4c453a7c4ace246}} is assigned to regionserver-11. Its store directory contains HFileLinks pointing back to the source region directories.
> h4. 09:41:16Z — Merge 2 Initiated (RegionNormalizer)
> The normalizer identified regions {{449b39800}} (2964 MB) and {{c8bcf55f}} (1694 MB) and submitted {{MergeTableRegionsProcedure}} (pid=2307118).
> INFO [normalizer-worker-0] master.HMaster: merge regions [449b39800c6efb9ce3eca36410d292c1, c8bcf55fde67e62655c51caec1a2ca96]
> h4. 09:41:19Z — Merge 2 Completes → {{d5243cea}} created
> INFO [PEWorker-9] procedure2.ProcedureExecutor: Finished pid=2307118, state=SUCCESS; MergeTableRegionsProcedure table=aeris_v2, regions=[449b39800c6efb9ce3eca36410d292c1, c8bcf55fde67e62655c51caec1a2ca96], force=false in 2.7930 sec
> Merged region {{d5243cea04f0a22e4292c1d006580385}} is assigned to regionserver-88. Contains HFileLinks to {{449b39800}} and {{c8bcf55f}} source HFiles.
> h4. 14:02:59Z — Major Compaction Starts on {{d5243cea}} (regionserver-88)
> The merged region immediately triggers a high-priority major compaction (labeled as "recently split daughter region" — this is the HBase labeling for merged regions). The compaction includes 9 files, among which are HFileLinks to the source regions:
> INFO [longCompactions-0] regionserver.HStore: Starting compaction of d5243cea.../e... hdfs://.../d5243cea.../e/d2d48db764e645d1ad5eb2bb1d1409c0.c8bcf55fde67e62655c51caec1a2ca96->... hdfs://.../d5243cea.../e/3c9628b6aeaa4ae5944667d35ddf0fc3.c8bcf55fde67e62655c51caec1a2ca96->... totalSize=4.6 G
> *This compaction will take 44 minutes and 53 seconds.*
> h4. ~14:24Z — ⚠️ Snapshot Taken WHILE Compaction is Running
> A snapshot {{aeris_v2_triforce_snapshot_1773843841734812535}} is taken of {{{}aeris_v2{}}}. The snapshot name encodes a Unix timestamp of approximately {{1773843841}} seconds, corresponding to ~14:24 UTC March 18, 2026.
> At this moment: - {{d5243cea}} still contains HFileLinks to {{{}449b39800{}}}/{{{}c8bcf55f{}}} (compaction has NOT finished) - {{43bb39bf}} still contains HFileLinks to {{{}e95408fd{}}}/{{{}565ae77f{}}}
> The snapshot manifest captures this state — the merged regions' HFileLinks to the source regions are part of the snapshot.
> h4. 14:24:15Z - 14:28:05Z — Clone Created: {{aeris_v2_triforce_archive_1773843841734812535}}
> {{CloneSnapshotProcedure}} runs, invoking {{RestoreSnapshotHelper}} which creates HFileLinks in the new archive table pointing directly to the source region directories (the ones being cleaned up after merge). Key HFileLinks added:
> INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper: Adding HFileLink 37211e9b8f2744f3a7e2707594b39fbb.565ae77f2c2e16d6ee539dd95540a240 from cloned region in snapshot aeris_v2_triforce_snapshot_1773843841734812535 to table=aeris_v2_triforce_archive_1773843841734812535 INFO [RestoreSnapshot-pool-7] snapshot.RestoreSnapshotHelper: Adding HFileLink 09aa8849c2d64984942a80b026e4a19e.449b39800c6efb9ce3eca36410d292c1 from cloned region in snapshot aeris_v2_triforce_snapshot_1773843841734812535 to table=aeris_v2_triforce_archive_1773843841734812535
> Clone procedure completes at 14:28:05Z:
> INFO [PEWorker-16] procedure.CloneSnapshotProcedure: Clone snapshot=aeris_v2_triforce_snapshot_1773843841734812535 on table=aeris_v2_triforce_archive_1773843841734812535 completed!
> h4. 14:47:52Z — ⚠️ Compaction of {{d5243cea}} Completes
> After 44 minutes and 53 seconds, the major compaction finishes:
> INFO [longCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,...d5243cea04f0a22e4292c1d006580385., storeName=d5243cea04f0a22e4292c1d006580385/e, duration=44mins, 53sec INFO [longCompactions-0] regionserver.HStore: Completed compaction of 9 (all) file(s) in d5243cea04f0a22e4292c1d006580385/e into 9e904c6e8b2247cf9c0463aecc1d67b1 (size=4.4 G)
> The HFileLinks to {{449b39800}} and {{c8bcf55f}} are *removed* from {{{}d5243cea{}}}'s store directory. These source region files are no longer referenced by any live region in {{{}aeris_v2{}}}. This is the gate condition for {{GCMultipleMergedRegionsProcedure}} to proceed.
> h4. 14:51:50Z — 💥 GCMultipleMergedRegionsProcedure Deletes Source Regions (Group 2)
> {{GCMultipleMergedRegionsProcedure}} (pid=2326702) runs immediately (only 93ms total execution):
> INFO [PEWorker-11] procedure.MasterProcedureScheduler: Took xlock for pid=2326702, state=RUNNABLE:GC_MERGED_REGIONS_PREPARE; GCMultipleMergedRegionsProcedure child=d5243cea04f0a22e4292c1d006580385, parents:[449b39800c6efb9ce3eca36410d292c1], [c8bcf55fde67e62655c51caec1a2ca96] INFO [PEWorker-4] hbase.MetaTableAccessor: Deleted aeris_v2,...449b39800c6efb9ce3eca36410d292c1. INFO [PEWorker-11] hbase.MetaTableAccessor: Deleted aeris_v2,...c8bcf55fde67e62655c51caec1a2ca96. INFO [PEWorker-10] procedure2.ProcedureExecutor: Finished pid=2326702, state=SUCCESS; GCMultipleMergedRegionsProcedure child=d5243cea04f0a22e4292c1d006580385, parents:[449b39800c6efb9ce3eca36410d292c1], [c8bcf55fde67e62655c51caec1a2ca96] in 93 msec
> The HDFS directories for {{449b39800c6efb9ce3eca36410d292c1}} and {{c8bcf55fde67e62655c51caec1a2ca96}} — including both {{/hbase/data/}} and {{/hbase/archive/}} paths — are deleted. The HFileLinks in {{aeris_v2_triforce_archive}} now point to non-existent paths.
> h4. 15:27:37Z — Compaction Triggered on {{43bb39bf}}
> A flush on {{43bb39bf}} completes, requesting compaction:
> INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of 43bb39bf414f2c2bb4c453a7c4ace246 in 73ms, compaction requested=true
> h4. 17:15:36Z — First FileNotFoundException (Group 2: {{{}449b39800{}}})
> {{aeris_v2_triforce_archive}} attempts to read through HFileLinks, fails:
> java.io.FileNotFoundException: HFileLink locations=[ hdfs://aeris/hbase/data/default/aeris_v2/449b39800c6efb9ce3eca36410d292c1/e/09aa8849c2d64984942a80b026e4a19e, hdfs://aeris/hbase/archive/data/default/aeris_v2/449b39800c6efb9ce3eca36410d292c1/e/09aa8849c2d64984942a80b026e4a19e, hdfs://aeris/hbase/.tmp/data/default/aeris_v2/449b39800c6efb9ce3eca36410d292c1/e/09aa8849c2d64984942a80b026e4a19e, hdfs://aeris/hbase/mobdir/data/default/aeris_v2/449b39800c6efb9ce3eca36410d292c1/e/09aa8849c2d64984942a80b026e4a19e]
> All 4 candidate locations checked by {{HFileLink.open()}} are missing.
> h4. 18:06:05Z — Compaction of {{43bb39bf}} Completes (regionserver-11)
> INFO [longCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,...43bb39bf414f2c2bb4c453a7c4ace246., storeName=e, fileCount=6, fileSize=4.5 G, duration=1mins, 35sec
> HFileLinks to {{e95408fd}} and {{565ae77f}} are removed from {{{}43bb39bf{}}}. GC gate condition met for Group 1.
> h4. 18:11:52Z — 💥 GCMultipleMergedRegionsProcedure Deletes Source Regions (Group 1)
> INFO [PEWorker-13] hbase.MetaTableAccessor: Deleted aeris_v2,...565ae77f2c2e16d6ee539dd95540a240. INFO [PEWorker-7] hbase.MetaTableAccessor: Deleted aeris_v2,...e95408fd3383e18df6f3a171a6bfc05d. INFO [PEWorker-12] procedure2.ProcedureExecutor: Finished pid=2357877, state=SUCCESS; GCMultipleMergedRegionsProcedure child=43bb39bf414f2c2bb4c453a7c4ace246, parents:[e95408fd3383e18df6f3a171a6bfc05d], [565ae77f2c2e16d6ee539dd95540a240] in 73 msec
> HDFS directories for {{e95408fd3383e18df6f3a171a6bfc05d}} and {{565ae77f2c2e16d6ee539dd95540a240}} are deleted.
> h4. 21:16:49Z — First FileNotFoundException (Group 1: {{{}565ae77f{}}})
> java.io.FileNotFoundException: HFileLink locations=[ hdfs://aeris/hbase/data/default/aeris_v2/565ae77f2c2e16d6ee539dd95540a240/e/37211e9b8f2744f3a7e2707594b39fbb, ...]
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)