[PR] HBASE-30291 Load backup HFile references once per cleaner run [hbase]

["Sigma-Ma (via GitHub)" <gi...@apache.org>]

Sigma-Ma opened a new pull request, #8539:
URL: https://github.com/apache/hbase/pull/8539

   JIRA: https://issues.apache.org/jira/browse/HBASE-30291
   
   ## What changes were proposed in this pull request?
   
   This pull request changes `BackupHFileCleaner` to load the referenced HFile names once in `preClean()` and reuse the resulting snapshot for all `getDeletableFiles()` calls in the same cleaner cycle.
   
   The snapshot is published only after loading completes. If loading the references fails, the snapshot is marked unavailable and all files are kept for that cleaner cycle.
   
   The existing `previousCleaningCompletionTimestamp` protection is preserved to prevent recently archived HFiles from being deleted before the next snapshot refresh.
   
   Tests are added to verify that:
   
   - multiple `getDeletableFiles()` calls in one cleaner cycle load the references only once;
   - the snapshot is refreshed in the next cleaner cycle;
   - a failed refresh does not reuse a stale snapshot or make files deletable.
   
   ## Why are the changes needed?
   
   `BackupHFileCleaner#getDeletableFiles()` is invoked concurrently for each non-empty archive directory. It currently scans the backup system tables on every invocation to load the same set of backed-up tables and bulk-loaded HFile references.
   
   On clusters with many regions, this causes tens of thousands of redundant system-table scans in a single cleaner cycle. The resulting I/O load can prevent the cleaner from keeping up, causing archived HFiles to accumulate.
   
   Loading the references once per cleaner cycle removes these redundant scans without introducing a cross-cycle cache or changing the general `CleanerChore` framework.
   
   ## How was this patch tested?
   
   The following checks were run successfully:
   
   ```shell
   mvn -pl hbase-backup -am \
     -Dtest=TestBackupHFileCleaner \
     -Dsurefire.failIfNoSpecifiedTests=false \
     -DskipITs \
     test
   
   mvn -pl hbase-backup -DskipITs test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] HBASE-30291 Load backup HFile references once per cleaner run [hbase]

["Sigma-Ma (via GitHub)" <gi...@apache.org>]

Sigma-Ma commented on PR #8539:
URL: https://github.com/apache/hbase/pull/8539#issuecomment-5357638887

   Hi @ndimiduk and @Apache9,
   
   Could you please review this PR when you have a chance?
   
   This patch updates `BackupHFileCleaner` to load the backup HFile references once in `preClean()` and reuse the snapshot for all `getDeletableFiles()` calls within the same cleaner cycle. It preserves the existing fail-closed behavior and the protection for recently created files, and adds focused tests for the cleaner-cycle snapshot behavior.
   
   The Yetus checks currently show `action_required`. If maintainer approval is required to run them, could you please help approve the workflows as well?
   
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org