[jira] [Resolved] (HDDS-16884) Intermittent timeout in TestSnapshotBackgroundServices#testBackupCompactionFilesPruningBackgroundService

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng resolved HDDS-16884.
-------------------------------
    Fix Version/s: 2.3.0
       Resolution: Fixed

> Intermittent timeout in TestSnapshotBackgroundServices#testBackupCompactionFilesPruningBackgroundService
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16884
>                 URL: https://issues.apache.org/jira/browse/HDDS-16884
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Huang Kuan Hao
>            Assignee: Huang Kuan Hao
>            Priority: Major
>              Labels: ozone-flaky-test, pull-request-available
>             Fix For: 2.3.0
>
>
> {{TestSnapshotBackgroundServices#testBackupCompactionFilesPruningBackgroundService}} intermittently times out waiting for the backup SST directory's file count to decrease after pruning is resumed.
> Observed CI failure: https://github.com/apache/ozone/actions/runs/35732766354/job/106767352881
> {code}
> TestSnapshotBackgroundServices.testBackupCompactionFilesPruningBackgroundService:504
>  -> checkIfCompactionBackupFilesWerePruned:574
>  -> TimeoutException
> {code}
> The unchanged full class also reproduced the timeout locally on master {{6818707698}} with JDK 21 (4 tests, 1 error):
> {code}
> mvn -pl :ozone-integration-test test -Dtest=TestSnapshotBackgroundServices -DskipShade -DskipRecon -DskipDocs
> {code}
> Snapshot creation does not guarantee a completed tracked compaction. The fixture relies on automatic compaction and shared-cluster state to produce eligible backup SST inputs. Passing diagnostic runs removed inputs from a previous test. A net directory count can also hide removals when new backup files are created.
> Disabling automatic compaction with the existing {{DBProfile.TEST}} reproduced the missing precondition: the backup directory, compaction log and DAG remained empty, and the original wait timed out. This establishes a fixture defect; the historical CI log does not contain enough metadata to confirm its exact trigger.
> The proposed fix forces a keyTable compaction using {{CompactDBUtil.compactTable}} with bottommost {{kForce}} while pruning is suspended, captures existing backup inputs from newly recorded compactions, asserts that they are non-empty, and waits for those files to disappear after pruning resumes. It preserves the leadership-transfer and snapshot-diff checks and the existing 10-second timeout. Default {{compactDB()}} produced no new inputs in a verification run, so forcing bottommost compaction is needed.
> Local validation of the test-only fix:
> * The controlled no-automatic-compaction probe fails with the original test and passes with the fix.
> * Five separate runs of the normal-profile full class passed: 20 tests, 0 failures/errors/skipped.
> * Checkstyle passed with 0 violations.
> Related: HDDS-9206 (resolved older pruning timeout), HDDS-10306 (shared test cluster).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org