You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "smengcl (via GitHub)" <gi...@apache.org> on 2026/07/08 21:15:18 UTC

[PR] HDDS-15788. Timeout in testSnapshotAndKeyDeletionBackgroundServices [ozone]

smengcl opened a new pull request, #10692:
URL: https://github.com/apache/ozone/pull/10692

   ## What changes were proposed in this pull request?
   
   10s wait looks to be too tight for rocksdb v10. Testing 30s
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-15788
   
   ## How was this patch tested?
   
   - CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15788. Fix SST filtering BOOTSTRAP_LOCK contention introduced by RocksDB 10 upgrade [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #10692:
URL: https://github.com/apache/ozone/pull/10692#issuecomment-4928476118

   Thanks @adoroszlai and @sadanand48 for reviewing this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15788. Don't hold the OM bootstrap lock during SST file deletion wait [ozone]

Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #10692:
URL: https://github.com/apache/ozone/pull/10692#discussion_r3550130898


##########
hadoop-hdds/managed-rocksdb/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedRocksDB.java:
##########
@@ -211,7 +218,7 @@ public void deleteSstFileRange(
     } catch (RocksDBException e) {
       throw new RocksDatabaseException("Failed to delete " + file, e);
     }
-    ManagedRocksObjectUtils.waitForFileDelete(file, Duration.ofSeconds(60));
+    return file;

Review Comment:
   ```java
   if (getLiveMetadataForSSTFiles(get()).containsKey(
       FilenameUtils.getBaseName(fileToBeDeleted.fileName()))) {
     throw new RocksDatabaseException("deleteFilesInRanges was a no-op for " + file);
   }
   ```
   I think we can remove the wait altogether, If the file is removed off of the live file metadata it is guaranteed to be physically removed from disk. Instead of the wait we just check if the file is still in the metadata list and if yes throw an exception. This is instantaneous and avoids long poll for 60s.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15788. Fix SST filtering BOOTSTRAP_LOCK contention introduced by RocksDB 10 upgrade [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10692:
URL: https://github.com/apache/ozone/pull/10692#issuecomment-4928106008

   Thanks @smengcl for the patch, @sadanand48 for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15788. Don't hold the OM bootstrap lock during SST file deletion wait [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10692:
URL: https://github.com/apache/ozone/pull/10692#issuecomment-4922529390

   Thanks @smengcl for working on this.
   
   > * 10x10 run with the fix (100% pass rate): https://github.com/smengcl/hadoop-ozone/actions/runs/28994064808
   
   ```
   Running test: testSnapshotAndKeyDeletionBackgroundServices from TestSnapshotBackgroundServices
   ```
   
   > * 10x10 run w/o the fix for reference (50% pass rate): https://github.com/smengcl/hadoop-ozone/actions/runs/28989591533
   
   ```
   Running all tests from TestSnapshotBackgroundServices
   ```
   
   Previous run for single method passed even without the fix: https://github.com/smengcl/hadoop-ozone/actions/runs/28987422211
   
   To properly validate the fix, please run it with `ALL`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15788. Don't hold the OM bootstrap lock during SST file deletion wait [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #10692:
URL: https://github.com/apache/ozone/pull/10692#issuecomment-4922804983

   > Thanks @smengcl for working on this.
   > 
   > > * 10x10 run with the fix (100% pass rate): [smengcl/hadoop-ozone/actions/runs/28994064808](https://github.com/smengcl/hadoop-ozone/actions/runs/28994064808)
   > 
   > ```
   > Running test: testSnapshotAndKeyDeletionBackgroundServices from TestSnapshotBackgroundServices
   > ```
   > 
   > > * 10x10 run w/o the fix for reference (50% pass rate): [smengcl/hadoop-ozone/actions/runs/28989591533](https://github.com/smengcl/hadoop-ozone/actions/runs/28989591533)
   > 
   > ```
   > Running all tests from TestSnapshotBackgroundServices
   > ```
   > 
   > Previous run for single method passed even without the fix: [smengcl/hadoop-ozone/actions/runs/28987422211](https://github.com/smengcl/hadoop-ozone/actions/runs/28987422211)
   > 
   > To properly validate the fix, please run it with `ALL`.
   
   Good catch. I thought I ran `ALL` on the 10x10 branch with the fix but it didn't. Running now: https://github.com/smengcl/hadoop-ozone/actions/runs/29002349928


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15788. Don't hold the OM bootstrap lock during SST file deletion wait [ozone]

Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #10692:
URL: https://github.com/apache/ozone/pull/10692#discussion_r3550158791


##########
hadoop-hdds/managed-rocksdb/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedRocksDB.java:
##########
@@ -211,7 +218,7 @@ public void deleteSstFileRange(
     } catch (RocksDBException e) {
       throw new RocksDatabaseException("Failed to delete " + file, e);
     }
-    ManagedRocksObjectUtils.waitForFileDelete(file, Duration.ofSeconds(60));
+    return file;

Review Comment:
   via cursor
   > When deleteFilesInRanges successfully removes an SST from the live file set ([getLiveFilesMetaData()](https://github.com/facebook/rocksdb/wiki/Delete-Stale-Files)), RocksDB updates the MANIFEST via LogAndApply and then synchronously calls [PurgeObsoleteFiles()](https://github.com/facebook/rocksdb/blob/v10.10.1/db/db_impl/db_impl.cc#L5007-L5015), which unlinks table files through [DeleteDBFile()](https://github.com/facebook/rocksdb/blob/v10.10.1/db/db_impl/db_impl_files.cc#L377-L385) before the API returns — the same path [documented as immediately reclaiming space](https://github.com/facebook/rocksdb/wiki/Delete-A-Range-Of-Keys)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15788. Don't hold the OM bootstrap lock during SST file deletion wait [ozone]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10692:
URL: https://github.com/apache/ozone/pull/10692#discussion_r3549584873


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SstFilteringService.java:
##########
@@ -256,6 +260,21 @@ public BackgroundTaskResult call() throws Exception {
                 }
               }
             }
+            if (sstFilteringIssued) {
+              // Confirm the physical SST removal outside the bootstrap read lock
+              // and the open snapshot. On RocksDB 10 a range delete may not
+              // remove the file promptly, so this wait can take up to a minute;
+              // keeping it out of BOOTSTRAP_LOCK prevents it from starving the OM

Review Comment:
   The comment says the wait "can take up to a minute", but the code may wait up to 60s per file (and can therefore exceed a minute total when multiple SST files are returned). Adjust the wording to avoid under-stating the potential delay.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15788. Don't hold the OM bootstrap lock during SST file deletion wait [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #10692:
URL: https://github.com/apache/ozone/pull/10692#discussion_r3550752028


##########
hadoop-hdds/managed-rocksdb/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedRocksDB.java:
##########
@@ -211,7 +218,7 @@ public void deleteSstFileRange(
     } catch (RocksDBException e) {
       throw new RocksDatabaseException("Failed to delete " + file, e);
     }
-    ManagedRocksObjectUtils.waitForFileDelete(file, Duration.ofSeconds(60));
+    return file;

Review Comment:
   Thanks @sadanand48 . I think your approach is cleaner and better. Let me rerun the 10x10.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15788. Fix SST filtering BOOTSTRAP_LOCK contention introduced by RocksDB 10 upgrade [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #10692:
URL: https://github.com/apache/ozone/pull/10692


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org