You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "andyhuangdev (via GitHub)" <gi...@apache.org> on 2026/10/15 03:12:57 UTC

[PR] HDDS-17013. Remove interrupted container import files at startup [ozone]

andyhuangdev opened a new pull request, #11547:
URL: https://github.com/apache/ozone/pull/11547

   ## What changes were proposed in this pull request?
   
   Container imports interrupted by a datanode crash leave tarballs and partially extracted directories under the volume's tmp/container-copy directory. These files are not cleaned up on restart and can accumulate across crashes, consuming disk space.
   
   This patch cleans the directory in ContainerReader before loading containers at datanode startup. Cleanup runs before the replication server starts, ensuring no imports are in progress. The debug CLI preserves these files, and cleanup failures are logged without preventing container loading.
   
   ## What is the link to the Apache JIRA
   
   [HDDS-17013](https://issues.apache.org/jira/browse/HDDS-17013)
   
   ## How was this patch tested?
   
   Regression tests were added first and confirmed to fail across all eight container layout/schema combinations before the fix. They verify removal of tarballs and extracted directories, preservation in debug mode, and continued container loading when cleanup fails.
   
   After the fix, all 147 tests passed across TestContainerReader, TestSendContainerRequestHandler, TestContainerImporter, TestGrpcReplicationService, and TestOzoneContainer.
   
   Full-project checkstyle and git diff --check also passed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-17013. Remove interrupted container import files at startup [ozone]

Posted by "chungen0126 (via GitHub) via issues" <is...@ozone.apache.org>.
chungen0126 commented on code in PR #11547:
URL: https://github.com/apache/ozone/pull/11547#discussion_r4490068668


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java:
##########
@@ -103,6 +108,18 @@ public void run() {
     }
   }
 
+  private void cleanupContainerCopyDir() {
+    // Startup loads containers before starting the replication server, so no imports can be in progress here.
+    try {
+      File copyDir = ContainerImporter.getUntarDirectory(hddsVolume).toFile();

Review Comment:
   Right, I should have been clearer. I was referring to the other calls of this method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-17013. Remove interrupted container import files at startup [ozone]

Posted by "chungen0126 (via GitHub) via issues" <is...@ozone.apache.org>.
chungen0126 commented on code in PR #11547:
URL: https://github.com/apache/ozone/pull/11547#discussion_r4479288799


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java:
##########
@@ -103,6 +108,18 @@ public void run() {
     }
   }
 
+  private void cleanupContainerCopyDir() {
+    // Startup loads containers before starting the replication server, so no imports can be in progress here.
+    try {
+      File copyDir = ContainerImporter.getUntarDirectory(hddsVolume).toFile();
+      if (copyDir.exists()) {
+        FileUtils.cleanDirectory(copyDir);
+      }
+    } catch (IOException | RuntimeException e) {
+      LOG.warn("Failed to remove leftover container import files under {}", hddsVolume.getVolumeRootDir(), e);
+    }
+  }

Review Comment:
   Thanks @andyhuangdev for working on this. I'd suggest listing the files inside the directory and deleting them individually. If deleting a specific file fails, we can log its path.
   
   The difference from the current approach is that if any file fails to be deleted, the logs will clearly show which one it is, allowing the user to manually remove it based on the log.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-17013. Remove interrupted container import files at startup [ozone]

Posted by "adoroszlai (via GitHub) via issues" <is...@ozone.apache.org>.
adoroszlai merged PR #11547:
URL: https://github.com/apache/ozone/pull/11547


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-17013. Remove interrupted container import files at startup [ozone]

Posted by "chungen0126 (via GitHub) via issues" <is...@ozone.apache.org>.
chungen0126 commented on code in PR #11547:
URL: https://github.com/apache/ozone/pull/11547#discussion_r4493639537


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java:
##########
@@ -103,6 +108,29 @@ public void run() {
     }
   }
 
+  private void cleanupContainerCopyDir() {
+    // Startup loads containers before starting the replication server, so no imports can be in progress here.
+    try {
+      File copyDir = ContainerImporter.getUntarDirectory(hddsVolume).toFile();
+      if (copyDir.exists()) {
+        File[] files = copyDir.listFiles();
+        if (files == null) {
+          throw new IOException("Failed to list " + copyDir);
+        }

Review Comment:
   ```suggestion
           if (files == null) {
             LOG.warn("Failed to list files in container import directory: {}", copyDir);
             return;
           }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-17013. Remove interrupted container import files at startup [ozone]

Posted by "chungen0126 (via GitHub) via issues" <is...@ozone.apache.org>.
chungen0126 commented on code in PR #11547:
URL: https://github.com/apache/ozone/pull/11547#discussion_r4493597066


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java:
##########
@@ -103,6 +108,29 @@ public void run() {
     }
   }
 
+  private void cleanupContainerCopyDir() {
+    // Startup loads containers before starting the replication server, so no imports can be in progress here.
+    try {
+      File copyDir = ContainerImporter.getUntarDirectory(hddsVolume).toFile();
+      if (copyDir.exists()) {
+        File[] files = copyDir.listFiles();
+        if (files == null) {
+          throw new IOException("Failed to list " + copyDir);
+        }
+        for (File file : files) {
+          try {
+            FileUtils.forceDelete(file);
+          } catch (IOException | RuntimeException e) {
+            LOG.warn("Failed to remove leftover container import file {}", file.getAbsolutePath(), e);

Review Comment:
   I suggest not to log exception. It will be to long.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-17013. Remove interrupted container import files at startup [ozone]

Posted by "adoroszlai (via GitHub) via issues" <is...@ozone.apache.org>.
adoroszlai commented on PR #11547:
URL: https://github.com/apache/ozone/pull/11547#issuecomment-6709540905

   Thanks @andyhuangdev for the patch, @chungen0126, @smengcl for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-17013. Remove interrupted container import files at startup [ozone]

Posted by "chungen0126 (via GitHub) via issues" <is...@ozone.apache.org>.
chungen0126 commented on code in PR #11547:
URL: https://github.com/apache/ozone/pull/11547#discussion_r4493656242


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java:
##########
@@ -103,6 +108,29 @@ public void run() {
     }
   }
 
+  private void cleanupContainerCopyDir() {
+    // Startup loads containers before starting the replication server, so no imports can be in progress here.
+    try {
+      File copyDir = ContainerImporter.getUntarDirectory(hddsVolume).toFile();
+      if (copyDir.exists()) {
+        File[] files = copyDir.listFiles();
+        if (files == null) {
+          throw new IOException("Failed to list " + copyDir);
+        }
+        for (File file : files) {
+          try {
+            FileUtils.forceDelete(file);
+          } catch (IOException | RuntimeException e) {
+            LOG.warn("Failed to remove leftover container import file {}", file.getAbsolutePath(), e);
+          }
+        }
+      }
+    } catch (IOException | RuntimeException e) {
+      LOG.warn("Failed to remove leftover container import files under {}: {}",
+          hddsVolume.getVolumeRootDir(), e.getMessage(), e);

Review Comment:
   I don't think we need to log here. Simply logging when `files == null` and returning there should be enough.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-17013. Remove interrupted container import files at startup [ozone]

Posted by "chungen0126 (via GitHub) via issues" <is...@ozone.apache.org>.
chungen0126 commented on code in PR #11547:
URL: https://github.com/apache/ozone/pull/11547#discussion_r4479444525


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java:
##########
@@ -103,6 +108,18 @@ public void run() {
     }
   }
 
+  private void cleanupContainerCopyDir() {
+    // Startup loads containers before starting the replication server, so no imports can be in progress here.
+    try {
+      File copyDir = ContainerImporter.getUntarDirectory(hddsVolume).toFile();

Review Comment:
   I don't think this method actually throws any exceptions. We could remove throws so callers don't need to wrap it in a try-catch every time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-17013. Remove interrupted container import files at startup [ozone]

Posted by "andyhuangdev (via GitHub) via issues" <is...@ozone.apache.org>.
andyhuangdev commented on code in PR #11547:
URL: https://github.com/apache/ozone/pull/11547#discussion_r4480344434


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java:
##########
@@ -103,6 +108,18 @@ public void run() {
     }
   }
 
+  private void cleanupContainerCopyDir() {
+    // Startup loads containers before starting the replication server, so no imports can be in progress here.
+    try {
+      File copyDir = ContainerImporter.getUntarDirectory(hddsVolume).toFile();

Review Comment:
   The cleanup method still needs the try-catch because FileUtils.cleanDirectory() can throw IOException, and cleanup failures should not prevent container loading.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org