You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Gargi-jais11 (via GitHub)" <gi...@apache.org> on 2026/05/19 07:30:02 UTC

[PR] HDDS-15311. [DiskBalancer] Fix DiskBalancer DN container Log Format [ozone]

Gargi-jais11 opened a new pull request, #10307:
URL: https://github.com/apache/ozone/pull/10307

   ## What changes were proposed in this pull request?
   DN container log file format as key worded columns is different in case of DiskBalancer compared to existing log lines. This may cause issue in any dn container formatter or parser in future.
   
   ```
   2026-05-15 04:24:11,501 | INFO  | ID=11 | Index=0 | BCSID=1374 | State=CLOSED | Volume=/hadoop-ozone/datanode/data/hdds | DataChecksum=47c5ddd3 |
   2026-05-15 05:02:20,188 | INFO  | ID=11 | SrcVolume=/hadoop-ozone/datanode/data/hdds | DestVolume=/data0/nvme1n1/hadoop-ozone/datanode/data/hdds | Size=268435456 bytes | TimeTaken=490 ms | Container is moved from SrcVolume to DestVolume |
   2026-05-15 07:12:08,285 | INFO  | ID=11 | Index=0 | BCSID=1374 | State=CLOSED | Volume=/data0/nvme1n1/hadoop-ozone/datanode/data/hdds | DataChecksum=47c5ddd3 | Container reconciled with peer a49d6e1b-ee75-46c5-84eb-3ebde58b686f(datanode1.com/xx.xx.xx.xx). No change in checksum. | 
   ```
   DiskBalancer should maintain the same structure.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-15311
   
   ## How was this patch tested?
   
   Tested manually.
   **Before:**
   ```
   2026-05-15 05:02:20,188 | INFO  | ID=11 | SrcVolume=/hadoop-ozone/datanode/data/hdds | DestVolume=/data0/nvme1n1/hadoop-ozone/datanode/data/hdds | Size=268435456 bytes | TimeTaken=490 ms | Container is moved from SrcVolume to DestVolume |
   ```
   
   **After:**
   ```
   2026-05-19 04:51:12,068 | INFO  | ID=13 | Index=0 | BCSID=16965 | State=CLOSED | Volume=/data/hdds1/hdds | DataChecksum=2cc81c5e |
   2026-05-19 04:51:12,069 | INFO  | ID=14 | Index=0 | BCSID=16973 | State=CLOSED | Volume=/data/hdds1/hdds | DataChecksum=595237d2 |
   2026-05-19 04:54:17,660 | INFO  | ID=13 | Index=0 | BCSID=16965 | State=CLOSED | SrcVolume=/data/hdds1/hdds | DestVolume=/data/hdds2/hdds | Size=967835648 bytes | TimeTaken=1777 ms | Container is moved from SrcVolume to DestVolume |
   2026-05-19 04:55:17,405 | INFO  | ID=14 | Index=0 | BCSID=16973 | State=CLOSED | SrcVolume=/data/hdds1/hdds | DestVolume=/data/hdds2/hdds | Size=967835648 bytes | TimeTaken=1520 ms | Container is moved from SrcVolume to DestVolume |
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15311. [DiskBalancer] Fix DiskBalancer DN container Log Format [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi merged PR #10307:
URL: https://github.com/apache/ozone/pull/10307


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15311. [DiskBalancer] Fix DiskBalancer DN container Log Format [ozone]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10307:
URL: https://github.com/apache/ozone/pull/10307#discussion_r3270777335


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerLogger.java:
##########
@@ -181,40 +181,38 @@ public static void logReconciled(ContainerData containerData, long oldDataChecks
   /**
    * Logged when a container is successfully moved from one data volume to another.
    *
-   * @param containerId The ID of the moved container.
+   * @param containerData The container after it has been moved to the destination volume.
    * @param sourceVolume The source volume path.
    * @param destinationVolume The destination volume path.
    * @param containerSize The size of data moved from container in bytes.
    * @param timeTaken The time taken for the move in milliseconds.
    */
-  public static void logMoveSuccess(long containerId, StorageVolume sourceVolume,
+  public static void logMoveSuccess(ContainerData containerData, StorageVolume sourceVolume,
       StorageVolume destinationVolume, long containerSize, long timeTaken) {
-    LOG.info(getMessage(containerId, sourceVolume, destinationVolume, containerSize, timeTaken));
+    LOG.info(getMessage(containerData,
+        "SrcVolume=" + sourceVolume,
+        "DestVolume=" + destinationVolume,
+        "Size=" + containerSize + " bytes",
+        "TimeTaken=" + timeTaken + " ms",
+        "Container is moved from SrcVolume to DestVolume"));
   }

Review Comment:
   Please add/extend a test to assert the move-success container log line includes the expected keyworded columns (ID/Index/BCSID/State + move fields). There are existing DiskBalancer task tests; capturing the `ContainerLogger.LOG_NAME` logger output during a successful move would help prevent future regressions in the log format relied on by tooling/parsers.



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java:
##########
@@ -636,8 +637,10 @@ public BackgroundTaskResult call() {
         if (moveSucceeded) {
           // Add current old container to pendingDeletionContainers.
           pendingDeletionContainers.put(System.currentTimeMillis() + replicaDeletionDelay, container);
-          ContainerLogger.logMoveSuccess(containerId, sourceVolume,
-              destVolume, containerSize, Time.monotonicNow() - startTime);
+          if (newContainer != null) {
+            ContainerLogger.logMoveSuccess(newContainer.getContainerData(), sourceVolume,
+                destVolume, containerSize, Time.monotonicNow() - startTime);
+          }

Review Comment:
   In the success path, `moveSucceeded` can remain true for unchecked exceptions (only `IOException` is caught). If an unchecked exception occurs before `newContainer` is assigned, the `finally` block will still enqueue the old container for deletion and record success metrics, which risks deleting the only good replica. Consider catching `Exception` (or at least `RuntimeException`) to set `moveSucceeded=false`, and/or assert `newContainer` is non-null when `moveSucceeded` is true and skip pending-deletion bookkeeping otherwise.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15311. [DiskBalancer] Fix DiskBalancer DN container Log Format [ozone]

Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10307:
URL: https://github.com/apache/ozone/pull/10307#discussion_r3271890089


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java:
##########
@@ -636,8 +637,10 @@ public BackgroundTaskResult call() {
         if (moveSucceeded) {
           // Add current old container to pendingDeletionContainers.
           pendingDeletionContainers.put(System.currentTimeMillis() + replicaDeletionDelay, container);
-          ContainerLogger.logMoveSuccess(containerId, sourceVolume,
-              destVolume, containerSize, Time.monotonicNow() - startTime);
+          if (newContainer != null) {
+            ContainerLogger.logMoveSuccess(newContainer.getContainerData(), sourceVolume,
+                destVolume, containerSize, Time.monotonicNow() - startTime);
+          }

Review Comment:
   no need of throwing runtime exception for background task. And if moveSucceeded is false then there is already skips pending-deletion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15311. [DiskBalancer] Fix DiskBalancer DN container Log Format [ozone]

Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #10307:
URL: https://github.com/apache/ozone/pull/10307#discussion_r3271882059


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerLogger.java:
##########
@@ -181,40 +181,38 @@ public static void logReconciled(ContainerData containerData, long oldDataChecks
   /**
    * Logged when a container is successfully moved from one data volume to another.
    *
-   * @param containerId The ID of the moved container.
+   * @param containerData The container after it has been moved to the destination volume.
    * @param sourceVolume The source volume path.
    * @param destinationVolume The destination volume path.
    * @param containerSize The size of data moved from container in bytes.
    * @param timeTaken The time taken for the move in milliseconds.
    */
-  public static void logMoveSuccess(long containerId, StorageVolume sourceVolume,
+  public static void logMoveSuccess(ContainerData containerData, StorageVolume sourceVolume,
       StorageVolume destinationVolume, long containerSize, long timeTaken) {
-    LOG.info(getMessage(containerId, sourceVolume, destinationVolume, containerSize, timeTaken));
+    LOG.info(getMessage(containerData,
+        "SrcVolume=" + sourceVolume,
+        "DestVolume=" + destinationVolume,
+        "Size=" + containerSize + " bytes",
+        "TimeTaken=" + timeTaken + " ms",
+        "Container is moved from SrcVolume to DestVolume"));
   }

Review Comment:
   not needed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-15311. [DiskBalancer] Fix DiskBalancer DN container Log Format [ozone]

Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on PR #10307:
URL: https://github.com/apache/ozone/pull/10307#issuecomment-4497663041

   @ChenSammi Please re-review, I have addressed the comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org