You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "aryangupta1998 (via GitHub)" <gi...@apache.org> on 2025/04/28 11:53:10 UTC

[PR] TransactionToDNeCommitMap too large causes GC to pause for a long time [ozone]

aryangupta1998 opened a new pull request, #8347:
URL: https://github.com/apache/ozone/pull/8347

   ## What changes were proposed in this pull request?
   
   When deleting the block status from the SCM record, there will be a `transactionToDNsCommitMap` structure record. We encountered this structure accumulating too much content and occupying too much memory, causing the SCM to have a long GC time of 327 seconds.
   
   ```
   GC log:
   
   2025-01-15 08:25:14,789 [JvmPauseMonitor0] ERROR org.apache.ratis.server.RaftServer: 127e9d82-790c-40c5-af90-050564a06a45: JVM pause detected 372.305s longer than the close-threshold 120s, shutting down ... 
   ```
   
   Before iterating the deleteBlocks table and sending a delete request to dn, if the content in the `transactionToDNsCommitMap` is found to be too large and exceeds a certain threshold, pause the iteration of the deleteBlocks table for a while.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-12087
   
   ## How was this patch tested?
   
   Tested Manually.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12087. TransactionToDNCommitMap too large causes GC to pause for a long time [ozone]

Posted by "aryangupta1998 (via GitHub)" <gi...@apache.org>.
aryangupta1998 commented on code in PR #8347:
URL: https://github.com/apache/ozone/pull/8347#discussion_r2063902056


##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java:
##########
@@ -74,6 +74,10 @@ public final class HddsConfigKeys {
   public static final String HDDS_CONTAINER_CLOSE_THRESHOLD =
       "hdds.container.close.threshold";
   public static final float HDDS_CONTAINER_CLOSE_THRESHOLD_DEFAULT = 0.9f;
+
+  public static final String HDDS_SCM_TXN_DN_COMMIT_MAP_SIZE = "hdds.scm.txn.dn.commit.map.size";
+
+  public static final int HDDS_SCM_TXN_DN_COMMIT_MAP_SIZE_DEFAULT = 5000000;

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12087. TransactionToDNCommitMap too large causes GC to pause for a long time [ozone]

Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 merged PR #8347:
URL: https://github.com/apache/ozone/pull/8347


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12087. TransactionToDNCommitMap too large causes GC to pause for a long time [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8347:
URL: https://github.com/apache/ozone/pull/8347#discussion_r2064076361


##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java:
##########
@@ -74,6 +74,10 @@ public final class HddsConfigKeys {
   public static final String HDDS_CONTAINER_CLOSE_THRESHOLD =
       "hdds.container.close.threshold";
   public static final float HDDS_CONTAINER_CLOSE_THRESHOLD_DEFAULT = 0.9f;
+
+  public static final String HDDS_SCM_TXN_DN_COMMIT_MAP_SIZE = "hdds.scm.txn.dn.commit.map.size";
+
+  public static final int HDDS_SCM_TXN_DN_COMMIT_MAP_SIZE_DEFAULT = 5000000;

Review Comment:
   Thanks for the improvement.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12087. TransactionToDNCommitMap too large causes GC to pause for a long time [ozone]

Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on PR #8347:
URL: https://github.com/apache/ozone/pull/8347#issuecomment-2861987706

   > @ashishkumar50, considering a txn is mapped to 3 DNs, Each entry in map will require about approx 200 bytes(24 + 80 + 32 X 3). So with 500k entries, 100 MB will be occupied, with 1 M entries, 200 MB, and with 5M entries, 1 GB would be required!
   
   Thanks for the clarification.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12087. TransactionToDNCommitMap too large causes GC to pause for a long time [ozone]

Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #8347:
URL: https://github.com/apache/ozone/pull/8347#issuecomment-2835842445

    @ptlrs can you please take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12087. TransactionToDNCommitMap too large causes GC to pause for a long time [ozone]

Posted by "aryangupta1998 (via GitHub)" <gi...@apache.org>.
aryangupta1998 commented on PR #8347:
URL: https://github.com/apache/ozone/pull/8347#issuecomment-2858047318

   @ashishkumar50, considering a txn is mapped to 3 DNs,
   Each entry in map will require about approx 200 bytes(24 + 80 + 32 X 3).
   So with 500k entries, 100 MB will be occupied, with 1 M entries, 200 MB, and with 5M entries, 1 GB would be required!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12087. TransactionToDNCommitMap too large causes GC to pause for a long time [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8347:
URL: https://github.com/apache/ozone/pull/8347#discussion_r2063615603


##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java:
##########
@@ -74,6 +74,10 @@ public final class HddsConfigKeys {
   public static final String HDDS_CONTAINER_CLOSE_THRESHOLD =
       "hdds.container.close.threshold";
   public static final float HDDS_CONTAINER_CLOSE_THRESHOLD_DEFAULT = 0.9f;
+
+  public static final String HDDS_SCM_TXN_DN_COMMIT_MAP_SIZE = "hdds.scm.txn.dn.commit.map.size";
+
+  public static final int HDDS_SCM_TXN_DN_COMMIT_MAP_SIZE_DEFAULT = 5000000;

Review Comment:
   Please add new property in `ScmConfig` instead, like:
   
   https://github.com/apache/ozone/blob/319d5a4cfed43cae2885946452145b4fc3e461e2/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfig.java#L105-L117
   
   Probably should include `block.deletion` in the name.  Also, it should indicate that this is a limit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org