[GitHub] [ozone] hanishakoneru commented on a diff in pull request #3024: HDDS-6162. Limit OM pending request size to avoid taking too much memory

[GitBox <gi...@apache.org>]

hanishakoneru commented on code in PR #3024:
URL: https://github.com/apache/ozone/pull/3024#discussion_r841986473


##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java:
##########
@@ -292,4 +292,8 @@ private OMConfigKeys() {
   public static final long OZONE_OM_ADMIN_PROTOCOL_WAIT_BETWEEN_RETRIES_DEFAULT
       = 1000;
 
+  public static final String OZONE_OM_MAX_PENDING_REQ_COUNT =
+      "ozone.om.max.pending.req.count";
+  public static final int OZONE_OM_MAX_PENDING_REQ_COUNT_DEFAULT = 10000;

Review Comment:
   > in fact , the object cached in the current queue and ready queue of ozoneDoubleBuffer is DoubleBufferEntry and it has a fixed size 24 bytes.
   
   DoubleBufferEntry has the transaction log index and the client response. The client response size would change based on the request type. For example, KeyCreate Request would have a bigger response size than 24 bytes while SetBucketProperty would have a lesser size. But the size should not be too big irrespective of this.
   
   > so , 10000 DoubleBufferEntry will take about 256MB memory, it is small enough for almost all the physical machine.
   Taking 24 bytes, 10,000 DoubleBufferEntries would take by only 256KB of memory. 
   
   Even if we take the average DoubleBufferEntry size as 1KB, 200000 entries should not take up around 200MB of space. 
   So I am not sure why the DoubleBuffer is taking up so much space.
   Could you please share your debug analysis to help me understand why DoubleBuffer is taking up too much memory?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org