[GitHub] [ozone] JacksonYao287 commented on a diff in pull request #3024: HDDS-6162. Limit OM pending request size to avoid taking too much memory

[GitBox <gi...@apache.org>]

JacksonYao287 commented on code in PR #3024:
URL: https://github.com/apache/ozone/pull/3024#discussion_r845969457


##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java:
##########
@@ -292,4 +292,8 @@ private OMConfigKeys() {
   public static final long OZONE_OM_ADMIN_PROTOCOL_WAIT_BETWEEN_RETRIES_DEFAULT
       = 1000;
 
+  public static final String OZONE_OM_MAX_PENDING_REQ_COUNT =
+      "ozone.om.max.pending.req.count";
+  public static final int OZONE_OM_MAX_PENDING_REQ_COUNT_DEFAULT = 10000;

Review Comment:
   i am sorry i can not find those dump file for now, i forget where i put them. but  it is not hard to reproduce this situation. when benchmarking , we started 30 freon instances (in different machine), each will start 10 clients thread to write object.  
   after i add the patch to our production cluster,  it works well under a heavy current pressure now, and the performance keeps the same as before.
   
   double buffer has two unlimited buffer: current buffer and ready buffer . in fact ,  if we do not limit the buffer size, several problems comes。
   1 a big ready buffer will produce a big rocksdb write batch.  for rocksdb , if a batch is too big , the write performance will downgrade。meanwhile ，a big write batch will also take much space.
   2 if rocksdb spend too much time on commiting a big write batch, then more and more request will be put in the current buffer of DoubleBuffer. a big current buffer will also take much space and  it will change to a big ready buffer and then a big write batch. It will be a vicious circle。what is more, if a full GC happens, The situation will get worse
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org