[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1720: [CELEBORN-799] Limit total inflight push requests

["pan3793 (via GitHub)" <gi...@apache.org>]

pan3793 commented on code in PR #1720:
URL: https://github.com/apache/incubator-celeborn/pull/1720#discussion_r1264639923


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2702,17 +2703,28 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(512)
 
-  val CLIENT_PUSH_MAX_REQS_IN_FLIGHT: ConfigEntry[Int] =
-    buildConf("celeborn.client.push.maxReqsInFlight")
+  val CLIENT_PUSH_MAX_REQS_IN_FLIGHT_TOTAL: ConfigEntry[Int] =
+    buildConf("celeborn.client.push.maxReqsInFlight.total")
       .withAlternative("celeborn.push.maxReqsInFlight")
       .categories("client")
       .version("0.3.0")
-      .doc("Amount of Netty in-flight requests per worker. The maximum memory is " +
-        "`celeborn.client.push.maxReqsInFlight` * `celeborn.push.buffer.max.size` * " +
-        "number of workers * compression ratio(1 in worst case), say we have 50 workers," +
-        "the maximum memory is: 64KiB * 16 * 50 = 50MiB")
+      .doc("Amount of total Netty in-flight requests. The maximum memory is " +
+        "`celeborn.client.push.maxReqsInFlight.total` * `celeborn.push.buffer.max.size` " +
+        "* compression ratio(1 in worst case): 64KiB * 256 = 32MiB")
       .intConf
-      .createWithDefault(16)
+      .createWithDefault(256)
+
+  val CLIENT_PUSH_MAX_REQS_IN_FLIGHT_PERWORKER: ConfigEntry[Int] =
+    buildConf("celeborn.client.push.maxReqsInFlight.perWorker")
+      .categories("client")
+      .version("0.3.0")

Review Comment:
   are you going to vote -1 for 0.3.0 rc1?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org