You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "SteNicholas (via GitHub)" <gi...@apache.org> on 2025/05/07 12:01:13 UTC

[PR] [CELEBORN-1993] CelebornConf introduces celeborn..io.threads to specify number of threads used in the server and client thread pool [celeborn]

SteNicholas opened a new pull request, #3245:
URL: https://github.com/apache/celeborn/pull/3245

   ### What changes were proposed in this pull request?
   
   `CelebornConf` introduces `celeborn.<module>.io.threads` to specify number of threads used in the server and client thread pool.
   
   ### Why are the changes needed?
   
   `ShuffleClientImpl` and `FlinkShuffleClientImpl` use fixed configuration expression as `conf.getInt("celeborn." + module + ".io.threads", 8)`. Therefore, `CelebornConf` should introduce `celeborn.<module>.io.threads` to specify number of threads used in the server and client thread pool.
   
   ### Does this PR introduce _any_ user-facing change?
   
   `CelebornConf` adds `celeborn.<module>.io.threads` config option.
   
   ### How was this patch tested?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1993] CelebornConf introduces celeborn..io.threads to specify number of threads used in the client thread pool [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3245:
URL: https://github.com/apache/celeborn/pull/3245#discussion_r2094545709


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2089,6 +2093,16 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(0)
 
+  val NETWORK_IO_THREADS: ConfigEntry[Int] =
+    buildConf("celeborn.<module>.io.threads")
+      .categories("network")
+      .doc(s"Default number of threads used in the server and client thread pool. " +
+        s"This specifies thread configuration based on JVM's allocation of cores. " +
+        s"If setting <module> to `${TransportModuleConstants.DATA_MODULE}`, " +
+        s"it works for shuffle client push and fetch data.")
+      .intConf
+      .createWithDefault(8)

Review Comment:
   seems behavior change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1993] CelebornConf introduces celeborn..io.threads to specify number of threads used in the client thread pool [celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3245:
URL: https://github.com/apache/celeborn/pull/3245#discussion_r2095039482


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2089,6 +2093,16 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(0)
 
+  val NETWORK_IO_THREADS: ConfigEntry[Int] =
+    buildConf("celeborn.<module>.io.threads")
+      .categories("network")
+      .doc(s"Default number of threads used in the server and client thread pool. " +
+        s"This specifies thread configuration based on JVM's allocation of cores. " +
+        s"If setting <module> to `${TransportModuleConstants.DATA_MODULE}`, " +
+        s"it works for shuffle client push and fetch data.")
+      .intConf
+      .createWithDefault(8)

Review Comment:
   @turboFei, there is no behavior change. Just add the config option in `CelebornConf` with the default value `8`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1993] CelebornConf introduces celeborn..io.threads to specify number of threads used in the client thread pool [celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3245:
URL: https://github.com/apache/celeborn/pull/3245#issuecomment-2876238910

   Ping @RexXiong. PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1993] CelebornConf introduces celeborn..io.threads to specify number of threads used in the client thread pool [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3245:
URL: https://github.com/apache/celeborn/pull/3245#discussion_r2094545709


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2089,6 +2093,16 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(0)
 
+  val NETWORK_IO_THREADS: ConfigEntry[Int] =
+    buildConf("celeborn.<module>.io.threads")
+      .categories("network")
+      .doc(s"Default number of threads used in the server and client thread pool. " +
+        s"This specifies thread configuration based on JVM's allocation of cores. " +
+        s"If setting <module> to `${TransportModuleConstants.DATA_MODULE}`, " +
+        s"it works for shuffle client push and fetch data.")
+      .intConf
+      .createWithDefault(8)

Review Comment:
   seems behavior change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1993] CelebornConf introduces celeborn..io.threads to specify number of threads used in the client thread pool [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX closed pull request #3245: [CELEBORN-1993] CelebornConf introduces celeborn.<module>.io.threads to specify number of threads used in the client thread pool
URL: https://github.com/apache/celeborn/pull/3245


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1993] CelebornConf introduces celeborn..io.threads to specify number of threads used in the server and client thread pool [celeborn]

Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #3245:
URL: https://github.com/apache/celeborn/pull/3245#discussion_r2083200633


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2093,6 +2097,15 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(0)
 
+  val NETWORK_IO_THREADS: ConfigEntry[Int] =
+    buildConf("celeborn.<module>.io.threads")

Review Comment:
   use `celeborn.<module>.io.clientThreads` together with `celeborn.<module>.io.serverThreads`?



##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2093,6 +2097,15 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(0)
 
+  val NETWORK_IO_THREADS: ConfigEntry[Int] =
+    buildConf("celeborn.<module>.io.threads")
+      .categories("network")
+      .doc("Number of threads used in the server and client thread pool. " +

Review Comment:
   Number of threads used in  client thread pool



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1993] CelebornConf introduces celeborn..io.threads to specify number of threads used in the server and client thread pool [celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3245:
URL: https://github.com/apache/celeborn/pull/3245#issuecomment-2866608906

   @FMX, @RexXiong.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org