You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "exmy (via GitHub)" <gi...@apache.org> on 2023/11/28 09:52:12 UTC

[PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

exmy opened a new pull request, #2118:
URL: https://github.com/apache/incubator-celeborn/pull/2118

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     - Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
     - Be sure to keep the PR description updated to reflect all changes.
     - Please write your PR title to summarize what this PR proposes.
     - If possible, provide a concise example to reproduce the issue for a faster review.
   -->
   
   ### What changes were proposed in this pull request?
   The `clientPushBufferMaxSize` config is also used by `CelebornInputStreamImpl`, it's a config about push side and should not be used by fetch side. This pr introduces a fetch config to replace it.
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   yes, a new config `celeborn.client.fetch.buffer.max.size` is introduced.
   
   ### How was this patch tested?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

Posted by "exmy (via GitHub)" <gi...@apache.org>.
exmy commented on code in PR #2118:
URL: https://github.com/apache/incubator-celeborn/pull/2118#discussion_r1407504103


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -3213,6 +3214,15 @@ object CelebornConf extends Logging {
       .timeConf(TimeUnit.MILLISECONDS)
       .createWithDefaultString("600s")
 
+  val CLIENT_FETCH_BUFFER_MAX_SIZE: ConfigEntry[Long] =
+    buildConf("celeborn.client.fetch.buffer.max.size")
+      .withAlternative("celeborn.fetch.buffer.max.size")

Review Comment:
   Copied from `celeborn.client.push.buffer.max.size`, is it necessary?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2118:
URL: https://github.com/apache/incubator-celeborn/pull/2118#issuecomment-1829756736

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2118?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: `1 lines` in your changes are missing coverage. Please review.
   > Comparison is base [(`6623309`)](https://app.codecov.io/gh/apache/incubator-celeborn/commit/662330923716d2b5fe70c960f04853063a9d737c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 46.80% compared to head [(`cfaf246`)](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2118?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 46.79%.
   
   | [Files](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2118?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...cala/org/apache/celeborn/common/CelebornConf.scala](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2118?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY29tbW9uL0NlbGVib3JuQ29uZi5zY2FsYQ==) | 87.50% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2118?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main    #2118      +/-   ##
   ==========================================
   - Coverage   46.80%   46.79%   -0.00%     
   ==========================================
     Files         173      173              
     Lines       10929    10937       +8     
     Branches     1009     1009              
   ==========================================
   + Hits         5114     5117       +3     
   - Misses       5477     5481       +4     
   - Partials      338      339       +1     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2118?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

Posted by "exmy (via GitHub)" <gi...@apache.org>.
exmy commented on code in PR #2118:
URL: https://github.com/apache/incubator-celeborn/pull/2118#discussion_r1408859784


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -3213,6 +3214,14 @@ object CelebornConf extends Logging {
       .timeConf(TimeUnit.MILLISECONDS)
       .createWithDefaultString("600s")
 
+  val CLIENT_FETCH_BUFFER_MAX_SIZE: ConfigEntry[Long] =
+    buildConf("celeborn.client.fetch.buffer.max.size")
+      .categories("client")
+      .version("0.4.0")
+      .doc("Max size of reducer partition buffer memory for shuffle reader.")

Review Comment:
   Updated. Do you have any suggestions for improving this config description?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

Posted by "exmy (via GitHub)" <gi...@apache.org>.
exmy commented on code in PR #2118:
URL: https://github.com/apache/incubator-celeborn/pull/2118#discussion_r1407544870


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -3213,6 +3214,15 @@ object CelebornConf extends Logging {
       .timeConf(TimeUnit.MILLISECONDS)
       .createWithDefaultString("600s")
 
+  val CLIENT_FETCH_BUFFER_MAX_SIZE: ConfigEntry[Long] =
+    buildConf("celeborn.client.fetch.buffer.max.size")
+      .withAlternative("celeborn.fetch.buffer.max.size")

Review Comment:
   Got it, thanks for the reminder.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2118:
URL: https://github.com/apache/incubator-celeborn/pull/2118#discussion_r1407516241


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -3213,6 +3214,15 @@ object CelebornConf extends Logging {
       .timeConf(TimeUnit.MILLISECONDS)
       .createWithDefaultString("600s")
 
+  val CLIENT_FETCH_BUFFER_MAX_SIZE: ConfigEntry[Long] =
+    buildConf("celeborn.client.fetch.buffer.max.size")
+      .withAlternative("celeborn.fetch.buffer.max.size")

Review Comment:
   understand what the code does before copying. it is for backward compatibility so that alternatives will also work, which is obviously not required here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #2118:
URL: https://github.com/apache/incubator-celeborn/pull/2118#issuecomment-1829470321

   @exmy thanks for your contribution, can you restore the PR description template and fill it in?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #2118:
URL: https://github.com/apache/incubator-celeborn/pull/2118#issuecomment-1833529252

   Thanks, merged to main for 0.4


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2118:
URL: https://github.com/apache/incubator-celeborn/pull/2118#discussion_r1407776642


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -3213,6 +3214,14 @@ object CelebornConf extends Logging {
       .timeConf(TimeUnit.MILLISECONDS)
       .createWithDefaultString("600s")
 
+  val CLIENT_FETCH_BUFFER_MAX_SIZE: ConfigEntry[Long] =
+    buildConf("celeborn.client.fetch.buffer.max.size")
+      .categories("client")
+      .version("0.4.0")
+      .doc("Max size of reducer partition buffer memory for shuffle reader.")

Review Comment:
   Could you add more detail for this config like `celeborn.client.push.buffer.max.size` to tell user how to configure the reasonable config value?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #2118: [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl
URL: https://github.com/apache/incubator-celeborn/pull/2118


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2118:
URL: https://github.com/apache/incubator-celeborn/pull/2118#discussion_r1407499266


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -3213,6 +3214,15 @@ object CelebornConf extends Logging {
       .timeConf(TimeUnit.MILLISECONDS)
       .createWithDefaultString("600s")
 
+  val CLIENT_FETCH_BUFFER_MAX_SIZE: ConfigEntry[Long] =
+    buildConf("celeborn.client.fetch.buffer.max.size")
+      .withAlternative("celeborn.fetch.buffer.max.size")

Review Comment:
   why add this alternative?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1145] Separate clientPushBufferMaxSize from CelebornInputStreamImpl [incubator-celeborn]

Posted by "exmy (via GitHub)" <gi...@apache.org>.
exmy commented on PR #2118:
URL: https://github.com/apache/incubator-celeborn/pull/2118#issuecomment-1829478481

   > @exmy thanks for your contribution, can you restore the PR description template and fill it in?
   
   done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org