You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "waitinfuture (via GitHub)" <gi...@apache.org> on 2023/07/10 07:20:13 UTC
[GitHub] [incubator-celeborn] waitinfuture opened a new pull request, #1697: [CELEBORN-782] Make max components configurable for FileWriter#flushB…
waitinfuture opened a new pull request, #1697:
URL: https://github.com/apache/incubator-celeborn/pull/1697
…uffer
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Make max components configurable for FileWriter#flushBuffer.
### Why are the changes needed?
When max components of ```CompositeByteBuf``` is too big (hard coded 256 before this PR), netty's offheap memory
usage will be several times bigger than true usage:
```
Direct memory usage: 1044.0 MiB/4.0 GiB, disk buffer size: 255.9 MiB
```
But when it set too low, for example 1, performance might decrease:
```
34s vs. 30s
```
This PR makes the parameter configurable.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Passes GA, and manual test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1697: [CELEBORN-782] Make max components configurable for FileWriter#flushB…
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1697:
URL: https://github.com/apache/incubator-celeborn/pull/1697#discussion_r1257835286
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2579,6 +2580,15 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(false)
+ val WORKER_PUSH_COMPOSITEBUFFER_MAXCOMPONENTS: ConfigEntry[Int] =
+ buildConf("celeborn.worker.push.compositeBuffer.maxComponents")
+ .internal
+ .categories("worker")
+ .version("0.3.0")
+ .doc("Max components in CompositeByteBuf for FileWriter's flushBuffer.")
Review Comment:
better to explain a bit more about this configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1697: [CELEBORN-782] Make max components configurable for FileWriter#flushBuffer
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1697:
URL: https://github.com/apache/incubator-celeborn/pull/1697#discussion_r1257850575
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2579,6 +2580,15 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(false)
+ val WORKER_PUSH_COMPOSITEBUFFER_MAXCOMPONENTS: ConfigEntry[Int] =
+ buildConf("celeborn.worker.push.compositeBuffer.maxComponents")
+ .internal
+ .categories("worker")
+ .version("0.3.0")
+ .doc("Max components in CompositeByteBuf for FileWriter's flushBuffer.")
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture closed pull request #1697: [CELEBORN-782] Make max components configurable for FileWriter#flushBuffer
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture closed pull request #1697: [CELEBORN-782] Make max components configurable for FileWriter#flushBuffer
URL: https://github.com/apache/incubator-celeborn/pull/1697
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1697: [CELEBORN-782] Make max components configurable for FileWriter#flushBuffer
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1697:
URL: https://github.com/apache/incubator-celeborn/pull/1697#discussion_r1257865015
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2579,6 +2580,18 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(false)
+ val WORKER_PUSH_COMPOSITEBUFFER_MAXCOMPONENTS: ConfigEntry[Int] =
+ buildConf("celeborn.worker.push.compositeBuffer.maxComponents")
+ .internal
+ .categories("worker")
+ .version("0.3.0")
+ .doc("Max components in FileWriter's flushBuffer. When this value is too big, i.e. 16, " +
+ "there will be many memory fragments in netty's memory pool, and total direct memory can by several " +
+ "times larger than disk buffer. When set to 1, netty's direct memory is close to disk buffer, but performance" +
+ "might decrease.")
Review Comment:
```suggestion
.doc("Max components of Netty's `CompositeByteBuf` in `FileWriter`'s `flushBuffer`. " +
"When this value is too big, i.e. 256, there will be many memory fragments in Netty's memory pool, "
"and total direct memory can be significantly larger than the disk buffer. " +
"When set to 1, Netty's direct memory is close to disk buffer, but performance " +
"might decrease due to frequent memory copy during compaction.")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on pull request #1697: [CELEBORN-782] Make max components configurable for FileWriter#flushB…
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on PR #1697:
URL: https://github.com/apache/incubator-celeborn/pull/1697#issuecomment-1628389536
> The memory overhead comes from the unused spaces of each component, and compaction can reduce those spaces?
Right. Without compaction, there could be many memory fragments. For small offheap setup, this could cause unnecessary trim.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1697: [CELEBORN-782] Make max components configurable for FileWriter#flushB…
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1697:
URL: https://github.com/apache/incubator-celeborn/pull/1697#issuecomment-1628386648
The memory overhead comes from the unused spaces of each component, and compaction can reduce those spaces?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on pull request #1697: [CELEBORN-782] Make max components configurable for FileWriter#flushB…
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on PR #1697:
URL: https://github.com/apache/incubator-celeborn/pull/1697#issuecomment-1628382328
cc @FMX @AngersZhuuuu @pan3793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1697: [CELEBORN-782] Make max components configurable for FileWriter#flushBuffer
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1697:
URL: https://github.com/apache/incubator-celeborn/pull/1697#discussion_r1257865015
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2579,6 +2580,18 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(false)
+ val WORKER_PUSH_COMPOSITEBUFFER_MAXCOMPONENTS: ConfigEntry[Int] =
+ buildConf("celeborn.worker.push.compositeBuffer.maxComponents")
+ .internal
+ .categories("worker")
+ .version("0.3.0")
+ .doc("Max components in FileWriter's flushBuffer. When this value is too big, i.e. 16, " +
+ "there will be many memory fragments in netty's memory pool, and total direct memory can by several " +
+ "times larger than disk buffer. When set to 1, netty's direct memory is close to disk buffer, but performance" +
+ "might decrease.")
Review Comment:
```suggestion
.doc("Max components of Netty `CompositeByteBuf` in `FileWriter`'s `flushBuffer`. " +
"When this value is too big, i.e. 256, there will be many memory fragments in Netty's memory pool, "
"and total direct memory can be significantly larger than the disk buffer. " +
"When set to 1, Netty's direct memory is close to disk buffer, but performance " +
"might decrease due to frequent memory copy during compaction.")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org