You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "rickyma (via GitHub)" <gi...@apache.org> on 2024/07/12 04:03:47 UTC
[PR] [#1892] improvement(docs): Provide guidance on configuring memory-related configs [incubator-uniffle]
rickyma opened a new pull request, #1893:
URL: https://github.com/apache/incubator-uniffle/pull/1893
### What changes were proposed in this pull request?
Provide guidance on configuring memory-related configs.
### Why are the changes needed?
For https://github.com/apache/incubator-uniffle/issues/1892.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1892] improvement(docs): Provide guidance on configuring memory-related configs [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma merged PR #1893:
URL: https://github.com/apache/incubator-uniffle/pull/1893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1892] improvement(docs): Provide guidance on configuring memory-related configs [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1893:
URL: https://github.com/apache/incubator-uniffle/pull/1893#discussion_r1675524656
##########
docs/server_guide.md:
##########
@@ -145,20 +145,46 @@ Finally, to improve the speed of writing to HDFS for a single partition, the val
### Netty
In version 0.8.0, we introduced Netty. Enabling Netty on ShuffleServer can significantly reduce GC time in high-throughput scenarios. We can enable Netty through the parameters `rss.server.netty.port` and `rss.rpc.server.type`. Note: After setting the parameter `rss.rpc.server.type` to `GRPC_NETTY`, ShuffleServer will be tagged with `GRPC_NETTY`, that is, the node can only be assigned to clients with `spark.rss.client.type=GRPC_NETTY`.
-When enabling Netty, we should also consider memory related configurations, the following is an example.
+When enabling Netty, we should also consider memory related configurations.
+
+#### Memory Configuration Principles
+
+- Reserve about `15%` of the machine's memory space (reserved space for OS slab, reserved, cache, buffer, kernel stack, etc.)
+- Recommended ratio for heap and off-heap memory: `1:9`
+- `rss.server.buffer.capacity` + `rss.server.read.buffer.capacity` + reserved = maximum off-heap
+
+Note: The reserved memory can be adjusted according to the actual situation, if the memory is relatively small, configuring 1g is completely sufficient.
+
+##### rss-env.sh
+
+Assuming the machine has 470g of memory.
+The machine reserves 15% of memory space, about 70g, following the above principle (heap:off-heap=1:9):
-#### rss-env.sh
```
-XMX_SIZE=20g
-MAX_DIRECT_MEMORY_SIZE=120g
+heap = (470 - 70) * 1 / 10 = 40g
+off-heap = (470 - 70) * 9 / 10 = 360g
+heap + off-heap = 400g
```
-#### server.conf
+
+So, `rss-env.sh` will be:
+
+```
+XMX_SIZE=40g
+MAX_DIRECT_MEMORY_SIZE=360g
+```
+
+##### server.conf
+
+Generally, `rss.server.read.buffer.capacity` of 20g is enough, you can pay more attention to the metric `read_used_buffer_size`.
Review Comment:
What's the percentage of the memory?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1892] improvement(docs): Provide guidance on configuring memory-related configs [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on code in PR #1893:
URL: https://github.com/apache/incubator-uniffle/pull/1893#discussion_r1675563368
##########
docs/server_guide.md:
##########
@@ -145,20 +145,46 @@ Finally, to improve the speed of writing to HDFS for a single partition, the val
### Netty
In version 0.8.0, we introduced Netty. Enabling Netty on ShuffleServer can significantly reduce GC time in high-throughput scenarios. We can enable Netty through the parameters `rss.server.netty.port` and `rss.rpc.server.type`. Note: After setting the parameter `rss.rpc.server.type` to `GRPC_NETTY`, ShuffleServer will be tagged with `GRPC_NETTY`, that is, the node can only be assigned to clients with `spark.rss.client.type=GRPC_NETTY`.
-When enabling Netty, we should also consider memory related configurations, the following is an example.
+When enabling Netty, we should also consider memory related configurations.
+
+#### Memory Configuration Principles
+
+- Reserve about `15%` of the machine's memory space (reserved space for OS slab, reserved, cache, buffer, kernel stack, etc.)
+- Recommended ratio for heap and off-heap memory: `1:9`
+- `rss.server.buffer.capacity` + `rss.server.read.buffer.capacity` + reserved = maximum off-heap
+
+Note: The reserved memory can be adjusted according to the actual situation, if the memory is relatively small, configuring 1g is completely sufficient.
+
+##### rss-env.sh
+
+Assuming the machine has 470g of memory.
+The machine reserves 15% of memory space, about 70g, following the above principle (heap:off-heap=1:9):
-#### rss-env.sh
```
-XMX_SIZE=20g
-MAX_DIRECT_MEMORY_SIZE=120g
+heap = (470 - 70) * 1 / 10 = 40g
+off-heap = (470 - 70) * 9 / 10 = 360g
+heap + off-heap = 400g
```
-#### server.conf
+
+So, `rss-env.sh` will be:
+
+```
+XMX_SIZE=40g
+MAX_DIRECT_MEMORY_SIZE=360g
+```
+
+##### server.conf
+
+Generally, `rss.server.read.buffer.capacity` of 20g is enough, you can pay more attention to the metric `read_used_buffer_size`.
Review Comment:
We don't need that much read buffer. Currently, we recommend to set:
`rss.server.read.buffer.capacity` : `rss.server.buffer.capacity` = 1 : 18
which is approximately 20g : 330g in this example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1892] improvement(docs): Provide guidance on configuring memory-related configs [incubator-uniffle]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1893:
URL: https://github.com/apache/incubator-uniffle/pull/1893#issuecomment-2224719300
## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1893?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 53.40%. Comparing base [(`dddcced`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/dddcced0dd65431d3a987bfcaf0875998901306c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`dbb6bcd`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/dbb6bcd1d577f7bcd446dd3842532434af7861e8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 46 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #1893 +/- ##
============================================
- Coverage 53.53% 53.40% -0.14%
- Complexity 2356 2407 +51
============================================
Files 368 373 +5
Lines 16852 17257 +405
Branches 1540 1583 +43
============================================
+ Hits 9022 9216 +194
- Misses 7303 7497 +194
- Partials 527 544 +17
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1893?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1892] improvement(docs): Provide guidance on configuring memory-related configs [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on code in PR #1893:
URL: https://github.com/apache/incubator-uniffle/pull/1893#discussion_r1675621132
##########
docs/server_guide.md:
##########
@@ -145,20 +145,46 @@ Finally, to improve the speed of writing to HDFS for a single partition, the val
### Netty
In version 0.8.0, we introduced Netty. Enabling Netty on ShuffleServer can significantly reduce GC time in high-throughput scenarios. We can enable Netty through the parameters `rss.server.netty.port` and `rss.rpc.server.type`. Note: After setting the parameter `rss.rpc.server.type` to `GRPC_NETTY`, ShuffleServer will be tagged with `GRPC_NETTY`, that is, the node can only be assigned to clients with `spark.rss.client.type=GRPC_NETTY`.
-When enabling Netty, we should also consider memory related configurations, the following is an example.
+When enabling Netty, we should also consider memory related configurations.
+
+#### Memory Configuration Principles
+
+- Reserve about `15%` of the machine's memory space (reserved space for OS slab, reserved, cache, buffer, kernel stack, etc.)
+- Recommended ratio for heap and off-heap memory: `1:9`
+- `rss.server.buffer.capacity` + `rss.server.read.buffer.capacity` + reserved = maximum off-heap
+
+Note: The reserved memory can be adjusted according to the actual situation, if the memory is relatively small, configuring 1g is completely sufficient.
+
+##### rss-env.sh
+
+Assuming the machine has 470g of memory.
+The machine reserves 15% of memory space, about 70g, following the above principle (heap:off-heap=1:9):
-#### rss-env.sh
```
-XMX_SIZE=20g
-MAX_DIRECT_MEMORY_SIZE=120g
+heap = (470 - 70) * 1 / 10 = 40g
+off-heap = (470 - 70) * 9 / 10 = 360g
+heap + off-heap = 400g
```
-#### server.conf
+
+So, `rss-env.sh` will be:
+
+```
+XMX_SIZE=40g
+MAX_DIRECT_MEMORY_SIZE=360g
+```
+
+##### server.conf
+
+Generally, `rss.server.read.buffer.capacity` of 20g is enough, you can pay more attention to the metric `read_used_buffer_size`.
Review Comment:
I have added into the doc.
##########
docs/server_guide.md:
##########
@@ -145,20 +145,46 @@ Finally, to improve the speed of writing to HDFS for a single partition, the val
### Netty
In version 0.8.0, we introduced Netty. Enabling Netty on ShuffleServer can significantly reduce GC time in high-throughput scenarios. We can enable Netty through the parameters `rss.server.netty.port` and `rss.rpc.server.type`. Note: After setting the parameter `rss.rpc.server.type` to `GRPC_NETTY`, ShuffleServer will be tagged with `GRPC_NETTY`, that is, the node can only be assigned to clients with `spark.rss.client.type=GRPC_NETTY`.
-When enabling Netty, we should also consider memory related configurations, the following is an example.
+When enabling Netty, we should also consider memory related configurations.
+
+#### Memory Configuration Principles
+
+- Reserve about `15%` of the machine's memory space (reserved space for OS slab, reserved, cache, buffer, kernel stack, etc.)
+- Recommended ratio for heap and off-heap memory: `1:9`
+- `rss.server.buffer.capacity` + `rss.server.read.buffer.capacity` + reserved = maximum off-heap
+
+Note: The reserved memory can be adjusted according to the actual situation, if the memory is relatively small, configuring 1g is completely sufficient.
+
+##### rss-env.sh
+
+Assuming the machine has 470g of memory.
+The machine reserves 15% of memory space, about 70g, following the above principle (heap:off-heap=1:9):
-#### rss-env.sh
```
-XMX_SIZE=20g
-MAX_DIRECT_MEMORY_SIZE=120g
+heap = (470 - 70) * 1 / 10 = 40g
+off-heap = (470 - 70) * 9 / 10 = 360g
+heap + off-heap = 400g
```
-#### server.conf
+
+So, `rss-env.sh` will be:
+
+```
+XMX_SIZE=40g
+MAX_DIRECT_MEMORY_SIZE=360g
+```
+
+##### server.conf
+
+Generally, `rss.server.read.buffer.capacity` of 20g is enough, you can pay more attention to the metric `read_used_buffer_size`.
Review Comment:
I have added this into the doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1892] improvement(docs): Provide guidance on configuring memory-related configs [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1893:
URL: https://github.com/apache/incubator-uniffle/pull/1893#issuecomment-2225576529
Merged to master. Thanks for your reviews @jerqi @xianjingfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1892] improvement(docs): Provide guidance on configuring memory-related configs [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1893:
URL: https://github.com/apache/incubator-uniffle/pull/1893#issuecomment-2224535749
## Test Results
2 656 files - 1 2 656 suites - 1 5h 32m 2s :stopwatch: -39s
946 tests ±0 944 :white_check_mark: ±0 1 :zzz: ±0 1 :x: ±0
11 798 runs - 1 11 782 :white_check_mark: - 1 15 :zzz: ±0 1 :x: ±0
For more details on these failures, see [this check](https://github.com/apache/incubator-uniffle/runs/27356304603).
Results for commit efafd71a. ± Comparison against base commit 7a28f9fe.
[test-results]:data:application/gzip;base64,H4sIANGwkGYC/1XNyQ6DIBSF4VcxrLvgEpn6Mg0yJKQqDcOq6bv3atTS5fmAnzcJcfaF3AcmuLgNpLRY++1aNjWmFQW0ZgwJz+t2Q4/iXI/SrN1p/NEzvrZXFwQT5z/wOaeMQlFyW7cmgNTqmGcUTbHLjio/oavuu4/atCyx4iA+mOAkmAkmocArKimlo+LaSYZfBgjo1jPy+QIc8QCGEgEAAA==
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org