You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "lifeSo (via GitHub)" <gi...@apache.org> on 2023/12/27 07:48:10 UTC
[PR] [#1024] improvement [tez]：Optimize user switch to shuffle mode local/remote. [incubator-uniffle]
lifeSo opened a new pull request, #1397:
URL: https://github.com/apache/incubator-uniffle/pull/1397
### What changes were proposed in this pull request?
Currently, users need to set the parameter tez.am.launch.launch.cmd-opts to switch between local shuffle and remote shuffle, but this parameter is difficult for users to understand.
We need a simpler and easier to understand parameter，such as tez.shuffle.mode = local/remote
### Why are the changes needed?
> [Improvement] [tez]：Optimize user switch to shuffle mode local/remote. #1024
Fix: https://github.com/apache/incubator-uniffle/issues/1024
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
unit test or cluster prd test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1024] improvement [tez]：Optimize user switch to shuffle mode local/remote. [incubator-uniffle]
Posted by "lifeSo (via GitHub)" <gi...@apache.org>.
lifeSo commented on PR #1397:
URL: https://github.com/apache/incubator-uniffle/pull/1397#issuecomment-1870041909
@baitian77
is this PR satifify your requirement ?
Task use RSS default,
and we can set "tez.shuffle.mode=false" to fail back to local mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1024] improvement(tez): Optimize user switch to shuffle mode local/remote. [incubator-uniffle]
Posted by "lifeSo (via GitHub)" <gi...@apache.org>.
lifeSo commented on code in PR #1397:
URL: https://github.com/apache/incubator-uniffle/pull/1397#discussion_r1439894435
##########
client-tez/src/main/java/org/apache/tez/dag/app/RssDAGAppMaster.java:
##########
@@ -342,6 +342,20 @@ public static void main(String[] args) {
// reconfigure.
reconfigureLog4j();
+ Configuration conf = new Configuration(new YarnConfiguration());
Review Comment:
If move the code to Line 329, then will be no am task log due to no log class and "log4j.configuration" config.
eg:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1024] improvement [tez]：Optimize user switch to shuffle mode local/remote. [incubator-uniffle]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1397:
URL: https://github.com/apache/incubator-uniffle/pull/1397#issuecomment-1870047441
## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1397?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`249c737`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/249c7378474e0fa642c35b8fd2dc53e98ca47a80?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 53.23% compared to head [(`3e1ca1b`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1397?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 54.17%.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #1397 +/- ##
============================================
+ Coverage 53.23% 54.17% +0.94%
+ Complexity 2718 2333 -385
============================================
Files 419 360 -59
Lines 23966 16697 -7269
Branches 2042 1521 -521
============================================
- Hits 12758 9046 -3712
+ Misses 10419 7150 -3269
+ Partials 789 501 -288
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1397?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1024] improvement(tez): Optimize user switch to shuffle mode local/remote. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #1397:
URL: https://github.com/apache/incubator-uniffle/pull/1397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1024] improvement(tez): Optimize user switch to shuffle mode local/remote. [incubator-uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #1397:
URL: https://github.com/apache/incubator-uniffle/pull/1397#discussion_r1439122262
##########
client-tez/src/main/java/org/apache/tez/dag/app/RssDAGAppMaster.java:
##########
@@ -342,6 +342,20 @@ public static void main(String[] args) {
// reconfigure.
reconfigureLog4j();
+ Configuration conf = new Configuration(new YarnConfiguration());
Review Comment:
Can we move this loop to Line 329-339?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1024] improvement(tez): Optimize user switch to shuffle mode local/remote. [incubator-uniffle]
Posted by "lifeSo (via GitHub)" <gi...@apache.org>.
lifeSo commented on code in PR #1397:
URL: https://github.com/apache/incubator-uniffle/pull/1397#discussion_r1439893334
##########
client-tez/src/main/java/org/apache/tez/dag/app/RssDAGAppMaster.java:
##########
@@ -342,6 +342,20 @@ public static void main(String[] args) {
// reconfigure.
reconfigureLog4j();
+ Configuration conf = new Configuration(new YarnConfiguration());
Review Comment:
No, the code before is init log class and log config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1024] improvement(tez): Optimize user switch to shuffle mode local/remote. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1397:
URL: https://github.com/apache/incubator-uniffle/pull/1397#discussion_r1440188140
##########
docs/client_guide/tez_client_guide.md:
##########
@@ -45,6 +45,7 @@ Note that the RssDAGAppMaster will automatically disable slow start (i.e., `tez.
| tez.rss.client.max.buffer.size | 3k | The max buffer size in map side. Control the size of each segment(WrappedBuffer) in the buffer. |
| tez.rss.client.batch.trigger.num | 50 | The max batch of buffers to send data in map side. Affect the number of blocks sent to the server in each batch, and may affect rss_worker_used_buffer_size |
| tez.rss.client.send.thread.num | 5 | The thread pool size for the client to send data to the server. |
+| tez.shuffle.mode | remote | Use Remote Shuffle if the value is set to 'remote' or use default config value, or set 'local' to use local shuffle when needs fall boak. |
Review Comment:
`needs fall boak` -> `needs to fall back`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1024] improvement(tez): Optimize user switch to shuffle mode local/remote. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1397:
URL: https://github.com/apache/incubator-uniffle/pull/1397#issuecomment-1871691040
@zhengchenyu Could you help me review this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org