You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2024/03/26 06:43:47 UTC
[PR] [#1579][part-2] feat(spark): allow to register shuffle in parallel [incubator-uniffle]
zuston opened a new pull request, #1604:
URL: https://github.com/apache/incubator-uniffle/pull/1604
### What changes were proposed in this pull request?
allow to register shuffle in parallel
### Why are the changes needed?
this is the subtask for #1579
After the #1584, the registerShuffle operation may be slow compared with the previous operation due to the previous data purge. Based on this, it's necessary to register shuffle in parallel.
### Does this PR introduce _any_ user-facing change?
Yes. `rss.client.registerShuffle.maxParallelism=5` will be added.
### How was this patch tested?
Existing tests are enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1579][part-2] feat(spark): allow to register shuffle in parallel [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston closed pull request #1604: [#1579][part-2] feat(spark): allow to register shuffle in parallel
URL: https://github.com/apache/incubator-uniffle/pull/1604
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1579][part-2] feat(spark): allow to register shuffle in parallel [incubator-uniffle]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1604:
URL: https://github.com/apache/incubator-uniffle/pull/1604#issuecomment-2019517413
## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1604?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 54.08%. Comparing base [(`3a1b4d2`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/3a1b4d23e58d772f57733179025f53e819671344?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`4d4dc7a`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1604?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #1604 +/- ##
============================================
+ Coverage 54.02% 54.08% +0.05%
+ Complexity 2863 2674 -189
============================================
Files 438 398 -40
Lines 24858 21003 -3855
Branches 2114 1984 -130
============================================
- Hits 13430 11359 -2071
+ Misses 10587 8929 -1658
+ Partials 841 715 -126
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1604?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1579][part-2] feat(spark): allow to register shuffle in parallel [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1604:
URL: https://github.com/apache/incubator-uniffle/pull/1604#discussion_r1538895586
##########
common/src/main/java/org/apache/uniffle/common/config/RssClientConf.java:
##########
@@ -209,4 +209,10 @@ public class RssClientConf {
.defaultValue(false)
.withDescription(
"Whether to support task write failed retry internal, default value is false.");
+
+ public static final ConfigOption<Integer> RSS_CLIENT_REGISTER_SHUFFLE_MAX_PARALLELISM =
+ ConfigOptions.key("rss.client.registerShuffle.maxParallelism")
Review Comment:
+1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1579][part-2] feat(spark): allow to register shuffle in parallel [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1604:
URL: https://github.com/apache/incubator-uniffle/pull/1604#issuecomment-2019527797
## Test Results
2 109 files - 231 2 109 suites - 231 3h 18m 17s :stopwatch: - 1h 13m 51s
893 tests - 15 892 :white_check_mark: - 15 1 :zzz: ±0 0 :x: ±0
10 009 runs - 532 9 995 :white_check_mark: - 532 14 :zzz: ±0 0 :x: ±0
Results for commit 4d4dc7a5. ± Comparison against base commit 3a1b4d23.
<details>
<summary>This pull request <b>removes</b> 15 tests.</summary>
```
org.apache.spark.shuffle.RssShuffleManagerTest ‑ testCreateShuffleManagerServer
org.apache.spark.shuffle.RssShuffleManagerTest ‑ testGetDataDistributionType
org.apache.spark.shuffle.RssShuffleManagerTest ‑ testRssShuffleManagerInterface
org.apache.spark.shuffle.RssShuffleManagerTest ‑ testRssShuffleManagerRegisterShuffle{int}[1]
org.apache.spark.shuffle.RssShuffleManagerTest ‑ testRssShuffleManagerRegisterShuffle{int}[2]
org.apache.spark.shuffle.RssShuffleManagerTest ‑ testRssShuffleManagerRegisterShuffle{int}[3]
org.apache.spark.shuffle.RssSpark3ShuffleUtilsTest ‑ testCreateFetchFailedException
org.apache.spark.shuffle.RssSpark3ShuffleUtilsTest ‑ testIsStageResubmitSupported
org.apache.spark.shuffle.SparkVersionUtilsTest ‑ testSpark3Version
org.apache.spark.shuffle.writer.RssShuffleWriterTest ‑ dataConsistencyWhenSpillTriggeredTest
…
```
</details>
[test-results]:data:application/gzip;base64,H4sIAOZyAmYC/03NTQ6DIBCG4asY1l0ADur0Mg1lICH1p0FYNb17wShl+T3Ay4c5P9ud3TspON46ticf200p6Oi3NYsQE46Z8nksNybsr/XYkzEHyT+9/Lu8quC0nzPwCjaELZwS0lqagvPj2zKvKCKqSmcULmiix26bZlsWH/NgQEBm1Iq4pF7Bc6IBnDOi12QJAWnQCpSV7PsDRdsYRBEBAAA=
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1579][part-2] feat(spark): allow to register shuffle in parallel [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1604:
URL: https://github.com/apache/incubator-uniffle/pull/1604#issuecomment-2020111791
Parallelism here is fine independently of #1584. Please see https://github.com/apache/incubator-uniffle/pull/1584#issuecomment-2020103123 and https://github.com/apache/incubator-uniffle/pull/1584#issuecomment-2020110918.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1579][part-2] feat(spark): allow to register shuffle in parallel [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1604:
URL: https://github.com/apache/incubator-uniffle/pull/1604#discussion_r1538755426
##########
common/src/main/java/org/apache/uniffle/common/config/RssClientConf.java:
##########
@@ -209,4 +209,10 @@ public class RssClientConf {
.defaultValue(false)
.withDescription(
"Whether to support task write failed retry internal, default value is false.");
+
+ public static final ConfigOption<Integer> RSS_CLIENT_REGISTER_SHUFFLE_MAX_PARALLELISM =
+ ConfigOptions.key("rss.client.registerShuffle.maxParallelism")
Review Comment:
Our config options aren't good enough. Some config options have bad name. Maybe we should refactor them.
I thinks we should use `rss.client.group.camelStyleConfigOption`. I prefer using `rss.client.rpc.maxRegisterParallelism`. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org