You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2023/02/22 10:08:09 UTC
[GitHub] [incubator-uniffle] zuston opened a new pull request, #644: [#635] feat(client): enable LOCAL_ORDER by default for Spark AQE
zuston opened a new pull request, #644:
URL: https://github.com/apache/incubator-uniffle/pull/644
### What changes were proposed in this pull request?
enable LOCAL_ORDER by default for Spark AQE
### Why are the changes needed?
Currently, the local_order data distribution type should be activated explicitly. It's better to enable it when using AQE by default.
Fix: #635
### Does this PR introduce _any_ user-facing change?
Yes
### How was this patch tested?
1. Existing UTs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston commented on pull request #644: [#635] feat(client): enable LOCAL_ORDER by default for Spark AQE
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #644:
URL: https://github.com/apache/incubator-uniffle/pull/644#issuecomment-1439746418
PTAL @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #644: [#635] feat(client): enable LOCAL_ORDER by default for Spark AQE
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #644:
URL: https://github.com/apache/incubator-uniffle/pull/644#discussion_r1114204565
##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -220,6 +222,17 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
}
}
+ @VisibleForTesting
+ protected static ShuffleDataDistributionType getDataDistributionType(SparkConf sparkConf) {
+ RssConf rssConf = RssSparkConfig.toRssConf(sparkConf);
+ if ((boolean) sparkConf.get(SQLConf.ADAPTIVE_EXECUTION_ENABLED())
Review Comment:
`sparkConf.get(SQLConf.ADAPTIVE_EXECUTION_ENABLED()` should be used to be consistent with spark, because different spark versions have different default value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #644: [#635] feat(client): enable LOCAL_ORDER by default for Spark AQE
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #644:
URL: https://github.com/apache/incubator-uniffle/pull/644#discussion_r1114409591
##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -220,6 +222,17 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
}
}
+ @VisibleForTesting
+ protected static ShuffleDataDistributionType getDataDistributionType(SparkConf sparkConf) {
+ RssConf rssConf = RssSparkConfig.toRssConf(sparkConf);
+ if ((boolean) sparkConf.get(SQLConf.ADAPTIVE_EXECUTION_ENABLED())
Review Comment:
is `(boolean)` necessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #644: [#635] feat(client): enable LOCAL_ORDER by default for Spark AQE
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #644:
URL: https://github.com/apache/incubator-uniffle/pull/644#discussion_r1114203303
##########
common/src/main/java/org/apache/uniffle/common/config/RssClientConf.java:
##########
@@ -40,7 +40,7 @@ public class RssClientConf {
public static final ConfigOption<ShuffleDataDistributionType> DATA_DISTRIBUTION_TYPE = ConfigOptions
.key("rss.client.shuffle.data.distribution.type")
.enumType(ShuffleDataDistributionType.class)
- .defaultValue(ShuffleDataDistributionType.NORMAL)
+ .defaultValue(ShuffleDataDistributionType.LOCAL_ORDER)
Review Comment:
Make sense. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #644: [#635] feat(client): enable LOCAL_ORDER by default for Spark AQE
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #644:
URL: https://github.com/apache/incubator-uniffle/pull/644#discussion_r1114204762
##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -220,6 +222,17 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
}
}
+ @VisibleForTesting
+ protected static ShuffleDataDistributionType getDataDistributionType(SparkConf sparkConf) {
+ RssConf rssConf = RssSparkConfig.toRssConf(sparkConf);
+ if ((boolean) sparkConf.get(SQLConf.ADAPTIVE_EXECUTION_ENABLED())
Review Comment:
cc @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #644: [#635] feat(client): enable LOCAL_ORDER by default for Spark AQE
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #644:
URL: https://github.com/apache/incubator-uniffle/pull/644#discussion_r1114150815
##########
common/src/main/java/org/apache/uniffle/common/config/RssClientConf.java:
##########
@@ -40,7 +40,7 @@ public class RssClientConf {
public static final ConfigOption<ShuffleDataDistributionType> DATA_DISTRIBUTION_TYPE = ConfigOptions
.key("rss.client.shuffle.data.distribution.type")
.enumType(ShuffleDataDistributionType.class)
- .defaultValue(ShuffleDataDistributionType.NORMAL)
+ .defaultValue(ShuffleDataDistributionType.LOCAL_ORDER)
Review Comment:
We should use LOCAL_ORDER when we use Spark AQE. We shouldn't use it as default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi merged pull request #644: [#635] feat(client): enable LOCAL_ORDER by default for Spark AQE
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #644:
URL: https://github.com/apache/incubator-uniffle/pull/644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #644: [#635] feat(client): enable LOCAL_ORDER by default for Spark AQE
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #644:
URL: https://github.com/apache/incubator-uniffle/pull/644#issuecomment-1439759205
# [Codecov](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#644](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ede29e5) into [master](https://codecov.io/gh/apache/incubator-uniffle/commit/8b347a2afe94afa738182180d7ad4be627f9e8a4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8b347a2) will **increase** coverage by `2.28%`.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## master #644 +/- ##
============================================
+ Coverage 60.61% 62.90% +2.28%
- Complexity 1798 1799 +1
============================================
Files 216 202 -14
Lines 12462 10491 -1971
Branches 1052 1052
============================================
- Hits 7554 6599 -955
+ Misses 4500 3545 -955
+ Partials 408 347 -61
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/uniffle/common/config/RssClientConf.java](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL2NvbW1vbi9jb25maWcvUnNzQ2xpZW50Q29uZi5qYXZh) | `93.75% <100.00%> (ø)` | |
| [.../apache/uniffle/coordinator/ClientConfManager.java](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3IvQ2xpZW50Q29uZk1hbmFnZXIuamF2YQ==) | `91.54% <0.00%> (-1.41%)` | :arrow_down: |
| [deploy/kubernetes/operator/pkg/utils/rss.go](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3V0aWxzL3Jzcy5nbw==) | | |
| [...rnetes/operator/pkg/webhook/inspector/inspector.go](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3dlYmhvb2svaW5zcGVjdG9yL2luc3BlY3Rvci5nbw==) | | |
| [...y/kubernetes/operator/pkg/webhook/inspector/pod.go](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3dlYmhvb2svaW5zcGVjdG9yL3BvZC5nbw==) | | |
| [...oy/kubernetes/operator/pkg/utils/shufflerserver.go](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3V0aWxzL3NodWZmbGVyc2VydmVyLmdv) | | |
| [...bernetes/operator/pkg/controller/controller/rss.go](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL2NvbnRyb2xsZXIvY29udHJvbGxlci9yc3MuZ28=) | | |
| [...y/kubernetes/operator/pkg/webhook/inspector/rss.go](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3dlYmhvb2svaW5zcGVjdG9yL3Jzcy5nbw==) | | |
| [deploy/kubernetes/operator/pkg/utils/config.go](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3V0aWxzL2NvbmZpZy5nbw==) | | |
| [deploy/kubernetes/operator/pkg/utils/util.go](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3V0aWxzL3V0aWwuZ28=) | | |
| ... and [7 more](https://codecov.io/gh/apache/incubator-uniffle/pull/644?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #644: [#635] feat(client): enable LOCAL_ORDER by default for Spark AQE
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #644:
URL: https://github.com/apache/incubator-uniffle/pull/644#issuecomment-1441122074
thanks @zuston , merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org