You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "guixiaowen (via GitHub)" <gi...@apache.org> on 2025/09/29 02:54:29 UTC
[PR] [AURON #1372]Do not convert into ShuffleExchangeExec into Native when shuffle manger not set [auron]
guixiaowen opened a new pull request, #1373:
URL: https://github.com/apache/auron/pull/1373
<!--
Thanks for sending a pull request! Please keep the following tips in mind:
- Start the PR title with the related issue ID, e.g. '[AURON #XXXX] Short summary...'.
- Make your PR title clear and descriptive, summarizing what this PR changes.
- Provide a concise example to reproduce the issue, if possible.
- Keep the PR description up to date with all changes.
-->
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #1372
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
Check whether the shuffle manager is AuronShuffleManager, AuronUniffleShuffleManager, or AuronCelebornShuffleManager.
If it is one of these, convert to native; if not, do not perform the conversion.
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
# How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1372]Do not convert into ShuffleExchangeExec into Native when shuffle manger not set [auron]
Posted by "guixiaowen (via GitHub)" <gi...@apache.org>.
guixiaowen commented on code in PR #1373:
URL: https://github.com/apache/auron/pull/1373#discussion_r2387068685
##########
spark-extension-shims-spark3/src/test/scala/org/apache/spark/sql/auron/AuronCheckConvertShuffleExchangeSuite.scala:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.auron
+
+import org.apache.spark.sql.{QueryTest, Row, SparkSession}
+import org.apache.spark.sql.execution.auron.plan.NativeShuffleExchangeExec
+import org.apache.spark.sql.execution.exchange.ShuffleExchangeExec
+import org.apache.spark.sql.test.SharedSparkSession
+
+class AuronCheckConvertShuffleExchangeSuite
+ extends QueryTest
+ with SharedSparkSession
+ with AuronSQLTestHelper
+ with org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper {
+
+ test("test set auron shuffle manager convert to native shuffle exchange") {
+ withTable("test_shuffle") {
+ val spark = SparkSession
+ .builder()
+ .master("local[2]")
+ .appName("checkConvertToNativeShuffleManger")
+ .config("spark.sql.shuffle.partitions", "4") // 自定义 SQL 参数
Review Comment:
> meaningless comment
@richox I delete it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1372]add spark.auron.enable.shuffleexchange to control ShuffleExchangeExec into Native or not [auron]
Posted by "guixiaowen (via GitHub)" <gi...@apache.org>.
guixiaowen commented on code in PR #1373:
URL: https://github.com/apache/auron/pull/1373#discussion_r2394423129
##########
spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala:
##########
@@ -138,12 +138,21 @@ object AuronConverters extends Logging {
getBooleanConf("spark.auron.enable.scan.parquet", defaultValue = true)
def enableScanOrc: Boolean =
getBooleanConf("spark.auron.enable.scan.orc", defaultValue = true)
+ def enableShuffleExechange: Boolean =
+ getBooleanConf("spark.auron.enable.shuffleExchange", defaultValue = true)
private val extConvertProviders = ServiceLoader.load(classOf[AuronConvertProvider]).asScala
def extConvertSupported(exec: SparkPlan): Boolean = {
extConvertProviders.exists(_.isSupported(exec))
}
+ def enableExchange(): Boolean = {
+ val shuffleMangerName = SQLConf.get.getConfString(config.SHUFFLE_MANAGER.key)
+ enableShuffleExechange && !shuffleMangerName.isEmpty && (shuffleMangerName.contains(
+ "AuronShuffleManager") || shuffleMangerName.contains(
+ "AuronUniffleShuffleManager") || shuffleMangerName.contains("AuronCelebornShuffleManager"))
Review Comment:
> ```scala
> AuronCelebornShuffleManager
> ```
@ShreyeshArangath she Yes, I think you’re right — this part is indeed hard-coded. The main reason is that Celeborn or Uniffle may not be introduced directly here, so the class AuronCelebornShuffleManager may not necessarily be available to reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1372]add spark.auron.enable.shuffleexchange to control ShuffleExchangeExec into Native or not [auron]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #1373:
URL: https://github.com/apache/auron/pull/1373#discussion_r2389774863
##########
spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala:
##########
@@ -138,12 +138,21 @@ object AuronConverters extends Logging {
getBooleanConf("spark.auron.enable.scan.parquet", defaultValue = true)
def enableScanOrc: Boolean =
getBooleanConf("spark.auron.enable.scan.orc", defaultValue = true)
+ def enableShuffleExechange: Boolean =
+ getBooleanConf("spark.auron.enable.shuffleexchange", defaultValue = true)
Review Comment:
```suggestion
getBooleanConf("spark.auron.enable.shuffleExchange", defaultValue = true)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1372]add spark.auron.enable.shuffleexchange to control ShuffleExchangeExec into Native or not [auron]
Posted by "guixiaowen (via GitHub)" <gi...@apache.org>.
guixiaowen commented on code in PR #1373:
URL: https://github.com/apache/auron/pull/1373#discussion_r2389929193
##########
spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala:
##########
@@ -138,12 +138,21 @@ object AuronConverters extends Logging {
getBooleanConf("spark.auron.enable.scan.parquet", defaultValue = true)
def enableScanOrc: Boolean =
getBooleanConf("spark.auron.enable.scan.orc", defaultValue = true)
+ def enableShuffleExechange: Boolean =
+ getBooleanConf("spark.auron.enable.shuffleexchange", defaultValue = true)
Review Comment:
>
@turboFei done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1372]Do not convert into ShuffleExchangeExec into Native when shuffle manger not set [auron]
Posted by "richox (via GitHub)" <gi...@apache.org>.
richox commented on code in PR #1373:
URL: https://github.com/apache/auron/pull/1373#discussion_r2387038181
##########
spark-extension-shims-spark3/src/test/scala/org/apache/spark/sql/auron/AuronCheckConvertShuffleExchangeSuite.scala:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.auron
+
+import org.apache.spark.sql.{QueryTest, Row, SparkSession}
+import org.apache.spark.sql.execution.auron.plan.NativeShuffleExchangeExec
+import org.apache.spark.sql.execution.exchange.ShuffleExchangeExec
+import org.apache.spark.sql.test.SharedSparkSession
+
+class AuronCheckConvertShuffleExchangeSuite
+ extends QueryTest
+ with SharedSparkSession
+ with AuronSQLTestHelper
+ with org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper {
+
+ test("test set auron shuffle manager convert to native shuffle exchange") {
+ withTable("test_shuffle") {
+ val spark = SparkSession
+ .builder()
+ .master("local[2]")
+ .appName("checkConvertToNativeShuffleManger")
+ .config("spark.sql.shuffle.partitions", "4") // 自定义 SQL 参数
Review Comment:
meaningless comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1372]add spark.auron.enable.shuffleexchange to control ShuffleExchangeExec into Native or not [auron]
Posted by "ShreyeshArangath (via GitHub)" <gi...@apache.org>.
ShreyeshArangath commented on code in PR #1373:
URL: https://github.com/apache/auron/pull/1373#discussion_r2392193241
##########
spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala:
##########
@@ -138,12 +138,21 @@ object AuronConverters extends Logging {
getBooleanConf("spark.auron.enable.scan.parquet", defaultValue = true)
def enableScanOrc: Boolean =
getBooleanConf("spark.auron.enable.scan.orc", defaultValue = true)
+ def enableShuffleExechange: Boolean =
+ getBooleanConf("spark.auron.enable.shuffleExchange", defaultValue = true)
private val extConvertProviders = ServiceLoader.load(classOf[AuronConvertProvider]).asScala
def extConvertSupported(exec: SparkPlan): Boolean = {
extConvertProviders.exists(_.isSupported(exec))
}
+ def enableExchange(): Boolean = {
+ val shuffleMangerName = SQLConf.get.getConfString(config.SHUFFLE_MANAGER.key)
+ enableShuffleExechange && !shuffleMangerName.isEmpty && (shuffleMangerName.contains(
+ "AuronShuffleManager") || shuffleMangerName.contains(
+ "AuronUniffleShuffleManager") || shuffleMangerName.contains("AuronCelebornShuffleManager"))
Review Comment:
Instead of hard coding the name of the classes, should we do something like the following to be more robust?
```scala
classOf[AuronCelebornShuffleManager].getCanonicalName
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1372]add spark.auron.enable.shuffleexchange to control ShuffleExchangeExec into Native or not [auron]
Posted by "ShreyeshArangath (via GitHub)" <gi...@apache.org>.
ShreyeshArangath commented on code in PR #1373:
URL: https://github.com/apache/auron/pull/1373#discussion_r2396445571
##########
spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala:
##########
@@ -138,12 +138,21 @@ object AuronConverters extends Logging {
getBooleanConf("spark.auron.enable.scan.parquet", defaultValue = true)
def enableScanOrc: Boolean =
getBooleanConf("spark.auron.enable.scan.orc", defaultValue = true)
+ def enableShuffleExechange: Boolean =
+ getBooleanConf("spark.auron.enable.shuffleExchange", defaultValue = true)
private val extConvertProviders = ServiceLoader.load(classOf[AuronConvertProvider]).asScala
def extConvertSupported(exec: SparkPlan): Boolean = {
extConvertProviders.exists(_.isSupported(exec))
}
+ def enableExchange(): Boolean = {
+ val shuffleMangerName = SQLConf.get.getConfString(config.SHUFFLE_MANAGER.key)
+ enableShuffleExechange && !shuffleMangerName.isEmpty && (shuffleMangerName.contains(
+ "AuronShuffleManager") || shuffleMangerName.contains(
+ "AuronUniffleShuffleManager") || shuffleMangerName.contains("AuronCelebornShuffleManager"))
Review Comment:
Ah, I see. Makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1372]add spark.auron.enable.shuffleExchange to control ShuffleExchangeExec into Native or not [auron]
Posted by "richox (via GitHub)" <gi...@apache.org>.
richox merged PR #1373:
URL: https://github.com/apache/auron/pull/1373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1372]add spark.auron.enable.shuffleExchange to control ShuffleExchangeExec into Native or not [auron]
Posted by "guixiaowen (via GitHub)" <gi...@apache.org>.
guixiaowen commented on PR #1373:
URL: https://github.com/apache/auron/pull/1373#issuecomment-3385286842
> is there any scenario that we set `spark.auron.enable` to true while not setting shuffle manager?
> is there any scenario that we set `spark.auron.enable` to true while not setting shuffle manager?
@richox ri Thank you very much for reviewing the code.
If spark.auron.enable=true is configured but the shuffle manager is not set, Spark will use the default shuffle manager.
Currently, if the default sort-based ESS is used in this case, there should be no problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org