You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2024/05/22 22:09:48 UTC

[PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

viirya opened a new pull request, #460:
URL: https://github.com/apache/datafusion-comet/pull/460

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #.
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## How are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460#discussion_r1612105585


##########
spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala:
##########
@@ -1131,9 +1129,8 @@ class CometAggregateSuite extends CometTestBase with AdaptiveSparkPlanHelper {
 
   test("var_pop and var_samp") {
     withSQLConf(CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "true") {
-      Seq(true, false).foreach { cometColumnShuffleEnabled =>
-        withSQLConf(
-          CometConf.COMET_COLUMNAR_SHUFFLE_ENABLED.key -> cometColumnShuffleEnabled.toString) {
+      Seq("native", "jvm").foreach { cometColumnShuffleEnabled =>

Review Comment:
   Okay.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460#discussion_r1611986030


##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -131,14 +132,18 @@ object CometConf {
       .booleanConf
       .createWithDefault(false)
 
-  val COMET_COLUMNAR_SHUFFLE_ENABLED: ConfigEntry[Boolean] =
-    conf("spark.comet.columnar.shuffle.enabled")
-      .doc(
-        "Whether to enable Arrow-based columnar shuffle for Comet and Spark regular operators. " +
-          "If this is enabled, Comet prefers columnar shuffle than native shuffle. " +
-          "By default, this config is true.")
-      .booleanConf
-      .createWithDefault(true)
+  val COMET_SHUFFLE_MODE: ConfigEntry[String] = conf(s"$COMET_EXEC_CONFIG_PREFIX.shuffle.mode")
+    .doc(
+      "The mode of Comet shuffle. This config is only effective only if Comet shuffle " +
+        "is enabled. Available modes are 'native', 'jvm', and 'auto'. " +
+        "'native' is for native shuffle which has best performance in general." +
+        "'jvm' is for jvm-based columnar shuffle which has higher coverage than native shuffle." +
+        "'auto' is for Comet to choose the best shuffle mode based on the query plan." +
+        "By default, this config is 'jvm'.")
+    .stringConf
+    .transform(_.toLowerCase(Locale.ROOT))
+    .checkValues(Set("native", "jvm", "auto"))

Review Comment:
   👍 



##########
spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala:
##########
@@ -1131,9 +1129,8 @@ class CometAggregateSuite extends CometTestBase with AdaptiveSparkPlanHelper {
 
   test("var_pop and var_samp") {
     withSQLConf(CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "true") {
-      Seq(true, false).foreach { cometColumnShuffleEnabled =>
-        withSQLConf(
-          CometConf.COMET_COLUMNAR_SHUFFLE_ENABLED.key -> cometColumnShuffleEnabled.toString) {
+      Seq("native", "jvm").foreach { cometColumnShuffleEnabled =>

Review Comment:
   nit: maybe update the variable name too 



##########
spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala:
##########
@@ -134,14 +134,14 @@ class CometExecSuite extends CometTestBase {
         .toDF("c1", "c2")
         .createOrReplaceTempView("v")
 
-      Seq(true, false).foreach { columnarShuffle =>
+      Seq("native", "jvm").foreach { columnarShuffle =>

Review Comment:
   nit: `shuffleMode`? 



##########
docs/source/user-guide/configs.md:
##########
@@ -39,6 +38,7 @@ Comet provides the following configuration settings.
 | spark.comet.exec.memoryFraction | The fraction of memory from Comet memory overhead that the native memory manager can use for execution. The purpose of this config is to set aside memory for untracked data structures, as well as imprecise size estimation during memory acquisition. Default value is 0.7. | 0.7 |
 | spark.comet.exec.shuffle.codec | The codec of Comet native shuffle used to compress shuffle data. Only zstd is supported. | zstd |
 | spark.comet.exec.shuffle.enabled | Whether to enable Comet native shuffle. By default, this config is false. Note that this requires setting 'spark.shuffle.manager' to 'org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager'. 'spark.shuffle.manager' must be set before starting the Spark application and cannot be changed during the application. | false |
+| spark.comet.exec.shuffle.mode | The mode of Comet shuffle. This config is only effective only if Comet shuffle is enabled. Available modes are 'native', 'jvm', and 'auto'. 'native' is for native shuffle which has best performance in general.'jvm' is for jvm-based columnar shuffle which has higher coverage than native shuffle.'auto' is for Comet to choose the best shuffle mode based on the query plan.By default, this config is 'jvm'. | jvm |

Review Comment:
   also spaces before `jvm` and `auto` 



##########
docs/source/user-guide/configs.md:
##########
@@ -39,6 +38,7 @@ Comet provides the following configuration settings.
 | spark.comet.exec.memoryFraction | The fraction of memory from Comet memory overhead that the native memory manager can use for execution. The purpose of this config is to set aside memory for untracked data structures, as well as imprecise size estimation during memory acquisition. Default value is 0.7. | 0.7 |
 | spark.comet.exec.shuffle.codec | The codec of Comet native shuffle used to compress shuffle data. Only zstd is supported. | zstd |
 | spark.comet.exec.shuffle.enabled | Whether to enable Comet native shuffle. By default, this config is false. Note that this requires setting 'spark.shuffle.manager' to 'org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager'. 'spark.shuffle.manager' must be set before starting the Spark application and cannot be changed during the application. | false |
+| spark.comet.exec.shuffle.mode | The mode of Comet shuffle. This config is only effective only if Comet shuffle is enabled. Available modes are 'native', 'jvm', and 'auto'. 'native' is for native shuffle which has best performance in general.'jvm' is for jvm-based columnar shuffle which has higher coverage than native shuffle.'auto' is for Comet to choose the best shuffle mode based on the query plan.By default, this config is 'jvm'. | jvm |

Review Comment:
   `is only effective only` -> `is only effective`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460#issuecomment-2127852781

   ```
   Error:  /Users/runner/work/datafusion-comet/datafusion-comet/spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala:1215: value COMET_COLUMNAR_SHUFFLE_ENABLED is not a member of object org.apache.comet.CometConf
   ```
   
   Weird. CI reports the above compilation error, but I don't see `COMET_COLUMNAR_SHUFFLE_ENABLED` in `CometAggregateSuite` locally...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460#discussion_r1610733007


##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -724,89 +700,80 @@ class CometSparkSessionExtensions
       }
 
       // We shouldn't transform Spark query plan if Comet is disabled.
-      if (!isCometEnabled(conf)) return plan
-
-      if (!isCometExecEnabled(conf)) {
-        // Comet exec is disabled, but for Spark shuffle, we still can use Comet columnar shuffle
-        if (isCometShuffleEnabled(conf)) {
-          applyCometShuffle(plan)
-        } else {
-          plan
-        }

Review Comment:
   I think with native execution gets more mature, it will be very rare we would like to disable native execution but trigger Comet shuffle. This is not necessary anymore so delete it to make code clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460#discussion_r1610951350


##########
docs/source/user-guide/tuning.md:
##########
@@ -39,22 +39,26 @@ It must be set before the Spark context is created. You can enable or disable Co
 at runtime by setting `spark.comet.exec.shuffle.enabled` to `true` or `false`.
 Once it is disabled, Comet will fallback to the default Spark shuffle manager.
 
-### Columnar Shuffle
+### Shuffle Mode
 
-By default, once `spark.comet.exec.shuffle.enabled` is enabled, Comet uses columnar shuffle
+Comet provides three shuffle modes: Columnar Shuffle, Native Shuffle and Auto Mode.
+
+#### Columnar Shuffle
+
+By default, once `spark.comet.exec.shuffle.enabled` is enabled, Comet uses JVM-based columnar shuffle
 to improve the performance of shuffle operations. Columnar shuffle supports HashPartitioning,
-RoundRobinPartitioning, RangePartitioning and SinglePartitioning.
+RoundRobinPartitioning, RangePartitioning and SinglePartitioning. This mode has the highest
+query coverage.
 
-Columnar shuffle can be disabled by setting `spark.comet.columnar.shuffle.enabled` to `false`.
+Columnar shuffle can be enabled by setting `spark.comet.exec.shuffle.mode` to `jvm`.
 
-### Native Shuffle
+#### Native Shuffle
 
 Comet also provides a fully native shuffle implementation that can be used to improve the performance.
-To enable native shuffle, just disable `spark.comet.columnar.shuffle.enabled`.
+To enable native shuffle, just set `spark.comet.exec.shuffle.mode` to `native`
 
 Native shuffle only supports HashPartitioning and SinglePartitioning.
 
+### Auto Mode
 
-
-
-
+`spark.comet.exec.shuffle.mode` to `auto` will let Comet choose the best shuffle mode based on the query plan.

Review Comment:
   :+1: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460#discussion_r1610733007


##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -724,89 +700,80 @@ class CometSparkSessionExtensions
       }
 
       // We shouldn't transform Spark query plan if Comet is disabled.
-      if (!isCometEnabled(conf)) return plan
-
-      if (!isCometExecEnabled(conf)) {
-        // Comet exec is disabled, but for Spark shuffle, we still can use Comet columnar shuffle
-        if (isCometShuffleEnabled(conf)) {
-          applyCometShuffle(plan)
-        } else {
-          plan
-        }

Review Comment:
   I think with native execution gets more mature, it will be very rare we would like to disable native execution but trigger Comet shuffle. This is not necessary anymore so delete it to make code clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460#issuecomment-2127914834

   Oh, it is from one patch just merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya merged PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460#issuecomment-2127412513

   cc @sunchao 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed pull request #460: feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode
URL: https://github.com/apache/datafusion-comet/pull/460


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460#issuecomment-2127740760

   > LGTM in general. How do we pick shuffle mode when it is `auto`? I don't seem to find the logic in this PR.
   
   When it is `auto`, Comet chooses native shuffle if possible as it shows better performance. If it is not available (unsupported cases), Comet uses jvm-based columnar shuffle instead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460#issuecomment-2127820215

   Thank you @sunchao 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #460:
URL: https://github.com/apache/datafusion-comet/pull/460#issuecomment-2128021619

   Merged. Thanks @sunchao @kazuyukitanimura @andygrove 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org