You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/10/09 17:38:12 UTC

[PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

andygrove opened a new pull request, #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes https://github.com/apache/datafusion-comet/issues/1006
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   Improved performance
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## How are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2424119761

   Fresh benchmarks after upmerging.
   
   ![tpch_allqueries](https://github.com/user-attachments/assets/b1e50d0e-4d31-4515-93d1-1186b5440436)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1794312492


##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -263,6 +263,12 @@ object CometConf extends ShimCometConf {
       .booleanConf
       .createWithDefault(false)
 
+  val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =
+    conf(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin")
+      .doc("Whether to replace SortMergeJoin with ShuffledHashJoin for improved performance.")
+      .booleanConf
+      .createWithDefault(true)

Review Comment:
   For this PR, I disabled the feature by default. I created the following PR to enable it by default and update the tests. I will add documentation as part of this PR.
   
   https://github.com/apache/datafusion-comet/pull/1008



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1796085416


##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -263,6 +263,12 @@ object CometConf extends ShimCometConf {
       .booleanConf
       .createWithDefault(false)
 
+  val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =
+    conf(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin")
+      .doc("Whether to replace SortMergeJoin with ShuffledHashJoin for improved performance.")
+      .booleanConf
+      .createWithDefault(true)

Review Comment:
   Here is a new follow on issue for enabling by default:
   
   https://github.com/apache/datafusion-comet/issues/1011



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1794243470


##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -263,6 +263,12 @@ object CometConf extends ShimCometConf {
       .booleanConf
       .createWithDefault(false)
 
+  val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =
+    conf(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin")
+      .doc("Whether to replace SortMergeJoin with ShuffledHashJoin for improved performance.")
+      .booleanConf
+      .createWithDefault(true)

Review Comment:
   We should add some more descriptions here to mention the risk.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1794297212


##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -263,6 +263,12 @@ object CometConf extends ShimCometConf {
       .booleanConf
       .createWithDefault(false)
 
+  val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =
+    conf(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin")
+      .doc("Whether to replace SortMergeJoin with ShuffledHashJoin for improved performance.")
+      .booleanConf
+      .createWithDefault(true)

Review Comment:
   > Perhaps we should benchmark at large scale factors before and see if we run into any issues?
   
   Agreed. (Also, when I wrote SF1 and SF10 I meant 1TB, and 10TB which is really SF 1000 and SF 10000). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2406383445

   I have run into a deadlock when running TPC-DS benchmarks with this feature, so I am moving to draft while I investigate. It is possibly related to the memory pool issues that we are also working on in other PRs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1808881709


##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -263,6 +263,14 @@ object CometConf extends ShimCometConf {
       .booleanConf
       .createWithDefault(false)
 
+  val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =
+    conf(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin")
+      .doc("Experimental feature to force Spark to replace SortMergeJoin with ShuffledHashJoin " +
+        "for improved performance. See tuning guide for more information regarding stability of " +

Review Comment:
   Good point. Updated. Thanks @jaceklaskowski 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2403334431

   There is a small danger in enabling this without having a good estimate of the size of the build side. ShuffleHashJoin has limits on how much data it can process efficiently. If the build side hash table has no spilling then a large enough build side will cause OOMs and if there is spilling, then SMJ can frequently lead to better performance. We might even see this when we scale the benchmark from SF1 to say SF10.
   Is there a way for us to get cardinality and row size for the build side somehow? 
   Still worth adding this option though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2414162169

   After upmerging, I no longer see the deadlock, but instead get an error if I have insufficient memory allocated, which is improvement.
   
   ```
   org.apache.comet.CometNativeException (External error: Internal error: Partition is still not able to allocate enough memory for the array builders after spilling..
   ```
   
   However, when I increase memory, I see queries fail due to https://github.com/apache/datafusion-comet/issues/1019, and sometimes I see:
   
   ```
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2403440894

   Current benchmarks:
   
   ![tpch_allqueries](https://github.com/user-attachments/assets/36950a1b-40e0-46db-a476-287cfbd59909)
   
   Speedup of using HashJoin instead of SortMergeJoin:
   
   ![tpch_queries_speedup](https://github.com/user-attachments/assets/2344a59a-1285-4104-b3d6-d86a13ef3995)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2405460156

   I will add documentation to this PR today, explaining pros/cons of this feature in our tuning guide.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1794237336


##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -263,6 +263,12 @@ object CometConf extends ShimCometConf {
       .booleanConf
       .createWithDefault(false)
 
+  val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =
+    conf(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin")
+      .doc("Whether to replace SortMergeJoin with ShuffledHashJoin for improved performance.")
+      .booleanConf
+      .createWithDefault(true)

Review Comment:
   The other accelerators (Spark RAPIDS and Gluten) default this to true. Perhaps we should benchmark at large scale factors before and see if we run into any issues?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "jaceklaskowski (via GitHub)" <gi...@apache.org>.
jaceklaskowski commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1807929649


##########
docs/source/user-guide/configs.md:
##########
@@ -50,6 +50,7 @@ Comet provides the following configuration settings.
 | spark.comet.exec.localLimit.enabled | Whether to enable localLimit by default. | true |
 | spark.comet.exec.memoryFraction | The fraction of memory from Comet memory overhead that the native memory manager can use for execution. The purpose of this config is to set aside memory for untracked data structures, as well as imprecise size estimation during memory acquisition. Default value is 0.7. | 0.7 |
 | spark.comet.exec.project.enabled | Whether to enable project by default. | true |
+| spark.comet.exec.replaceSortMergeJoin | Experimental feature to force Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance. See tuning guide for more information regarding stability of this feature. | false |

Review Comment:
   Can we add a link to the tuning guide? 🙏 



##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -263,6 +263,14 @@ object CometConf extends ShimCometConf {
       .booleanConf
       .createWithDefault(false)
 
+  val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =
+    conf(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin")
+      .doc("Experimental feature to force Spark to replace SortMergeJoin with ShuffledHashJoin " +
+        "for improved performance. See tuning guide for more information regarding stability of " +

Review Comment:
   Can we add a link to the tuning guide?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2424302502

   TPC-DS excluding q97 (OOM with ShuffledHashJoin).
   
   ![tpcds_allqueries](https://github.com/user-attachments/assets/37bc8ea2-0268-4bf8-b6be-4f67f4afab94)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2403021870

   Here is a teaser for the performance improvement. This is for TPC-H q11 with broadcast joins disabled (I am looking into a regression with those). I ran the query 5 times each with rule enabled vs disabled.
   
   ## Rule Off
   
   ```
   79.87537693977356,
   77.76734256744385,
   75.35734295845032,
   75.44863200187683,
   72.88174152374268
   ```
   
   ## Rule On
   
   ```
   39.33945274353027,
   36.159271240234375,
   35.83299708366394,
   35.638232707977295,
   35.67777371406555
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1794242814


##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -263,6 +263,12 @@ object CometConf extends ShimCometConf {
       .booleanConf
       .createWithDefault(false)
 
+  val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =
+    conf(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin")
+      .doc("Whether to replace SortMergeJoin with ShuffledHashJoin for improved performance.")
+      .booleanConf
+      .createWithDefault(true)

Review Comment:
   I guess it is okay for the benchmark datasets like TPCDS or TPCH. The cases I worry about is the production ones. But it might be more internal cases.
   
   For OSS, maybe enabling it by default is okay.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2423221068

   I have now marked the feature as experimental and explained in the tuning guide that there is no spill to disk so this could result in OOM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add experimental feature to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2403336632

   > if there is spilling, then SMJ can frequently lead to better performance
   I have seen this happen with Spark with some TPC-DS queries at SF10.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1793944208


##########
spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q16/simplified.txt:
##########
@@ -1,44 +1,59 @@
-WholeStageCodegen (2)
+WholeStageCodegen (7)
   HashAggregate [sum,sum,count] [sum(UnscaledValue(cs_ext_ship_cost)),sum(UnscaledValue(cs_net_profit)),count(cs_order_number),order count ,total shipping cost ,total net profit ,sum,sum,count]
     InputAdapter
       Exchange #1
-        WholeStageCodegen (1)
+        WholeStageCodegen (6)
           HashAggregate [cs_order_number] [sum(UnscaledValue(cs_ext_ship_cost)),sum(UnscaledValue(cs_net_profit)),count(cs_order_number),sum,sum,count,sum,sum,count]
             HashAggregate [cs_order_number] [sum(UnscaledValue(cs_ext_ship_cost)),sum(UnscaledValue(cs_net_profit)),sum,sum,sum,sum]
-              ColumnarToRow
-                InputAdapter
-                  CometHashAggregate [cs_order_number,sum,sum,cs_ext_ship_cost,cs_net_profit]
-                    CometProject [cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                      CometBroadcastHashJoin [cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cc_call_center_sk]
-                        CometProject [cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                          CometBroadcastHashJoin [cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,ca_address_sk]
-                            CometProject [cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                              CometBroadcastHashJoin [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,d_date_sk]
-                                CometSortMergeJoin [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cr_order_number]
-                                  CometProject [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                                    CometSortMergeJoin [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_order_number,cs_warehouse_sk]
-                                      CometSort [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                                        CometExchange [cs_order_number] #2
-                                          CometProject [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                                            CometFilter [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_sold_date_sk]
-                                              CometScan parquet spark_catalog.default.catalog_sales [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_sold_date_sk]
-                                      CometSort [cs_warehouse_sk,cs_order_number]
-                                        CometExchange [cs_order_number] #3
-                                          CometProject [cs_warehouse_sk,cs_order_number]
-                                            CometScan parquet spark_catalog.default.catalog_sales [cs_warehouse_sk,cs_order_number,cs_sold_date_sk]
-                                  CometSort [cr_order_number]
-                                    CometExchange [cr_order_number] #4
-                                      CometProject [cr_order_number]
-                                        CometScan parquet spark_catalog.default.catalog_returns [cr_order_number,cr_returned_date_sk]
-                                CometBroadcastExchange [d_date_sk] #5
-                                  CometProject [d_date_sk]
-                                    CometFilter [d_date_sk,d_date]
-                                      CometScan parquet spark_catalog.default.date_dim [d_date_sk,d_date]
-                            CometBroadcastExchange [ca_address_sk] #6
-                              CometProject [ca_address_sk]
-                                CometFilter [ca_address_sk,ca_state]
-                                  CometScan parquet spark_catalog.default.customer_address [ca_address_sk,ca_state]
-                        CometBroadcastExchange [cc_call_center_sk] #7
-                          CometProject [cc_call_center_sk]
-                            CometFilter [cc_call_center_sk,cc_county]
-                              CometScan parquet spark_catalog.default.call_center [cc_call_center_sk,cc_county]
+              HashAggregate [cs_order_number,cs_ext_ship_cost,cs_net_profit] [sum(UnscaledValue(cs_ext_ship_cost)),sum(UnscaledValue(cs_net_profit)),sum,sum,sum,sum]
+                Project [cs_order_number,cs_ext_ship_cost,cs_net_profit]
+                  BroadcastHashJoin [cs_call_center_sk,cc_call_center_sk]
+                    Project [cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
+                      BroadcastHashJoin [cs_ship_addr_sk,ca_address_sk]
+                        Project [cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
+                          BroadcastHashJoin [cs_ship_date_sk,d_date_sk]
+                            ShuffledHashJoin [cs_order_number,cr_order_number]
+                              InputAdapter
+                                WholeStageCodegen (1)
+                                  ColumnarToRow
+                                    InputAdapter
+                                      CometProject [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
+                                        CometHashJoin [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_order_number,cs_warehouse_sk]
+                                          CometExchange [cs_order_number] #2
+                                            CometProject [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
+                                              CometFilter [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_sold_date_sk]
+                                                CometScan parquet spark_catalog.default.catalog_sales [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_sold_date_sk]
+                                          CometExchange [cs_order_number] #3
+                                            CometProject [cs_warehouse_sk,cs_order_number]
+                                              CometScan parquet spark_catalog.default.catalog_sales [cs_warehouse_sk,cs_order_number,cs_sold_date_sk]
+                              InputAdapter
+                                WholeStageCodegen (2)
+                                  ColumnarToRow
+                                    InputAdapter
+                                      CometExchange [cr_order_number] #4
+                                        CometProject [cr_order_number]
+                                          CometScan parquet spark_catalog.default.catalog_returns [cr_order_number,cr_returned_date_sk]
+                            InputAdapter
+                              BroadcastExchange #5
+                                WholeStageCodegen (3)
+                                  ColumnarToRow
+                                    InputAdapter
+                                      CometProject [d_date_sk]
+                                        CometFilter [d_date_sk,d_date]
+                                          CometScan parquet spark_catalog.default.date_dim [d_date_sk,d_date]
+                        InputAdapter
+                          BroadcastExchange #6
+                            WholeStageCodegen (4)
+                              ColumnarToRow
+                                InputAdapter
+                                  CometProject [ca_address_sk]
+                                    CometFilter [ca_address_sk,ca_state]
+                                      CometScan parquet spark_catalog.default.customer_address [ca_address_sk,ca_state]
+                    InputAdapter
+                      BroadcastExchange #7

Review Comment:
   This is a regression that I am looking into.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2403600852

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1007?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `18.18182%` with `27 lines` in your changes missing coverage. Please review.
   > Project coverage is 34.19%. Comparing base [(`c3023c5`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/c3023c573592f52132cde0b0e5303b00884d3416?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`99eca10`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/99eca10b774d4d768c8754023484b254057677bc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 25 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/1007?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...ain/scala/org/apache/comet/rules/RewriteJoin.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/1007?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Frules%2FRewriteJoin.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9ydWxlcy9SZXdyaXRlSm9pbi5zY2FsYQ==) | 0.00% | [24 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/1007?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...org/apache/comet/CometSparkSessionExtensions.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/1007?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2FCometSparkSessionExtensions.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9Db21ldFNwYXJrU2Vzc2lvbkV4dGVuc2lvbnMuc2NhbGE=) | 40.00% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/1007?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##               main    #1007      +/-   ##
   ============================================
   + Coverage     34.03%   34.19%   +0.15%     
   - Complexity      875      881       +6     
   ============================================
     Files           112      113       +1     
     Lines         43289    43458     +169     
     Branches       9572     9628      +56     
   ============================================
   + Hits          14734    14861     +127     
   - Misses        25521    25535      +14     
   - Partials       3034     3062      +28     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/datafusion-comet/pull/1007/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [](https://app.codecov.io/gh/apache/datafusion-comet/pull/1007/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/1007?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1795751218


##########
kube/Dockerfile:
##########
@@ -65,4 +65,4 @@ ENV SCALA_VERSION=2.12
 USER root
 
 # note the use of a wildcard in the file name so that this works with both snapshot and final release versions
-COPY --from=builder  /comet/spark/target/comet-spark-spark${SPARK_VERSION}_$SCALA_VERSION-0.2.0*.jar $SPARK_HOME/jars
+COPY --from=builder  /comet/spark/target/comet-spark-spark${SPARK_VERSION}_$SCALA_VERSION-*.jar $SPARK_HOME/jars

Review Comment:
   unrelated, but ran into this hard-coded version number during testing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1793944208


##########
spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q16/simplified.txt:
##########
@@ -1,44 +1,59 @@
-WholeStageCodegen (2)
+WholeStageCodegen (7)
   HashAggregate [sum,sum,count] [sum(UnscaledValue(cs_ext_ship_cost)),sum(UnscaledValue(cs_net_profit)),count(cs_order_number),order count ,total shipping cost ,total net profit ,sum,sum,count]
     InputAdapter
       Exchange #1
-        WholeStageCodegen (1)
+        WholeStageCodegen (6)
           HashAggregate [cs_order_number] [sum(UnscaledValue(cs_ext_ship_cost)),sum(UnscaledValue(cs_net_profit)),count(cs_order_number),sum,sum,count,sum,sum,count]
             HashAggregate [cs_order_number] [sum(UnscaledValue(cs_ext_ship_cost)),sum(UnscaledValue(cs_net_profit)),sum,sum,sum,sum]
-              ColumnarToRow
-                InputAdapter
-                  CometHashAggregate [cs_order_number,sum,sum,cs_ext_ship_cost,cs_net_profit]
-                    CometProject [cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                      CometBroadcastHashJoin [cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cc_call_center_sk]
-                        CometProject [cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                          CometBroadcastHashJoin [cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,ca_address_sk]
-                            CometProject [cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                              CometBroadcastHashJoin [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,d_date_sk]
-                                CometSortMergeJoin [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cr_order_number]
-                                  CometProject [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                                    CometSortMergeJoin [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_order_number,cs_warehouse_sk]
-                                      CometSort [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                                        CometExchange [cs_order_number] #2
-                                          CometProject [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
-                                            CometFilter [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_sold_date_sk]
-                                              CometScan parquet spark_catalog.default.catalog_sales [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_sold_date_sk]
-                                      CometSort [cs_warehouse_sk,cs_order_number]
-                                        CometExchange [cs_order_number] #3
-                                          CometProject [cs_warehouse_sk,cs_order_number]
-                                            CometScan parquet spark_catalog.default.catalog_sales [cs_warehouse_sk,cs_order_number,cs_sold_date_sk]
-                                  CometSort [cr_order_number]
-                                    CometExchange [cr_order_number] #4
-                                      CometProject [cr_order_number]
-                                        CometScan parquet spark_catalog.default.catalog_returns [cr_order_number,cr_returned_date_sk]
-                                CometBroadcastExchange [d_date_sk] #5
-                                  CometProject [d_date_sk]
-                                    CometFilter [d_date_sk,d_date]
-                                      CometScan parquet spark_catalog.default.date_dim [d_date_sk,d_date]
-                            CometBroadcastExchange [ca_address_sk] #6
-                              CometProject [ca_address_sk]
-                                CometFilter [ca_address_sk,ca_state]
-                                  CometScan parquet spark_catalog.default.customer_address [ca_address_sk,ca_state]
-                        CometBroadcastExchange [cc_call_center_sk] #7
-                          CometProject [cc_call_center_sk]
-                            CometFilter [cc_call_center_sk,cc_county]
-                              CometScan parquet spark_catalog.default.call_center [cc_call_center_sk,cc_county]
+              HashAggregate [cs_order_number,cs_ext_ship_cost,cs_net_profit] [sum(UnscaledValue(cs_ext_ship_cost)),sum(UnscaledValue(cs_net_profit)),sum,sum,sum,sum]
+                Project [cs_order_number,cs_ext_ship_cost,cs_net_profit]
+                  BroadcastHashJoin [cs_call_center_sk,cc_call_center_sk]
+                    Project [cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
+                      BroadcastHashJoin [cs_ship_addr_sk,ca_address_sk]
+                        Project [cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
+                          BroadcastHashJoin [cs_ship_date_sk,d_date_sk]
+                            ShuffledHashJoin [cs_order_number,cr_order_number]
+                              InputAdapter
+                                WholeStageCodegen (1)
+                                  ColumnarToRow
+                                    InputAdapter
+                                      CometProject [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
+                                        CometHashJoin [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_order_number,cs_warehouse_sk]
+                                          CometExchange [cs_order_number] #2
+                                            CometProject [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
+                                              CometFilter [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_sold_date_sk]
+                                                CometScan parquet spark_catalog.default.catalog_sales [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_sold_date_sk]
+                                          CometExchange [cs_order_number] #3
+                                            CometProject [cs_warehouse_sk,cs_order_number]
+                                              CometScan parquet spark_catalog.default.catalog_sales [cs_warehouse_sk,cs_order_number,cs_sold_date_sk]
+                              InputAdapter
+                                WholeStageCodegen (2)
+                                  ColumnarToRow
+                                    InputAdapter
+                                      CometExchange [cr_order_number] #4
+                                        CometProject [cr_order_number]
+                                          CometScan parquet spark_catalog.default.catalog_returns [cr_order_number,cr_returned_date_sk]
+                            InputAdapter
+                              BroadcastExchange #5
+                                WholeStageCodegen (3)
+                                  ColumnarToRow
+                                    InputAdapter
+                                      CometProject [d_date_sk]
+                                        CometFilter [d_date_sk,d_date]
+                                          CometScan parquet spark_catalog.default.date_dim [d_date_sk,d_date]
+                        InputAdapter
+                          BroadcastExchange #6
+                            WholeStageCodegen (4)
+                              ColumnarToRow
+                                InputAdapter
+                                  CometProject [ca_address_sk]
+                                    CometFilter [ca_address_sk,ca_state]
+                                      CometScan parquet spark_catalog.default.customer_address [ca_address_sk,ca_state]
+                    InputAdapter
+                      BroadcastExchange #7

Review Comment:
   This is a regression that I am looking into (falling back to Spark for BroadcastHashJoin)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#issuecomment-2405546782

   @viirya @parthchandra This is now ready for review. The new option is disabled by default and I added a section to the tuning guide explaining why users may want to enable this new option.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1794234836


##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -263,6 +263,12 @@ object CometConf extends ShimCometConf {
       .booleanConf
       .createWithDefault(false)
 
+  val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =
+    conf(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin")
+      .doc("Whether to replace SortMergeJoin with ShuffledHashJoin for improved performance.")
+      .booleanConf
+      .createWithDefault(true)

Review Comment:
   I think we should have a default value as false for stablility. Spark decides to use SMJ for some reasons including data statistics. If Spark thinks SHJ may not work, I think we better follow it except for explicitly asking by users.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: Add option to replace SortMergeJoin with ShuffledHashJoin [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1794243470


##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -263,6 +263,12 @@ object CometConf extends ShimCometConf {
       .booleanConf
       .createWithDefault(false)
 
+  val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =
+    conf(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin")
+      .doc("Whether to replace SortMergeJoin with ShuffledHashJoin for improved performance.")
+      .booleanConf
+      .createWithDefault(true)

Review Comment:
   At least, we should add some more descriptions here to mention the risk.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org