You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2025/11/07 01:04:05 UTC
[PR] chore: Fallback to Spark for windows functions [datafusion-comet]
comphead opened a new pull request, #2726:
URL: https://github.com/apache/datafusion-comet/pull/2726
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Related to #2721
Closes #.
## Rationale for this change
Currently windows functions are enabled by default, however we can spot numerous correctness issues which need to be addressed separately. List of correctness issues can be found in #2721
```
- window: MIN and MAX with ORDER BY (383 milliseconds)
- window: COUNT with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW *** FAILED *** (196 milliseconds)
Results do not match for query:
--
- window: SUM with ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING *** FAILED *** (190 milliseconds)
Results do not match for query:
--
- window: AVG with ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING *** FAILED *** (183 milliseconds)
Results do not match for query:
--
- window: LAG with default offset *** FAILED *** (192 milliseconds)
Results do not match for query:
--
- window: LAG with offset 2 and default value *** FAILED *** (184 milliseconds)
Results do not match for query:
--
- window: LEAD with default offset *** FAILED *** (176 milliseconds)
Results do not match for query:
--
- window: LEAD with offset 2 and default value *** FAILED *** (173 milliseconds)
Results do not match for query:
- window: multiple window functions in single query *** FAILED *** (195 milliseconds)
Results do not match for query:
- window: ROWS BETWEEN with negative offset *** FAILED *** (184 milliseconds)
Results do not match for query:
--
- window: all ranking functions together *** FAILED *** (197 milliseconds)
Results do not match for query:
```
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## How are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Fallback to Spark for windows functions [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2726:
URL: https://github.com/apache/datafusion-comet/pull/2726#discussion_r2506240503
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -251,7 +251,7 @@ object CometConf extends ShimCometConf {
val COMET_EXEC_EXPAND_ENABLED: ConfigEntry[Boolean] =
createExecEnabledConfig("expand", defaultValue = true)
val COMET_EXEC_WINDOW_ENABLED: ConfigEntry[Boolean] =
- createExecEnabledConfig("window", defaultValue = true)
+ createExecEnabledConfig("window", defaultValue = false)
Review Comment:
I think we need to double check how Comet works with `exec.*.enabled` params.
In this example I disabled `windows` and actually expected that Comet would fallback silently to Spark for windows, however I got a bunch of weirdest issues, including issues on native side
```
2025-11-08T05:18:43.0134899Z [info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3931.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3931.0 (TID 4375) (10306839d6c4 executor driver): org.apache.comet.CometNativeException: assertion failed: !self.format.is_null()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Fallback to Spark for windows functions [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2726:
URL: https://github.com/apache/datafusion-comet/pull/2726#discussion_r2501348049
##########
docs/source/user-guide/latest/configs.md:
##########
@@ -139,23 +139,23 @@ These settings can be used to determine which parts of the plan are accelerated
<!--BEGIN:CONFIG_TABLE[enable_exec]-->
| Config | Description | Default Value |
|--------|-------------|---------------|
-| `spark.comet.exec.aggregate.enabled` | Whether to enable aggregate by default. | true |
-| `spark.comet.exec.broadcastExchange.enabled` | Whether to enable broadcastExchange by default. | true |
-| `spark.comet.exec.broadcastHashJoin.enabled` | Whether to enable broadcastHashJoin by default. | true |
-| `spark.comet.exec.coalesce.enabled` | Whether to enable coalesce by default. | true |
-| `spark.comet.exec.collectLimit.enabled` | Whether to enable collectLimit by default. | true |
-| `spark.comet.exec.expand.enabled` | Whether to enable expand by default. | true |
-| `spark.comet.exec.filter.enabled` | Whether to enable filter by default. | true |
-| `spark.comet.exec.globalLimit.enabled` | Whether to enable globalLimit by default. | true |
-| `spark.comet.exec.hashJoin.enabled` | Whether to enable hashJoin by default. | true |
-| `spark.comet.exec.localLimit.enabled` | Whether to enable localLimit by default. | true |
-| `spark.comet.exec.project.enabled` | Whether to enable project by default. | true |
-| `spark.comet.exec.sort.enabled` | Whether to enable sort by default. | true |
-| `spark.comet.exec.sortMergeJoin.enabled` | Whether to enable sortMergeJoin by default. | true |
-| `spark.comet.exec.sortMergeJoinWithJoinFilter.enabled` | Experimental support for Sort Merge Join with filter | false |
-| `spark.comet.exec.takeOrderedAndProject.enabled` | Whether to enable takeOrderedAndProject by default. | true |
-| `spark.comet.exec.union.enabled` | Whether to enable union by default. | true |
-| `spark.comet.exec.window.enabled` | Whether to enable window by default. | true |
+| `spark.comet.exec.aggregate.enabled` | Whether to enable aggregate by default. | true |
+| `spark.comet.exec.broadcastExchange.enabled` | Whether to enable broadcastExchange by default. | true |
+| `spark.comet.exec.broadcastHashJoin.enabled` | Whether to enable broadcastHashJoin by default. | true |
+| `spark.comet.exec.coalesce.enabled` | Whether to enable coalesce by default. | true |
+| `spark.comet.exec.collectLimit.enabled` | Whether to enable collectLimit by default. | true |
+| `spark.comet.exec.expand.enabled` | Whether to enable expand by default. | true |
+| `spark.comet.exec.filter.enabled` | Whether to enable filter by default. | true |
+| `spark.comet.exec.globalLimit.enabled` | Whether to enable globalLimit by default. | true |
+| `spark.comet.exec.hashJoin.enabled` | Whether to enable hashJoin by default. | true |
+| `spark.comet.exec.localLimit.enabled` | Whether to enable localLimit by default. | true |
+| `spark.comet.exec.project.enabled` | Whether to enable project by default. | true |
+| `spark.comet.exec.sort.enabled` | Whether to enable sort by default. | true |
+| `spark.comet.exec.sortMergeJoin.enabled` | Whether to enable sortMergeJoin by default. | true |
+| `spark.comet.exec.sortMergeJoinWithJoinFilter.enabled` | Experimental support for Sort Merge Join with filter | false |
+| `spark.comet.exec.takeOrderedAndProject.enabled` | Whether to enable takeOrderedAndProject by default. | true |
+| `spark.comet.exec.union.enabled` | Whether to enable union by default. | true |
+| `spark.comet.exec.window.enabled` | Whether to enable window by default. | false |
Review Comment:
this was the only changes, looks like IDE reformatted the file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Fallback to Spark for windows functions [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #2726:
URL: https://github.com/apache/datafusion-comet/pull/2726#discussion_r2506992772
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -251,7 +251,7 @@ object CometConf extends ShimCometConf {
val COMET_EXEC_EXPAND_ENABLED: ConfigEntry[Boolean] =
createExecEnabledConfig("expand", defaultValue = true)
val COMET_EXEC_WINDOW_ENABLED: ConfigEntry[Boolean] =
- createExecEnabledConfig("window", defaultValue = true)
+ createExecEnabledConfig("window", defaultValue = false)
Review Comment:
yes, this section of code is very problematic and brittle. I am trying to understand this too.
https://github.com/apache/datafusion-comet/issues/2737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Fallback to Spark for windows functions [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #2726:
URL: https://github.com/apache/datafusion-comet/pull/2726#issuecomment-3500075746
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2726?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 10.82%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`76390cd`](https://app.codecov.io/gh/apache/datafusion-comet/commit/76390cd23aef0793aac09b1c3da137fb22876070?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 670 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2726 +/- ##
=============================================
- Coverage 56.12% 10.82% -45.30%
+ Complexity 976 232 -744
=============================================
Files 119 147 +28
Lines 11743 13806 +2063
Branches 2251 2353 +102
=============================================
- Hits 6591 1495 -5096
- Misses 4012 12136 +8124
+ Partials 1140 175 -965
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/2726?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Fallback to Spark for windows functions [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2726:
URL: https://github.com/apache/datafusion-comet/pull/2726#discussion_r2506240503
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -251,7 +251,7 @@ object CometConf extends ShimCometConf {
val COMET_EXEC_EXPAND_ENABLED: ConfigEntry[Boolean] =
createExecEnabledConfig("expand", defaultValue = true)
val COMET_EXEC_WINDOW_ENABLED: ConfigEntry[Boolean] =
- createExecEnabledConfig("window", defaultValue = true)
+ createExecEnabledConfig("window", defaultValue = false)
Review Comment:
I think we need to double check how Comet works with `exec.*.enabled` params.
In this example I disabled `windows` and actually expected that Comet would fallback silently to Spark for windows, however I got a bunch of weirdest issues, including corrupted schema on native side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Fallback to Spark for windows functions [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #2726:
URL: https://github.com/apache/datafusion-comet/pull/2726#discussion_r2505938148
##########
spark/src/test/scala/org/apache/comet/exec/CometWindowExecSuite.scala:
##########
@@ -39,12 +41,86 @@ class CometWindowExecSuite extends CometTestBase {
super.test(testName, testTags: _*) {
withSQLConf(
CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "true",
+ CometConf.COMET_EXEC_WINDOW_ENABLED.key -> "true",
CometConf.COMET_NATIVE_SCAN_IMPL.key -> CometConf.SCAN_AUTO) {
testFun
}
}
}
+ test("lead/lag should return the default value if the offset row does not exist") {
Review Comment:
It looks like the same test is here as well as in `CometWindowExecSuite` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Fallback to Spark for windows functions [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #2726:
URL: https://github.com/apache/datafusion-comet/pull/2726#discussion_r2501386240
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -251,7 +251,7 @@ object CometConf extends ShimCometConf {
val COMET_EXEC_EXPAND_ENABLED: ConfigEntry[Boolean] =
createExecEnabledConfig("expand", defaultValue = true)
val COMET_EXEC_WINDOW_ENABLED: ConfigEntry[Boolean] =
- createExecEnabledConfig("window", defaultValue = true)
+ createExecEnabledConfig("window", defaultValue = false)
Review Comment:
:+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Fallback to Spark for windows functions [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2726:
URL: https://github.com/apache/datafusion-comet/pull/2726#discussion_r2506199769
##########
spark/src/test/scala/org/apache/comet/exec/CometWindowExecSuite.scala:
##########
@@ -39,12 +41,86 @@ class CometWindowExecSuite extends CometTestBase {
super.test(testName, testTags: _*) {
withSQLConf(
CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "true",
+ CometConf.COMET_EXEC_WINDOW_ENABLED.key -> "true",
CometConf.COMET_NATIVE_SCAN_IMPL.key -> CometConf.SCAN_AUTO) {
testFun
}
}
}
+ test("lead/lag should return the default value if the offset row does not exist") {
Review Comment:
Moved windows related tests from all files to `CometWindowExecSuite `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Fallback to Spark for windows functions [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #2726:
URL: https://github.com/apache/datafusion-comet/pull/2726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org