You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "EmilyMatt (via GitHub)" <gi...@apache.org> on 2025/02/12 13:53:29 UTC
[PR] Ok push for now until I understand how to regenerate golden files [datafusion-comet]
EmilyMatt opened a new pull request, #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390
## What issue does this close?
Closes #1389 .
## Rationale for this change
As described in the issue, we'd like to prevent situations where despite the Partial aggregate being supported and converted, and the shuffle being supported and converted, the Final would not be converted, because the result expressions were not supported.
This leads to an unrecoverable state, where Spark expects an aggregate buffer to be created by the Partial HA and it doesn't exist.
## What changes are included in this PR?
I've separated the conversion of the hash aggregate into a separate function(I believe everything should be separated tbh, its very hard to manage rn), which also returns information about whether the result expressions were converted, when they are not, we create a new ProjectExec with those result expressions, convert the HA without them, and place a conversion between the two, that way we can ensure a valid state at all times.
This feature can be ignored by enforcing result conversion, using "spark.comet.exec.aggregate.enforceResults=true",
result enforcing is disabled by default.
## How are these changes tested?
Essentially a lot of the stability tests, will have a new plan where the aggregate is completed natively, and the ProjectExec runs in Spark, instead of the current situation, where the final stage of the HashAggregate is done in Spark completely.
Those tests currently fail because I am unable to run them with SPARK_GENERATE_GOLDEN_FILES, might be a skill issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#discussion_r1956799716
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -216,6 +216,17 @@ object CometConf extends ShimCometConf {
val COMET_EXEC_INITCAP_ENABLED: ConfigEntry[Boolean] =
createExecEnabledConfig("initCap", defaultValue = false)
+ val COMET_EXEC_AGGREGATE_ENFORCE_RESULTS: ConfigEntry[Boolean] =
+ conf("spark.comet.exec.aggregate.enforceResults")
+ .doc("Whether to enforce converting results in the Final stage of a HashAggregate, " +
+ "When enabled, Final-mode hashAggregates will not be converted to Comet, this can cause " +
+ "issues when native shuffle is enabled. " +
+ "If this is disabled, unsupported result expressions will be " +
+ "separated into a ProjectExec to allow HashAggregate to complete natively. " +
+ "This is disabled by default.")
Review Comment:
I would like to understand this more
`Final-mode hashAggregates will not be converted to Comet` so final aggregation falls back to Spark?
`this can cause isues when native shuffle is enabled.` why it is the case?
And when should we enable this option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#issuecomment-2656455632
@andygrove I've updated "test final min/max/count with result expressions" in the Aggregate suite to verify this more deeply, and I think many of the queries in the stability tests also have this issue.
The current implementation does not crash or anything but just converts the intermediate results back to Spark so they can be used in the Spark HashAggregate.
With this PR the aggregate will be fully native so hopefully this will be a performance boost.
I think a crash will be encountered in any Aggregate that uses an aggregate buffer, as the Partial CometHashAggregate will not generate the expected IntermediateAggBuffer representation Spark expects, and possibly not even the same data type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#issuecomment-2654307818
Thanks @EmilyMatt. Would it be possible to add a test to reproduce the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt closed pull request #1390: fix: AQE creating a non-supported Final HashAggregate post-shuffle
URL: https://github.com/apache/datafusion-comet/pull/1390
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#issuecomment-2660546336
#1389 mentioned
https://github.com/apache/datafusion-comet/blob/f099e6e40aa18441c7882e5bffd9d6dfb10c6c19/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala#L491-L494
```
I believe I've seen a few tests that are ignored because of this.
I don't think this is a valid situation, We should not crash based on previously Comet-ran operators if they were successful.
```
Are we planning to remove this comet shuffle requirement with this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#discussion_r1956865360
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -216,6 +216,17 @@ object CometConf extends ShimCometConf {
val COMET_EXEC_INITCAP_ENABLED: ConfigEntry[Boolean] =
createExecEnabledConfig("initCap", defaultValue = false)
+ val COMET_EXEC_AGGREGATE_ENFORCE_RESULTS: ConfigEntry[Boolean] =
+ conf("spark.comet.exec.aggregate.enforceResults")
+ .doc("Whether to enforce converting results in the Final stage of a HashAggregate, " +
+ "When enabled, Final-mode hashAggregates will not be converted to Comet, this can cause " +
+ "issues when native shuffle is enabled. " +
+ "If this is disabled, unsupported result expressions will be " +
+ "separated into a ProjectExec to allow HashAggregate to complete natively. " +
+ "This is disabled by default.")
Review Comment:
I guess `When the result expression of a Final-mode HashAggregate is unsupported, the entire HashAggregate will fall back to Spark with this option enabled` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on code in PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#discussion_r1963122920
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -216,6 +216,17 @@ object CometConf extends ShimCometConf {
val COMET_EXEC_INITCAP_ENABLED: ConfigEntry[Boolean] =
createExecEnabledConfig("initCap", defaultValue = false)
+ val COMET_EXEC_AGGREGATE_ENFORCE_RESULTS: ConfigEntry[Boolean] =
+ conf("spark.comet.exec.aggregate.enforceResults")
+ .doc("Whether to enforce converting results in the Final stage of a HashAggregate, " +
+ "When enabled, Final-mode hashAggregates will not be converted to Comet, this can cause " +
+ "issues when native shuffle is enabled. " +
+ "If this is disabled, unsupported result expressions will be " +
+ "separated into a ProjectExec to allow HashAggregate to complete natively. " +
+ "This is disabled by default.")
Review Comment:
Will update the documentation for the configuration option, hopefully will be able to regenerate the tpcds plans
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#issuecomment-2820434223
> @EmilyMatt I see that plan stability tests are failing. Do you plan to update the golden files?
I was unable to, I tried following the guide, as well as tried various changes, none of them worked, the compilation process was weird using the command in the docs, and did not contain my changes nor lots of other recent changes in main
I think I'll close this PR and open it when it is more relevant, as at the moment it is only a performance boost, not a crash fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#issuecomment-2689386875
Thanks @EmilyMatt
> I don't really know why this restriction was placed here, so I feel I can't really provide an opinion on any direction, I only saw the comment and figured it is relevant as it speaks of this specific issue.
Despite the comment and restriction, this still occurs many times in our tests, so I am unsure of its importance
I guess we can keep it as is for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on code in PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#discussion_r1963121789
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -216,6 +216,17 @@ object CometConf extends ShimCometConf {
val COMET_EXEC_INITCAP_ENABLED: ConfigEntry[Boolean] =
createExecEnabledConfig("initCap", defaultValue = false)
+ val COMET_EXEC_AGGREGATE_ENFORCE_RESULTS: ConfigEntry[Boolean] =
+ conf("spark.comet.exec.aggregate.enforceResults")
+ .doc("Whether to enforce converting results in the Final stage of a HashAggregate, " +
+ "When enabled, Final-mode hashAggregates will not be converted to Comet, this can cause " +
+ "issues when native shuffle is enabled. " +
+ "If this is disabled, unsupported result expressions will be " +
+ "separated into a ProjectExec to allow HashAggregate to complete natively. " +
+ "This is disabled by default.")
Review Comment:
@kazuyukitanimura
Apologies, I believe I intended to say when native shuffle is disabled(or more correctly - when columnar shuffle is used)^
Essentially whenever the Schema would come from Spark and not from the data/batch itself, we will start having problems.
Looking at the tpcds tests, we can see an example:
`HashAggregate [cp_catalog_page_id,sum,sum,isEmpty,sum,isEmpty] [sum(UnscaledValue(cs_ext_sales_price)),sum(coalesce(cast(cr_return_amount as decimal(12,2)), 0.00)),sum((cs_net_profit - coalesce(cast(cr_net_loss as decimal(12,2)), 0.00))),sales,returns,profit,channel,id,sum,sum,isEmpty,sum,isEmpty]
CometColumnarToRow
InputAdapter
CometExchange [cp_catalog_page_id] #10
CometHashAggregate `
See how the Partial HA is a Comet one, and the Final one is a HashAggregate with a conversion,
This is the current implementation, and works for both shuffles, as comet still produces the data Spark expects.
However, the moment we'll output something Spark does not expect(like having the partial results of a CollectSet, the columnar shuffle will crash, due to the mismatch in data types)
The issue with the shuffle can probably be circumvented by shuffling the aggregate buffer as a binary column regardless of its datatype, then reforming it in the Final aggregate, that way both shuffles will function.
However, as discussed in the issue, this will only delay the inevitable in cases such as an unsupported ResultExpression, as a CometColumnarToRow will not recreate the expected data type a regular HashAggregate will expect, and there is no other path forward but to let the Comet aggregate expressions run their course.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#issuecomment-2654449945
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1390?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 34.23%. Comparing base [(`f09f8af`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`a390ba1`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/a390ba1830d8b58a3fbf67360912c6fd9d2cc538?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 23 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #1390 +/- ##
=============================================
- Coverage 56.12% 34.23% -21.89%
- Complexity 976 1027 +51
=============================================
Files 119 141 +22
Lines 11743 48676 +36933
Branches 2251 10643 +8392
=============================================
+ Hits 6591 16666 +10075
- Misses 4012 28669 +24657
- Partials 1140 3341 +2201
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/1390?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#issuecomment-2670873286
> #1389 mentioned
>
> https://github.com/apache/datafusion-comet/blob/f099e6e40aa18441c7882e5bffd9d6dfb10c6c19/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala#L491-L494
>
> ```
> I believe I've seen a few tests that are ignored because of this.
> I don't think this is a valid situation, We should not crash based on previously Comet-ran operators if they were successful.
> ```
>
> Are we planning to remove this comet shuffle requirement with this PR?
I don't really know why this restriction was placed here, so I feel I can't really provide an opinion on any direction, I only saw the comment and figured it is relevant as it speaks of this specific issue.
Despite the comment and restriction, this still occurs many times in our tests, so I am unsure of its importance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: AQE creating a non-supported Final HashAggregate post-shuffle [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #1390:
URL: https://github.com/apache/datafusion-comet/pull/1390#issuecomment-2702515744
@EmilyMatt I see that plan stability tests are failing. Do you plan to update the golden files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org