You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/01/09 16:52:49 UTC
[I] Remove COMET_SHUFFLE_FALLBACK_TO_COLUMNAR config [datafusion-comet]
andygrove opened a new issue, #1254:
URL: https://github.com/apache/datafusion-comet/issues/1254
### What is the problem the feature request solves?
https://github.com/apache/datafusion-comet/pull/1209 enabled accelerating more queries with Comet but also exposed some new bugs so a COMET_SHUFFLE_FALLBACK_TO_COLUMNAR config was added as a temporary workaround to avoid Spark SQL tests failing.
We should remove this config once the bugs are fixed so that we test more of the Spark SQL queries with Comet shuffle.
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Spark SQL test failures when Comet JVM shuffle is used [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #1254:
URL: https://github.com/apache/datafusion-comet/issues/1254#issuecomment-2930122937
Barring AQE and DPP tets ( addressed in a different PR https://github.com/apache/datafusion-comet/pull/1811 ) , I am able to run these tests successfully . Not sure if this is still an issue @andygrove ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Spark SQL test failures when Comet JVM shuffle is used [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1254:
URL: https://github.com/apache/datafusion-comet/issues/1254#issuecomment-2936747407
> Barring AQE and DPP tets ( addressed in a different PR [#1811](https://github.com/apache/datafusion-comet/pull/1811) ) , I am able to run these tests successfully . Not sure if this is still an issue [@andygrove](https://github.com/andygrove) ?
The tests are still failing in CI in https://github.com/apache/datafusion-comet/pull/1736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Spark SQL test failures when Comet JVM shuffle is used [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1254:
URL: https://github.com/apache/datafusion-comet/issues/1254#issuecomment-2971117509
I removed the following from the scope of this issue since they turned out not to be bugs or correctness issues, but valid failures because Comet does not support DPP natively yet.
- SPARK-34637: DPP side broadcast query stage is created firstly *** FAILED *** (235 milliseconds)
- https://github.com/apache/datafusion-comet/issues/1839
- avoid reordering broadcast join keys to match input hash partitioning *** FAILED *** (1 second, 659 milliseconds)
- https://github.com/apache/datafusion-comet/issues/1839
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Spark SQL test failures when Comet JVM shuffle is used [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1254: [EPIC] Spark SQL test failures when Comet JVM shuffle is used
URL: https://github.com/apache/datafusion-comet/issues/1254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org