You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/06/03 22:03:50 UTC
[I] DPP exchange reuse issue when using columnar shuffle [datafusion-comet]
andygrove opened a new issue, #1839:
URL: https://github.com/apache/datafusion-comet/issues/1839
### Describe the bug
The test `SPARK-34637: DPP side broadcast query stage is created firstly` passes when using native shuffle, but fails when using columnar shuffle.
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Spark SQL] Enable all tests in DynamicPartitionPruningSuite [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich closed issue #1839: [Spark SQL] Enable all tests in DynamicPartitionPruningSuite
URL: https://github.com/apache/datafusion-comet/issues/1839
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Spark SQL] Enable all tests in DynamicPartitionPruningSuite [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1839:
URL: https://github.com/apache/datafusion-comet/issues/1839#issuecomment-2967483655
All of these tests will need to be ignored until we support DPP (cc @coderfender). This is not a correctness issue but a performance issue due to not reusing exchanges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Spark SQL] Enable all tests in DynamicPartitionPruningSuite [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1839:
URL: https://github.com/apache/datafusion-comet/issues/1839#issuecomment-2967466668
The "canocilization and exchange reuse" test is expected to fail and should be ignored until Comet supports DPP. The two exchanges are different. One contains a `CometScan` and one contains `FileScan` with a `SubqueryAdaptiveBroadcast` (due to fallback because of no DPP support).
```
Look for CometColumnarExchange hashpartitioning(store_id#3596, 5), ENSURE_REQUIREMENTS, [plan_id=543]
+- Project [store_id#3596]
+- Filter (isnotnull(units_sold#3595) AND (units_sold#3595 = 70))
+- FileScan parquet spark_catalog.default.fact_stats[units_sold#3595,store_id#3596] Batched: true, DataFilters: [isnotnull(units_sold#3595), (units_sold#3595 = 70)], Format: Parquet, Location: InMemoryFileIndex(25 paths)[file:/home/andy/git/apache/apache-spark-3.5.6/sql/core/spark-warehous..., PartitionFilters: [isnotnull(store_id#3596), dynamicpruningexpression(store_id#3596 IN dynamicpruning#3609)], PushedFilters: [IsNotNull(units_sold), EqualTo(units_sold,70)], ReadSchema: struct<units_sold:int>
+- SubqueryAdaptiveBroadcast dynamicpruning#3609, 0, true, Project [store_id#3608], [store_id#3608]
+- AdaptiveSparkPlan isFinalPlan=false
+- CometProject [store_id#3608], [store_id#3608]
+- CometFilter [units_sold#3607, store_id#3608], (isnotnull(units_sold#3607) AND (units_sold#3607 = 70))
+- CometScan parquet spark_catalog.default.fact_stats[units_sold#3607,store_id#3608] Batched: true, DataFilters: [isnotnull(units_sold#3607), (units_sold#3607 = 70)], Format: CometParquet, Location: InMemoryFileIndex(25 paths)[file:/home/andy/git/apache/apache-spark-3.5.6/sql/core/spark-warehous..., PartitionFilters: [isnotnull(store_id#3608)], PushedFilters: [IsNotNull(units_sold), EqualTo(units_sold,70)], ReadSchema: struct<units_sold:int>
getOrElseUpdate with CometColumnarExchange hashpartitioning(store_id#3596, 5), ENSURE_REQUIREMENTS, [plan_id=568]
+- CometProject [store_id#3596], [store_id#3596]
+- CometFilter [units_sold#3595, store_id#3596], (isnotnull(units_sold#3595) AND (units_sold#3595 = 70))
+- CometScan parquet spark_catalog.default.fact_stats[units_sold#3595,store_id#3596] Batched: true, DataFilters: [isnotnull(units_sold#3595), (units_sold#3595 = 70)], Format: CometParquet, Location: InMemoryFileIndex(25 paths)[file:/home/andy/git/apache/apache-spark-3.5.6/sql/core/spark-warehous..., PartitionFilters: [isnotnull(store_id#3596), dynamicpruningexpression(true)], PushedFilters: [IsNotNull(units_sold), EqualTo(units_sold,70)], ReadSchema: struct<units_sold:int>
NO REUSE
Look for CometColumnarExchange hashpartitioning(store_id#3608, 5), ENSURE_REQUIREMENTS, [plan_id=570]
+- CometProject [store_id#3608], [store_id#3608]
+- CometFilter [units_sold#3607, store_id#3608], (isnotnull(units_sold#3607) AND (units_sold#3607 = 70))
+- CometScan parquet spark_catalog.default.fact_stats[units_sold#3607,store_id#3608] Batched: true, DataFilters: [isnotnull(units_sold#3607), (units_sold#3607 = 70)], Format: CometParquet, Location: InMemoryFileIndex(25 paths)[file:/home/andy/git/apache/apache-spark-3.5.6/sql/core/spark-warehous..., PartitionFilters: [isnotnull(store_id#3608)], PushedFilters: [IsNotNull(units_sold), EqualTo(units_sold,70)], ReadSchema: struct<units_sold:int>
getOrElseUpdate with CometColumnarExchange hashpartitioning(store_id#3608, 5), ENSURE_REQUIREMENTS, [plan_id=580]
+- CometProject [store_id#3608], [store_id#3608]
+- CometFilter [units_sold#3607, store_id#3608], (isnotnull(units_sold#3607) AND (units_sold#3607 = 70))
+- CometScan parquet spark_catalog.default.fact_stats[units_sold#3607,store_id#3608] Batched: true, DataFilters: [isnotnull(units_sold#3607), (units_sold#3607 = 70)], Format: CometParquet, Location: InMemoryFileIndex(25 paths)[file:/home/andy/git/apache/apache-spark-3.5.6/sql/core/spark-warehous..., PartitionFilters: [isnotnull(store_id#3608)], PushedFilters: [IsNotNull(units_sold), EqualTo(units_sold,70)], ReadSchema: struct<units_sold:int>
NO REUSE
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org