You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mbutrovich (via GitHub)" <gi...@apache.org> on 2026/04/22 17:27:12 UTC
[PR] test: add non-AQE DPP edge case coverage for native Parquet scans [datafusion-comet]
mbutrovich opened a new pull request, #4037:
URL: https://github.com/apache/datafusion-comet/pull/4037
## Which issue does this PR close?
Related to #4033. Improves test coverage for #4011.
## Rationale for this change
While working on AQE DPP broadcast reuse for Iceberg (#4033), we identified several non-AQE DPP scenarios that #4011 supports but doesn't explicitly test: multiple broadcast joins, broadcast exchange fallback, empty DPP results, and dual partition filter resolution. Adding these tests strengthens confidence in #4011's implementation and documents edge cases for future work on V1 Parquet AQE DPP (#3510).
## What changes are included in this PR?
Four new non-AQE DPP tests in `CometExecSuite`:
- **non-AQE DPP with two separate broadcast joins**: Fact table partitioned by two columns, each joined to a different dimension table. Verifies `convertSubqueryBroadcasts` handles multiple independent DPP filters.
- **non-AQE DPP fallback when broadcast exchange is not Comet**: Disables Comet BHJ/broadcast exchange so `convertSubqueryBroadcasts` encounters a Spark `BroadcastExchangeExec` and correctly skips conversion. Verifies correct results via Spark's standard subquery path.
- **non-AQE DPP with empty broadcast result**: Dimension filter matches no rows, DPP prunes all partitions. Verifies empty result without errors.
- **non-AQE DPP resolves both outer and inner partition filters**: `CometNativeScanExec.partitionFilters` and `CometScanExec.partitionFilters` contain different `InSubqueryExec` instances. Both must be resolved for partition selection to work. Verifies correct results, which requires both filter sets to be resolved.
## How are these changes tested?
The PR is entirely tests. All four tests use `AQE=false` to exercise #4011's non-AQE DPP code paths (`PlanDynamicPruningFilters`, `convertSubqueryBroadcasts`, `ReuseExchangeAndSubquery`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: add non-AQE DPP edge case coverage for native Parquet scans [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich merged PR #4037:
URL: https://github.com/apache/datafusion-comet/pull/4037
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org