You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/03/13 22:24:08 UTC
[PR] fix: detect all DPP forms in native_datafusion scan fallback [datafusion-comet]
andygrove opened a new pull request, #3694:
URL: https://github.com/apache/datafusion-comet/pull/3694
## Which issue does this PR close?
Closes #3313.
## Rationale for this change
`CometNativeScan.isDynamicPruningFilter` only checked for `PlanExpression[_]` in partition filters to detect Dynamic Partition Pruning (DPP). This catches dynamic DPP (where subqueries are still present), but misses static DPP where Spark has already resolved the pruning expression to a literal wrapped in `DynamicPruningExpression(Literal.TrueLiteral)`. When static DPP slips through, `native_datafusion` replaces the scan but cannot properly handle DPP-related plan structures, causing test failures ("static scan metrics", "explain formatted - check presence of subquery in case of DPP") and potential runtime errors.
## What changes are included in this PR?
- **`CometNativeScan.scala`**: Updated `isDynamicPruningFilter` to also check for `DynamicPruningExpression` at the top level, catching both dynamic and static DPP. This ensures `native_datafusion` always falls back when any form of DPP is present in partition filters.
- **`CometExecSuite.scala`**: Added test "DPP fallback with native_datafusion scan" that verifies `native_datafusion` correctly falls back when DPP is detected (even with `COMET_DPP_FALLBACK_ENABLED=false`, since detection happens independently in `CometNativeScan.isSupported`).
- **`dev/diffs/3.5.8.diff`**: Removed `IgnoreCometNativeDataFusion` skip tags for issue #3313 from "static scan metrics" and "explain formatted - check presence of subquery in case of DPP" tests, since the fix makes these tests pass with `native_datafusion`. Diff regenerated via the documented Spark clone workflow.
## How are these changes tested?
- New unit test "DPP fallback with native_datafusion scan" in `CometExecSuite` validates the fix
- Existing "DPP fallback" test continues to pass
- Spark SQL tests previously skipped via `IgnoreCometNativeDataFusion` tags should now pass in CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: enable more Spark SQL tests for `native_datafusion` (`DynamicPartitionPruningSuite` / `ExplainSuite`) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3694:
URL: https://github.com/apache/datafusion-comet/pull/3694
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: enable more Spark SQL tests for `native_datafusion` (`DynamicPartitionPruningSuite` / `ExplainSuite`) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3694:
URL: https://github.com/apache/datafusion-comet/pull/3694#issuecomment-4070491258
moving to draft to avoid accidental merge - I need to rebase and merge each of the native_datafusion PRs individually to avoid corrupting the diff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org