You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/04/23 12:27:38 UTC
[PR] test: extend SPARK-43402 plan-match to CometNativeScanExec and retag to #4042 [datafusion-comet]
andygrove opened a new pull request, #4044:
URL: https://github.com/apache/datafusion-comet/pull/4044
## Which issue does this PR close?
Related to #4042.
## Rationale for this change
In #4041 the `SubquerySuite` "SPARK-43402: FileSourceScanExec supports push down data filter with scalar subquery" test was retagged from the umbrella #3321 to the broader #3315 ("plan structure differences"). A closer look shows the first failure in this test is just a missed pattern match: the test collects `FileSourceScanLike` and `CometScanExec` but not `CometNativeScanExec`. After adding `CometNativeScanExec` to the match, that assertion passes, but the test still fails on a genuine behavior gap where the pushed scalar subquery ends up as a plain `Subquery` rather than `ReusedSubqueryExec`, so reuse does not happen. That narrower problem is now tracked in #4042.
## What changes are included in this PR?
Updates `dev/diffs/4.0.1.diff` to:
- Add a `CometNativeScanExec` branch to both pattern matches in the SPARK-43402 test (the `collect` over the executed plan, and the subsequent match extracting `dataFilters`). `CometNativeScanExec` already exposes `dataFilters`, `numFiles`, and `numOutputRows`, so the rest of the assertions work with the added case without further changes.
- Retag the `IgnoreCometNativeDataFusion` reference from #3315 to the narrower #4042.
## How are these changes tested?
Ran the test locally against Spark 4.0.1 with `native_datafusion` in auto scan mode via `ENABLE_COMET=true ENABLE_COMET_ONHEAP=true build/sbt "sql/testOnly org.apache.spark.sql.SubquerySuite -- -z \"SPARK-43402: FileSourceScanExec supports push down data filter with scalar subquery\""`. Confirmed the first assertion (`dataSourceScanExec.size === 1`) now passes with the extended match, and the remaining failure (`was not instance of org.apache.spark.sql.execution.ReusedSubqueryExec`) matches the description in #4042.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: extend SPARK-43402 plan-match to CometNativeScanExec and retag to #4042 [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #4044:
URL: https://github.com/apache/datafusion-comet/pull/4044#issuecomment-4313751894
I converted this PR to draft since I think we have the fix in https://github.com/apache/datafusion-comet/pull/4053.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: extend SPARK-43402 plan-match to CometNativeScanExec and retag to #4042 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #4044:
URL: https://github.com/apache/datafusion-comet/pull/4044#issuecomment-4373866790
This PR is no longer needed. The test was enabled in https://github.com/apache/datafusion-comet/pull/4053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: extend SPARK-43402 plan-match to CometNativeScanExec and retag to #4042 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed pull request #4044: test: extend SPARK-43402 plan-match to CometNativeScanExec and retag to #4042
URL: https://github.com/apache/datafusion-comet/pull/4044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: extend SPARK-43402 plan-match to CometNativeScanExec and retag to #4042 [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #4044:
URL: https://github.com/apache/datafusion-comet/pull/4044#issuecomment-4306544536
I'm gonna try to open a PR to fix #4042 today (testing now), if you want to hold off on resolving the diff merge conflict for now. I will update if it's not looking good for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org