You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/01/29 01:09:25 UTC
[PR] Add fallback checks in nativeDataFusionScan for unsupported features [datafusion-comet]
andygrove opened a new pull request, #3322:
URL: https://github.com/apache/datafusion-comet/pull/3322
## Summary
- Add fallback checks in `CometNativeScan.isSupported()` for metadata columns, Parquet field ID reads, bucketed scans, and row index generation
- When these features are detected, the native DataFusion scan falls back to `native_iceberg_compat` in auto mode
- Covers issues #3312, #3316, #3317, #3319
## Test plan
- [ ] Run Spark SQL tests in auto scan mode to confirm fallback behavior
- [ ] Verify explain output shows fallback reasons
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add fallback checks in nativeDataFusionScan for unsupported features [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed pull request #3322: feat: Add fallback checks in nativeDataFusionScan for unsupported features
URL: https://github.com/apache/datafusion-comet/pull/3322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add fallback checks in nativeDataFusionScan for unsupported features [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #3322:
URL: https://github.com/apache/datafusion-comet/pull/3322#discussion_r2746950965
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -77,6 +78,18 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with Logging {
withInfo(scanExec, "Full native scan disabled because ignoreMissingFiles enabled")
}
+ if (scanExec.fileConstantMetadataColumns.nonEmpty) {
+ withInfo(scanExec, "Native DataFusion scan does not support metadata columns")
+ }
+
+ if (scanExec.bucketedScan) {
+ withInfo(scanExec, "Native DataFusion scan does not support bucketed scans")
Review Comment:
Yes it does, I think we just fail the Spark SQL test for not formatting the explain output in the same way. I'm confused how this PR is passing the test that #2888 added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add fallback checks in nativeDataFusionScan for unsupported features [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3322:
URL: https://github.com/apache/datafusion-comet/pull/3322#issuecomment-3814813111
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/3322?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `8 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 20.77%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`9e7ab2c`](https://app.codecov.io/gh/apache/datafusion-comet/commit/9e7ab2c2004ecb79ebe30133837250f25b7b3ba3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 905 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/3322?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../apache/comet/serde/operator/CometNativeScan.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3322?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fserde%2Foperator%2FCometNativeScan.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9zZXJkZS9vcGVyYXRvci9Db21ldE5hdGl2ZVNjYW4uc2NhbGE=) | 0.00% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #3322 +/- ##
=============================================
- Coverage 56.12% 20.77% -35.35%
+ Complexity 976 496 -480
=============================================
Files 119 175 +56
Lines 11743 16175 +4432
Branches 2251 2686 +435
=============================================
- Hits 6591 3361 -3230
- Misses 4012 12263 +8251
+ Partials 1140 551 -589
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/3322?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add fallback checks in nativeDataFusionScan for unsupported features [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3322:
URL: https://github.com/apache/datafusion-comet/pull/3322#discussion_r2764777877
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -77,6 +78,18 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with Logging {
withInfo(scanExec, "Full native scan disabled because ignoreMissingFiles enabled")
}
+ if (scanExec.fileConstantMetadataColumns.nonEmpty) {
+ withInfo(scanExec, "Native DataFusion scan does not support metadata columns")
+ }
+
+ if (scanExec.bucketedScan) {
+ withInfo(scanExec, "Native DataFusion scan does not support bucketed scans")
Review Comment:
See https://github.com/apache/datafusion-comet/pull/3392 for Spark SQL test updates related to bucketed scans. I will remove this fallback from this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add fallback checks in nativeDataFusionScan for unsupported features [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #3322:
URL: https://github.com/apache/datafusion-comet/pull/3322#discussion_r2746950965
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -77,6 +78,18 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with Logging {
withInfo(scanExec, "Full native scan disabled because ignoreMissingFiles enabled")
}
+ if (scanExec.fileConstantMetadataColumns.nonEmpty) {
+ withInfo(scanExec, "Native DataFusion scan does not support metadata columns")
+ }
+
+ if (scanExec.bucketedScan) {
+ withInfo(scanExec, "Native DataFusion scan does not support bucketed scans")
Review Comment:
Yes it does, I think we just fail the Spark SQL test for not formatting the explain output in the same way. I'm confused how we're still passing the test that #2888 added in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add fallback checks in nativeDataFusionScan for unsupported features [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #3322:
URL: https://github.com/apache/datafusion-comet/pull/3322#discussion_r2746950965
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -77,6 +78,18 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with Logging {
withInfo(scanExec, "Full native scan disabled because ignoreMissingFiles enabled")
}
+ if (scanExec.fileConstantMetadataColumns.nonEmpty) {
+ withInfo(scanExec, "Native DataFusion scan does not support metadata columns")
+ }
+
+ if (scanExec.bucketedScan) {
+ withInfo(scanExec, "Native DataFusion scan does not support bucketed scans")
Review Comment:
Yes it does, I think we just fail the Spark SQL test for not formatting the explain output i the same way. I'm confused how we're still passing the test that #2888 added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add fallback checks in nativeDataFusionScan for unsupported features [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3322:
URL: https://github.com/apache/datafusion-comet/pull/3322#issuecomment-3853548295
I have now incorporated these changes into https://github.com/apache/datafusion-comet/pull/3393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add fallback checks in nativeDataFusionScan for unsupported features [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #3322:
URL: https://github.com/apache/datafusion-comet/pull/3322#discussion_r2746950965
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -77,6 +78,18 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with Logging {
withInfo(scanExec, "Full native scan disabled because ignoreMissingFiles enabled")
}
+ if (scanExec.fileConstantMetadataColumns.nonEmpty) {
+ withInfo(scanExec, "Native DataFusion scan does not support metadata columns")
+ }
+
+ if (scanExec.bucketedScan) {
+ withInfo(scanExec, "Native DataFusion scan does not support bucketed scans")
Review Comment:
Yes it does, I think we just fail the Spark SQL test for not formatting the explain output i the same way. I'm confused how we're still passing the test that #2888 added in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add fallback checks in nativeDataFusionScan for unsupported features [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3322:
URL: https://github.com/apache/datafusion-comet/pull/3322#discussion_r2746963873
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -77,6 +78,18 @@ object CometNativeScan extends CometOperatorSerde[CometScanExec] with Logging {
withInfo(scanExec, "Full native scan disabled because ignoreMissingFiles enabled")
}
+ if (scanExec.fileConstantMetadataColumns.nonEmpty) {
+ withInfo(scanExec, "Native DataFusion scan does not support metadata columns")
+ }
+
+ if (scanExec.bucketedScan) {
+ withInfo(scanExec, "Native DataFusion scan does not support bucketed scans")
Review Comment:
I see. Thanks. I will move this to draft for now. Here is the related issue for the Spark SQL test failures, and yes, it looks like we need to update the diff for these. https://github.com/apache/datafusion-comet/issues/3319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org