You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/03/27 21:36:20 UTC
[PR] feat: enable native Iceberg reader by default [datafusion-comet]
andygrove opened a new pull request, #3819:
URL: https://github.com/apache/datafusion-comet/pull/3819
## Which issue does this PR close?
Closes #.
## Rationale for this change
The native Iceberg reader has matured enough to be enabled by default, removing the need for users to explicitly opt in via configuration.
## What changes are included in this PR?
- Change default value of `spark.comet.scan.icebergNative.enabled` from `false` to `true`
- Update user-guide Iceberg documentation to reflect the new default and remove the config from shell examples
- Update benchmark README to note the feature is enabled by default
## How are these changes tested?
Existing Iceberg test suites cover the native reader. This change only flips the default; no new behavior is introduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: enable native Iceberg reader by default [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #3819:
URL: https://github.com/apache/datafusion-comet/pull/3819#issuecomment-4260898210
Upmerged main and marking Ready for Review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: enable native Iceberg reader by default [datafusion-comet]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #3819:
URL: https://github.com/apache/datafusion-comet/pull/3819#discussion_r3020718907
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -140,7 +140,7 @@ object CometConf extends ShimCometConf {
"Iceberg tables are read directly through native execution, bypassing Spark's " +
"DataSource V2 API for better performance.")
Review Comment:
https://datafusion.apache.org/comet/user-guide/latest/iceberg.html says that Iceberg support is experimental. Some other config entries mention that they are experimental.
```suggestion
"DataSource V2 API for better performance. This feature is experimental.")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: enable native Iceberg reader by default [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3819:
URL: https://github.com/apache/datafusion-comet/pull/3819#discussion_r3094388686
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -140,7 +140,7 @@ object CometConf extends ShimCometConf {
"Iceberg tables are read directly through native execution, bypassing Spark's " +
"DataSource V2 API for better performance.")
Review Comment:
On second thoughts, I think we want to remove some of the experimental language from the iceberg page in the user docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: enable native Iceberg reader by default [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3819:
URL: https://github.com/apache/datafusion-comet/pull/3819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: enable native Iceberg reader by default [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #3819:
URL: https://github.com/apache/datafusion-comet/pull/3819#issuecomment-4163406255
I would say this issue is blocking for this PR: https://github.com/apache/datafusion-comet/issues/3856.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: enable native Iceberg reader by default [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3819:
URL: https://github.com/apache/datafusion-comet/pull/3819#discussion_r3094014429
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -140,7 +140,7 @@ object CometConf extends ShimCometConf {
"Iceberg tables are read directly through native execution, bypassing Spark's " +
"DataSource V2 API for better performance.")
Review Comment:
Thanks @martin-g. I will create a separate PR today to update the configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: enable native Iceberg reader by default [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3819:
URL: https://github.com/apache/datafusion-comet/pull/3819#issuecomment-4273681806
@mbutrovich Could you approve this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org