You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mbutrovich (via GitHub)" <gi...@apache.org> on 2025/11/07 21:39:58 UTC
[PR] test: upgrade DataFusion 51 and Arrow 57 [iceberg-rust]
mbutrovich opened a new pull request, #1830:
URL: https://github.com/apache/iceberg-rust/pull/1830
DataFusion is preparing for their 51.0.0 release, so this is a test PR to see how CI goes with main on a recent commit that likely reflects what DF 51 will be. Marking as draft since we wouldn't want this merged until crates are available and the community discusses the changes.
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- N/A.
## What changes are included in this PR?
<!--
Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates.
-->
- Bump DF to 51
- Bump Arrow/Parquet to 57
- Page index tests are skipped right now because of API changes in Arrow-rs. I will have to revisit what we can do to fix these since they were relying on a pretty low-level API
- Toolchain bump to support DF's 1.88. I need to understand if this would be within iceberg-rust's MSRV policy
## Are these changes tested?
<!--
Specify what test covers (unit test, integration test, etc.).
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
Creating this to run CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] test: upgrade DataFusion 51 and Arrow 57 [iceberg-rust]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich closed pull request #1830: test: upgrade DataFusion 51 and Arrow 57
URL: https://github.com/apache/iceberg-rust/pull/1830
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] test: upgrade DataFusion 51 and Arrow 57 [iceberg-rust]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #1830:
URL: https://github.com/apache/iceberg-rust/pull/1830#issuecomment-3506304115
i can probably help with the page index changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] test: upgrade DataFusion 51 and Arrow 57 [iceberg-rust]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #1830:
URL: https://github.com/apache/iceberg-rust/pull/1830#discussion_r2508135647
##########
Cargo.toml:
##########
@@ -62,9 +65,9 @@ bytes = "1.10"
chrono = "0.4.41"
clap = { version = "4.5.48", features = ["derive", "cargo"] }
ctor = "0.2.8"
-datafusion = "50"
-datafusion-cli = "50"
-datafusion-sqllogictest = "50"
+datafusion = { git = "https://github.com/apache/datafusion.git", rev = "f32984b2dbf9e5a193c20643ce624167295fbd61" }
Review Comment:
branch-51 didn't exist yet, so I just picked whatever the current commit on main was at the time. I could run it again with DataFusion's branch-51, but not expecting different results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] test: upgrade DataFusion 51 and Arrow 57 [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
dentiny commented on code in PR #1830:
URL: https://github.com/apache/iceberg-rust/pull/1830#discussion_r2507591141
##########
Cargo.toml:
##########
@@ -62,9 +65,9 @@ bytes = "1.10"
chrono = "0.4.41"
clap = { version = "4.5.48", features = ["derive", "cargo"] }
ctor = "0.2.8"
-datafusion = "50"
-datafusion-cli = "50"
-datafusion-sqllogictest = "50"
+datafusion = { git = "https://github.com/apache/datafusion.git", rev = "f32984b2dbf9e5a193c20643ce624167295fbd61" }
Review Comment:
just for my own curiosity, is it because latest stable release datafusion doesn't play with the other dependencies, that we have to use commit sha here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] test: upgrade DataFusion 51 and Arrow 57 [iceberg-rust]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #1830:
URL: https://github.com/apache/iceberg-rust/pull/1830#issuecomment-3527773531
It was green at one point, so I think from DataFusion's standpoint DF 51 is okay. This branch quickly fell behind with merge conflicts. I'll update it once DF 51 crates are available and we want to consider upgrading for real.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org