You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "CTTY (via GitHub)" <gi...@apache.org> on 2026/02/05 20:41:25 UTC
[PR] feat!(storage): Integration Storage trait with FileIO [iceberg-rust]
CTTY opened a new pull request, #2116:
URL: https://github.com/apache/iceberg-rust/pull/2116
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #2058
## What changes are included in this PR?
- Update `FileIO` to hold `dyn Storage` instead of `OpenDalStorage`
- Update `FileIOBuilder` accordingly
- Removed `Extensions` from `FileIO`
- Add `with_storage_factory` in `CatalogBuilder` and update all implementations accordingly
- Add an optional `dyn StorageFactory` to `IcebergTableProviderFactory`
- Update other FileIOBuilder usages accordingly
<!--
Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates.
-->
## Are these changes tested?
Mostly rely on the existing tests, added some uts for FileIOBuilder
<!--
Specify what test covers (unit test, integration test, etc.).
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!(storage): Integration Storage trait with FileIO [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #2116:
URL: https://github.com/apache/iceberg-rust/pull/2116#discussion_r2875715063
##########
crates/catalog/glue/src/catalog.rs:
##########
@@ -182,9 +198,11 @@ impl GlueCatalog {
let client = aws_sdk_glue::Client::new(&sdk_config);
- let file_io = FileIO::from_path(&config.warehouse)?
+ // Use provided factory or default to LocalFsStorageFactory
+ let factory = storage_factory.unwrap_or_else(|| Arc::new(LocalFsStorageFactory));
Review Comment:
Why it's not ideal? I think it's would be odd if the core iceberg crate depends on `iceberg-storage-opendal`, but it would be fine for other crates. Also we could use feature flag to allow user to stop depending on opendal if they don't want.
##########
bindings/python/src/datafusion_table_provider.rs:
##########
@@ -30,6 +30,29 @@ use pyo3::types::PyCapsule;
use crate::runtime::runtime;
+/// Parse the scheme from a URL and return the appropriate StorageFactory.
+fn storage_factory_from_path(path: &str) -> PyResult<Arc<dyn StorageFactory>> {
Review Comment:
Please create an issue to track this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!(storage): Integration Storage trait with FileIO [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2116:
URL: https://github.com/apache/iceberg-rust/pull/2116#discussion_r2875730857
##########
bindings/python/src/datafusion_table_provider.rs:
##########
@@ -30,6 +30,29 @@ use pyo3::types::PyCapsule;
use crate::runtime::runtime;
+/// Parse the scheme from a URL and return the appropriate StorageFactory.
+fn storage_factory_from_path(path: &str) -> PyResult<Arc<dyn StorageFactory>> {
Review Comment:
Sounds good. Created an issue: https://github.com/apache/iceberg-rust/issues/2200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!(storage): Integration Storage trait with FileIO [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2116:
URL: https://github.com/apache/iceberg-rust/pull/2116#discussion_r2875868726
##########
crates/catalog/glue/src/catalog.rs:
##########
@@ -182,9 +198,11 @@ impl GlueCatalog {
let client = aws_sdk_glue::Client::new(&sdk_config);
- let file_io = FileIO::from_path(&config.warehouse)?
+ // Use provided factory or default to LocalFsStorageFactory
+ let factory = storage_factory.unwrap_or_else(|| Arc::new(LocalFsStorageFactory));
Review Comment:
Updated glue to fall back to S3 storage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!(storage): Integration Storage trait with FileIO [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2116:
URL: https://github.com/apache/iceberg-rust/pull/2116#discussion_r2875727388
##########
crates/catalog/glue/src/catalog.rs:
##########
@@ -182,9 +198,11 @@ impl GlueCatalog {
let client = aws_sdk_glue::Client::new(&sdk_config);
- let file_io = FileIO::from_path(&config.warehouse)?
+ // Use provided factory or default to LocalFsStorageFactory
+ let factory = storage_factory.unwrap_or_else(|| Arc::new(LocalFsStorageFactory));
Review Comment:
Make sense to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!(storage): Integration Storage trait with FileIO [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk merged PR #2116:
URL: https://github.com/apache/iceberg-rust/pull/2116
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!(storage): Integration Storage trait with FileIO [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2116:
URL: https://github.com/apache/iceberg-rust/pull/2116#discussion_r2875727388
##########
crates/catalog/glue/src/catalog.rs:
##########
@@ -182,9 +198,11 @@ impl GlueCatalog {
let client = aws_sdk_glue::Client::new(&sdk_config);
- let file_io = FileIO::from_path(&config.warehouse)?
+ // Use provided factory or default to LocalFsStorageFactory
+ let factory = storage_factory.unwrap_or_else(|| Arc::new(LocalFsStorageFactory));
Review Comment:
Make sense to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!(storage): Integration Storage trait with FileIO [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2116:
URL: https://github.com/apache/iceberg-rust/pull/2116#discussion_r2875869472
##########
crates/catalog/hms/src/catalog.rs:
##########
@@ -194,9 +209,11 @@ impl HmsCatalog {
.build(),
};
- let file_io = FileIO::from_path(&config.warehouse)?
+ // Use provided factory or default to LocalFsStorageFactory
+ let factory = storage_factory.unwrap_or_else(|| Arc::new(LocalFsStorageFactory));
Review Comment:
Updated hms catalog to throw error if storage factory is not configured
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat!(storage): Integration Storage trait with FileIO [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2116:
URL: https://github.com/apache/iceberg-rust/pull/2116#discussion_r2771276184
##########
bindings/python/src/datafusion_table_provider.rs:
##########
@@ -30,6 +30,29 @@ use pyo3::types::PyCapsule;
use crate::runtime::runtime;
+/// Parse the scheme from a URL and return the appropriate StorageFactory.
+fn storage_factory_from_path(path: &str) -> PyResult<Arc<dyn StorageFactory>> {
Review Comment:
We probably need to add storage wrapper for user to better configure the storage backend they want to use. Right now it's hard wired to opendal for simplicity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org