You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "omerhadari (via GitHub)" <gi...@apache.org> on 2025/02/03 14:43:09 UTC
[I] Warehouse path is always required for Glue catalog, even when there isn't one [iceberg-rust]
omerhadari opened a new issue, #941:
URL: https://github.com/apache/iceberg-rust/issues/941
The [warehouse](https://github.com/apache/iceberg-rust/blob/c754e27f752a9e2f86180a9b40f69a3d4ab2dae9/crates/catalog/glue/src/catalog.rs#L46) config prop for Glue catalog is required, but it isn't always known or required. The Glue catalog ID is enough in order to resolve metadata locations.
Moreover - I don't think this path is used for anything but protocol (FileIO type) inference? I was able to work around this issue by passing the correct Glue catalog ID (which is the AWS account ID) and `s3://` as the warehouse, with no issue.
In my opinion `warehouse` should be replaced with something like `FileIO` type and only the Glue catalog ID should be required, but for backwards compatibility not passing warehouse should at least be allowed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Warehouse path is always required for Glue catalog, even when there isn't one [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on issue #941:
URL: https://github.com/apache/iceberg-rust/issues/941#issuecomment-2867994191
I think this makes sense and making the warehouse path optional matches the iceberg-java behavior. We should probably fix this while implementing the new catalog builder: https://github.com/apache/iceberg-rust/issues/1259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Warehouse path is always required for Glue catalog, even when there isn't one [iceberg-rust]
Posted by "David-Banquet (via GitHub)" <gi...@apache.org>.
David-Banquet commented on issue #941:
URL: https://github.com/apache/iceberg-rust/issues/941#issuecomment-5997952913
A status check on `main`, since the code has moved since this was opened.
The original concern is mostly gone: the Glue catalog no longer infers its FileIO from `warehouse`, it builds it from the storage factory and the catalog properties. The only remaining use of `warehouse` is the default table location in `create_table`, and only when the Glue database has no `LocationUri` (`get_default_table_location` in `utils.rs`).
What still differs from Java is when the check happens. `GlueCatalogBuilder::load` fails with "Catalog warehouse is required" before any call to Glue. Java's `GlueCatalog` accepts a missing warehouse and only throws in `defaultWarehouseLocation`, when a table is created without a location in a database that has no `LocationUri`. So a read-only client, or one that always creates tables in databases with a location, works in Java without a warehouse and is refused in Rust.
@blackmwk I see #3102 keeps the fail-fast validation on purpose. Would you take a follow-up that moves the check to `create_table`, matching Java? If so I can open it once #3102 is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Warehouse path is always required for Glue catalog, even when there isn't one [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #941:
URL: https://github.com/apache/iceberg-rust/issues/941#issuecomment-3832361861
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Warehouse path is always required for Glue catalog, even when there isn't one [iceberg-rust]
Posted by "David-Banquet (via GitHub)" <gi...@apache.org>.
David-Banquet commented on issue #941:
URL: https://github.com/apache/iceberg-rust/issues/941#issuecomment-6131305779
@blackmwk #3102 was closed by the stale bot last night. Do you plan to reopen it? If not, I can open the follow-up for this issue against main now: keep the `warehouse` property optional in `GlueCatalogBuilder::load` and only fail in `create_table` when the database has no `LocationUri`, as Java does. It is a small change and I can rebase it on #3102 if that comes back first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Warehouse path is always required for Glue catalog, even when there isn't one [iceberg-rust]
Posted by "chasehaddleton (via GitHub)" <gi...@apache.org>.
chasehaddleton commented on issue #941:
URL: https://github.com/apache/iceberg-rust/issues/941#issuecomment-2960176541
I believe there is an entire class of bugs caused by the current behavior. For example, if you pass `s3` as the warehouse, but the catalog contains tables with both `s3` and `s3a` files, then it erroneously errors saying that protocols do not match (even though it is immaterial). Moving to a `FileIOType` could solve this, assuming it's paired with a set of "valid protocols for a type".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Warehouse path is always required for Glue catalog, even when there isn't one [iceberg-rust]
Posted by "lliangyu-lin (via GitHub)" <gi...@apache.org>.
lliangyu-lin commented on issue #941:
URL: https://github.com/apache/iceberg-rust/issues/941#issuecomment-3153368973
Another issue is that when loading the table from glue, the table is [read using the catalog file io](https://github.com/apache/iceberg-rust/blob/main/crates/catalog/glue/src/catalog.rs#L464). So if catalog file io is set to S3 with setting the warehouse, but the table loaded is stored another storage, it will continue to use the S3 file io to read the metadata. I think the file io should be inferred during create table and load table instead of initialized with the catalog...
Reference in python
* https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/__init__.py#L348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Warehouse path is always required for Glue catalog, even when there isn't one [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on issue #941:
URL: https://github.com/apache/iceberg-rust/issues/941#issuecomment-3832531874
Don't close this, it's a but.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Warehouse path is always required for Glue catalog, even when there isn't one [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on issue #941:
URL: https://github.com/apache/iceberg-rust/issues/941#issuecomment-6142454904
Hi, @David-Banquet Yes, I'll reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Warehouse path is always required for Glue catalog, even when there isn't one [iceberg-rust]
Posted by "David-Banquet (via GitHub)" <gi...@apache.org>.
David-Banquet commented on issue #941:
URL: https://github.com/apache/iceberg-rust/issues/941#issuecomment-6162621310
Thanks. I'll wait for #3102 to be back and open the follow-up on top of it: keep `warehouse` optional in `GlueCatalogBuilder::load` and only fail in `create_table` when the database has no `LocationUri`. If you'd rather fold that into #3102 directly, that works for me too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org