You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jackye1995 (via GitHub)" <gi...@apache.org> on 2023/02/11 23:23:41 UTC
[GitHub] [iceberg] jackye1995 opened a new issue, #6810: Support table-specific FileIO for Glue and Hive catalog through table-specific catalog override
jackye1995 opened a new issue, #6810:
URL: https://github.com/apache/iceberg/issues/6810
### Feature Request / Improvement
cc @RussellSpitzer @pvary @amogh-jahagirdar @rajarshisarkar @singhpk234
We had some discussion that `FileIO` should technically be specific to each table so that all readers and writers use the same one for each table, but currently it is defined as catalog property and decided by the end user dynamically.
This issue is solved by REST catalog, but technically we can also achieve that in Glue, Hive, or any catalog that supports table parameters. (not the Iceberg table properties, but the ones we use to store `table_type=ICEBERG` and `metadata_location`)
The idea is that user can configure overrides of catalog property for specific tables through the table parameters part of the Glue/Hive metastore.
For example, if the Spark session default FileIO is HadoopFileIO, but I want to use S3FileIO for a specific table, I can update the table's parameters with the catalog properties like `io-impl` and any S3FileIO related configurations, and then we can update code to respect those overrides.
This is technically already happening today even in REST catalog through the `config` part of a `LoadTableResponse`: https://github.com/apache/iceberg/blob/master/open-api/rest-catalog-open-api.yaml#L1701, used in https://github.com/apache/iceberg/blob/b6b9972538ffcbae10b7e80e82cc444254d49103/core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java#L310
In Glue catalog, we are also technically doing table specific catalog properties override even today, for (1) LakeFormation related security configurations, (2) table specific S3 tags, because each table requires different credential provider for the same FIleIO: https://github.com/apache/iceberg/blob/master/aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java#L205-L230
Any thoughts?
### Query engine
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support table-specific FileIO for Glue and Hive catalog through catalog override in table parameters [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6810:
URL: https://github.com/apache/iceberg/issues/6810#issuecomment-1880238883
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6810: Support table-specific FileIO for Glue and Hive catalog through catalog override in table parameters
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6810:
URL: https://github.com/apache/iceberg/issues/6810#issuecomment-1674074145
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support table-specific FileIO for Glue and Hive catalog through catalog override in table parameters [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6810: Support table-specific FileIO for Glue and Hive catalog through catalog override in table parameters
URL: https://github.com/apache/iceberg/issues/6810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org