You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/08/29 21:58:58 UTC
[GitHub] [iceberg] kbendick opened a new issue #3050: Use the class loader provided by Flink's updated CatalogFactory to access hadoop configuration etc
kbendick opened a new issue #3050:
URL: https://github.com/apache/iceberg/issues/3050
As of Flink 1.13, a new interface `Factory` has been added to the Table API, to better support the usage of additional catalogs (particularly within Flink SQL): https://github.com/apache/flink/commit/4497e96b96724aee6637f19f64ffc8ba47a9b0ac
We already extend `CatalogFactory`, but in Flink 1.13 `CatalogFactory` was updated to also implement the new `Factory` interface. This provides many benefits including:
- A definitive way to get the ClassLoader to instantiate specific Iceberg catalogs and access the Hadoop configuration etc (class loading in Flink is somewhat complicated and this new utility method `CatalogFactory.Context#getClassLoader` would likely greatly simplify this)
- Remove our dependency on the deprecated `TableFactory` interface, which is still present for backwards compatibility but will be removed eventually
Our [`FlinkCatalogFactory`](https://github.com/apache/iceberg/blob/master/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalogFactory.java#L60) would need to be updated to implement the new `Factory` interface of `CatalogFactory`.
As the upstream Flink `CatalogFactory` seems to maintain backwards compatibility (I.e. it implements both `TableFactory` as before, as well as the new interface `Factory`), this should hopefully be achievable without having to have version specific code. We may, however, need to implement two separate classes to continue to support Flink 1.12.x and lower.
As the updated Flink `CatalogFactory` interface uses Java service provider interface, this seems like it should be doable.
cc @openinx @JingsongLi @stevenzwu for your thoughts. =)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #3050: Update FlinkCatalogFactory to implement the new Factory interface added to Flink's CatalogFactory (and not the deprecated TableFactory interface we currently implement)
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #3050:
URL: https://github.com/apache/iceberg/issues/3050#issuecomment-908590765
I will close this as a duplicate of https://github.com/apache/iceberg/issues/2558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on issue #3050: Update FlinkCatalogFactory to implement the new Factory interface added to Flink's CatalogFactory (and not the deprecated TableFactory interface we currently implement)
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on issue #3050:
URL: https://github.com/apache/iceberg/issues/3050#issuecomment-907941718
@zhangjun0x01 already created PR #2629 for Flink 1.13 upgrade that covers this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick closed issue #3050: Update FlinkCatalogFactory to implement the new Factory interface added to Flink's CatalogFactory (and not the deprecated TableFactory interface we currently implement)
Posted by GitBox <gi...@apache.org>.
kbendick closed issue #3050:
URL: https://github.com/apache/iceberg/issues/3050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #3050: Update FlinkCatalogFactory to implement the new Factory interface added to Flink's CatalogFactory (and not the deprecated TableFactory interface we currently implement)
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #3050:
URL: https://github.com/apache/iceberg/issues/3050#issuecomment-908590215
Oh that's great. Thank you @stevenzwu for the info.
We are hoping to reduce the tight coupling of Hadoop `Configuration` from the catalogs, as some folks have pointed out that it's difficult (or not presently possible as far as they know) to use things like `GlueCatalog` which don't actually require Hadoop in environments that are difficult to configure (specifically when using `AWS Kinesis Data Analytics`).
Further information can be found in this issue: https://github.com/apache/iceberg/issues/3044
Specifically, S3FileIO doesn't actually make use of anything from the `Configurable` interface, so there is an effort to try to make it so that `GlueCatalog` (and also `DynamoDbCatalog`) can be used in environments where Hadoop is not present and that is difficult to configure for them. This will greatly help with that goal. Eventually, it might wind up being beneficial to use the class loader that's provided by the new `Context` interface of Flink 1.13's `CatalogFactory`.
I'll leave the rest of the discussion over in that issue (and possibly bring it up in that PR, though it's something that can be done in a follow up PR once this open one has been merged in).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #3050: Update FlinkCatalogFactory to implement the new Factory interface added to Flink's CatalogFactory (and not the deprecated TableFactory interface we currently implement)
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #3050:
URL: https://github.com/apache/iceberg/issues/3050#issuecomment-908590215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick closed issue #3050: Update FlinkCatalogFactory to implement the new Factory interface added to Flink's CatalogFactory (and not the deprecated TableFactory interface we currently implement)
Posted by GitBox <gi...@apache.org>.
kbendick closed issue #3050:
URL: https://github.com/apache/iceberg/issues/3050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org