You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linfey90 (via GitHub)" <gi...@apache.org> on 2023/03/20 07:34:47 UTC
[GitHub] [hudi] linfey90 opened a new issue, #8240: [SUPPORT] dynamic catalog
linfey90 opened a new issue, #8240:
URL: https://github.com/apache/hudi/issues/8240
We need to implement a multi-catalog dynamic change, But much of hudi's code has hadoop parameters that are captured through the environment. In particular, the protocol corresponding to s3a, such as minio, needs to use secret key, value, etc., which will cause many problems. What I do now is transfer these environment parameters all the way down through the hiveConf in the HoodieHiveCatalog to FileSystem. To be honest, hudi should not support multiple different catalogs at this point, and needs to consider overall refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8240: [SUPPORT] dynamic catalog
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8240:
URL: https://github.com/apache/hudi/issues/8240#issuecomment-1573062029
The Flink SQL already supports passing hadoop options with `hadoop.` prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] linfey90 closed issue #8240: [SUPPORT] dynamic catalog
Posted by "linfey90 (via GitHub)" <gi...@apache.org>.
linfey90 closed issue #8240: [SUPPORT] dynamic catalog
URL: https://github.com/apache/hudi/issues/8240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #8240: [SUPPORT] dynamic catalog
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #8240:
URL: https://github.com/apache/hudi/issues/8240#issuecomment-1478983884
@linfey90 Can you elaborate your use case for the dynamic catalog? Also, what kind of class hierarchy and refactoring are you proposing, let's discuss in this thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] linfey90 commented on issue #8240: [SUPPORT] dynamic catalog
Posted by "linfey90 (via GitHub)" <gi...@apache.org>.
linfey90 commented on issue #8240:
URL: https://github.com/apache/hudi/issues/8240#issuecomment-1571398600
> @linfey90 Can you elaborate your use case for the dynamic catalog? Also, what kind of class hierarchy and refactoring are you proposing, let's discuss in this thread.
Our service implementation is a hive with multiple stores such as hdfs, minio, and obs, so hadoop parameters cannot be obtained through environment variables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org