You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "humengyu2012 (via GitHub)" <gi...@apache.org> on 2023/11/20 14:06:53 UTC
[PR] [Hive] HiveCatalog support hive 1.x [incubator-paimon]
humengyu2012 opened a new pull request, #2354:
URL: https://github.com/apache/incubator-paimon/pull/2354
Add 1.x HiveMetastore support.
Support Method:
`org.apache.hadoop.hive.metastore.RetryingMetaStoreClient#getProxy(org.apache.hadoop.hive.conf.HiveConf, org.apache.hadoop.hive.metastore.HiveMetaHookLoader, java.util.Map<java.lang.String,java.lang.Long>, java.lang.String)`
<img width="946" alt="image" src="https://github.com/apache/incubator-paimon/assets/45056332/96fe8fb4-fe4e-488a-8284-1a39713f459b">
<img width="922" alt="image" src="https://github.com/apache/incubator-paimon/assets/45056332/c2f69a1e-629a-42d1-8f84-ea83e3e02f8b">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Hive] HiveCatalog support hive 1.x [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #2354:
URL: https://github.com/apache/incubator-paimon/pull/2354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Hive] HiveCatalog support hive 1.x [incubator-paimon]
Posted by "humengyu2012 (via GitHub)" <gi...@apache.org>.
humengyu2012 commented on PR #2354:
URL: https://github.com/apache/incubator-paimon/pull/2354#issuecomment-1820529604
> Can you extract a class for client creation? for example `HmsClientFactory`?
>
> And other thing is testing, can we have any testing to cover this modification?
I added a class named `RetryingMetaStoreClientFactory`, can you take a look again? @JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Hive] HiveCatalog support hive 1.x [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2354:
URL: https://github.com/apache/incubator-paimon/pull/2354#discussion_r1400131986
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java:
##########
@@ -551,33 +555,97 @@ private Lock lock(Identifier identifier) {
return Lock.fromCatalog(lock, identifier);
}
- private static final List<Class<?>[]> GET_PROXY_PARAMS =
- Arrays.asList(
+ private static final Map<Class<?>[], HiveMetastoreProxySupplier> PROXY_SUPPLIERS =
+ ImmutableMap.<Class<?>[], HiveMetastoreProxySupplier>builder()
+ // for hive 1.x
+ .put(
+ new Class<?>[] {
+ HiveConf.class,
+ HiveMetaHookLoader.class,
+ ConcurrentHashMap.class,
+ String.class
+ },
+ (getProxyMethod, hiveConf, clientClassName) ->
+ (IMetaStoreClient)
+ getProxyMethod.invoke(
+ null,
+ hiveConf,
+ (HiveMetaHookLoader) (tbl -> null),
+ new ConcurrentHashMap<>(),
+ clientClassName))
// for hive 2.x
- new Class<?>[] {
- HiveConf.class,
- HiveMetaHookLoader.class,
- ConcurrentHashMap.class,
- String.class,
- Boolean.TYPE
- },
+ .put(
+ new Class<?>[] {
+ HiveConf.class,
+ HiveMetaHookLoader.class,
+ ConcurrentHashMap.class,
+ String.class,
+ Boolean.TYPE
+ },
+ (getProxyMethod, hiveConf, clientClassName) ->
+ (IMetaStoreClient)
+ getProxyMethod.invoke(
+ null,
+ hiveConf,
+ (HiveMetaHookLoader) (tbl -> null),
+ new ConcurrentHashMap<>(),
+ clientClassName,
+ true))
// for hive 3.x
- new Class<?>[] {
- Configuration.class,
- HiveMetaHookLoader.class,
- ConcurrentHashMap.class,
- String.class,
- Boolean.TYPE
- });
+ .put(
+ new Class<?>[] {
+ Configuration.class,
+ HiveMetaHookLoader.class,
+ ConcurrentHashMap.class,
+ String.class,
+ Boolean.TYPE
+ },
+ (getProxyMethod, hiveConf, clientClassName) ->
+ (IMetaStoreClient)
+ getProxyMethod.invoke(
+ null,
+ hiveConf,
+ (HiveMetaHookLoader) (tbl -> null),
+ new ConcurrentHashMap<>(),
+ clientClassName,
+ true))
+ .build();
+ // If clientClassName is HiveMetaStoreClient,
Review Comment:
Leave a empty line above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Hive] HiveCatalog support hive 1.x [incubator-paimon]
Posted by "humengyu2012 (via GitHub)" <gi...@apache.org>.
humengyu2012 commented on code in PR #2354:
URL: https://github.com/apache/incubator-paimon/pull/2354#discussion_r1400259910
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java:
##########
@@ -551,33 +555,97 @@ private Lock lock(Identifier identifier) {
return Lock.fromCatalog(lock, identifier);
}
- private static final List<Class<?>[]> GET_PROXY_PARAMS =
- Arrays.asList(
+ private static final Map<Class<?>[], HiveMetastoreProxySupplier> PROXY_SUPPLIERS =
+ ImmutableMap.<Class<?>[], HiveMetastoreProxySupplier>builder()
+ // for hive 1.x
+ .put(
+ new Class<?>[] {
+ HiveConf.class,
+ HiveMetaHookLoader.class,
+ ConcurrentHashMap.class,
+ String.class
+ },
+ (getProxyMethod, hiveConf, clientClassName) ->
+ (IMetaStoreClient)
+ getProxyMethod.invoke(
+ null,
+ hiveConf,
+ (HiveMetaHookLoader) (tbl -> null),
+ new ConcurrentHashMap<>(),
+ clientClassName))
// for hive 2.x
- new Class<?>[] {
- HiveConf.class,
- HiveMetaHookLoader.class,
- ConcurrentHashMap.class,
- String.class,
- Boolean.TYPE
- },
+ .put(
+ new Class<?>[] {
+ HiveConf.class,
+ HiveMetaHookLoader.class,
+ ConcurrentHashMap.class,
+ String.class,
+ Boolean.TYPE
+ },
+ (getProxyMethod, hiveConf, clientClassName) ->
+ (IMetaStoreClient)
+ getProxyMethod.invoke(
+ null,
+ hiveConf,
+ (HiveMetaHookLoader) (tbl -> null),
+ new ConcurrentHashMap<>(),
+ clientClassName,
+ true))
// for hive 3.x
- new Class<?>[] {
- Configuration.class,
- HiveMetaHookLoader.class,
- ConcurrentHashMap.class,
- String.class,
- Boolean.TYPE
- });
+ .put(
+ new Class<?>[] {
+ Configuration.class,
+ HiveMetaHookLoader.class,
+ ConcurrentHashMap.class,
+ String.class,
+ Boolean.TYPE
+ },
+ (getProxyMethod, hiveConf, clientClassName) ->
+ (IMetaStoreClient)
+ getProxyMethod.invoke(
+ null,
+ hiveConf,
+ (HiveMetaHookLoader) (tbl -> null),
+ new ConcurrentHashMap<>(),
+ clientClassName,
+ true))
+ .build();
+ // If clientClassName is HiveMetaStoreClient,
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org