You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2023/01/10 11:21:23 UTC
[GitHub] [flink-table-store] tsreaper opened a new pull request, #475: [FLINK-30620] Table Store Hive catalog supports specifying custom Hive metastore client
tsreaper opened a new pull request, #475:
URL: https://github.com/apache/flink-table-store/pull/475
Currently Hive metastore client class is hard coded in HiveCatalog, however users may want to specify custom Hive metastore client to read and write Hive compliant storage.
This PR is tested by hand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #475: [FLINK-30620] Table Store Hive catalog supports specifying custom Hive metastore client
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on code in PR #475:
URL: https://github.com/apache/flink-table-store/pull/475#discussion_r1067684753
##########
flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalog.java:
##########
@@ -408,12 +411,16 @@ private Lock lock(ObjectPath tablePath) {
return Lock.fromCatalog(lock, tablePath);
}
- static IMetaStoreClient createClient(HiveConf hiveConf) {
+ static IMetaStoreClient createClient(HiveConf hiveConf, String clientClassName) {
IMetaStoreClient client;
try {
client =
RetryingMetaStoreClient.getProxy(
- hiveConf, tbl -> null, HiveMetaStoreClient.class.getName());
+ hiveConf,
Review Comment:
We can verify multiple hive versions. I don't know if it is good to all supported versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #475: [FLINK-30620] Table Store Hive catalog supports specifying custom Hive metastore client
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on code in PR #475:
URL: https://github.com/apache/flink-table-store/pull/475#discussion_r1067684246
##########
docs/content/docs/how-to/creating-catalogs.md:
##########
@@ -31,6 +31,8 @@ Table Store catalogs currently support two types of metastores:
* `filesystem` metastore (default), which stores both metadata and table files in filesystems.
* `hive` metastore, which additionally stores metadata in Hive metastore. Users can directly access the tables from Hive.
+See [CatalogOptions]({{< ref "docs/maintenance/configurations#catalogoptions" >}}) for detailed options when creating a catalog.
Review Comment:
remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #475: [FLINK-30620] Table Store Hive catalog supports specifying custom Hive metastore client
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on code in PR #475:
URL: https://github.com/apache/flink-table-store/pull/475#discussion_r1066589722
##########
docs/layouts/shortcodes/generated/hive_catalog_configuration.html:
##########
@@ -0,0 +1,18 @@
+<table class="configuration table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Key</th>
+ <th class="text-left" style="width: 15%">Default</th>
+ <th class="text-left" style="width: 10%">Type</th>
+ <th class="text-left" style="width: 55%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>metastore.client.class</h5></td>
+ <td style="word-wrap: break-word;">"org.apache.flink.table.store.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient"</td>
Review Comment:
Maybe we don't need this documentation. I think this option should be inner.
It is not so good to expose a shaded interface. (`IMetaStoreClient` is shaded)
##########
flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalogFactory.java:
##########
@@ -41,13 +40,21 @@ public String identifier() {
public Catalog create(String warehouse, Configuration options) {
String uri =
Preconditions.checkNotNull(
- options.get(URI),
- URI.key() + " must be set for table store " + IDENTIFIER + " catalog");
+ options.get(CatalogOptions.URI),
+ CatalogOptions.URI.key()
+ + " must be set for table store "
+ + IDENTIFIER
+ + " catalog");
+
org.apache.hadoop.conf.Configuration hadoopConfig =
new org.apache.hadoop.conf.Configuration();
options.toMap().forEach(hadoopConfig::set);
+ hadoopConfig.set(
Review Comment:
We can add test for here. May a mock `IMetaStoreClient` just implement `getAllDatabases` to verify.
##########
flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalogOptions.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.hive;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+
+import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
+
+/** Specific config options for {@link HiveCatalogFactory}. */
+public class HiveCatalogOptions {
+
+ public static final ConfigOption<String> METASTORE_CLIENT_CLASS =
+ ConfigOptions.key("metastore.client.class")
+ .stringType()
+ .defaultValue(HiveMetaStoreClient.class.getName())
+ .withDescription("Class name of Hive metastore client");
Review Comment:
NOTE it should be shaded?
##########
flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalog.java:
##########
@@ -63,6 +64,9 @@
/** A catalog implementation for Hive. */
public class HiveCatalog extends AbstractCatalog {
+ public static final String INTERNAL_METASTORE_CLIENT_CLASS =
Review Comment:
Maybe we don't need this key.
We can just pass class by the constructor of `HiveCatalog`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a diff in pull request #475: [FLINK-30620] Table Store Hive catalog supports specifying custom Hive metastore client
Posted by GitBox <gi...@apache.org>.
tsreaper commented on code in PR #475:
URL: https://github.com/apache/flink-table-store/pull/475#discussion_r1067702513
##########
docs/content/docs/how-to/creating-catalogs.md:
##########
@@ -31,6 +31,8 @@ Table Store catalogs currently support two types of metastores:
* `filesystem` metastore (default), which stores both metadata and table files in filesystems.
* `hive` metastore, which additionally stores metadata in Hive metastore. Users can directly access the tables from Hive.
+See [CatalogOptions]({{< ref "docs/maintenance/configurations#catalogoptions" >}}) for detailed options when creating a catalog.
Review Comment:
I think this is a necessary information. This change is not related to Hive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a diff in pull request #475: [FLINK-30620] Table Store Hive catalog supports specifying custom Hive metastore client
Posted by GitBox <gi...@apache.org>.
tsreaper commented on code in PR #475:
URL: https://github.com/apache/flink-table-store/pull/475#discussion_r1068851515
##########
flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalog.java:
##########
@@ -408,12 +411,16 @@ private Lock lock(ObjectPath tablePath) {
return Lock.fromCatalog(lock, tablePath);
}
- static IMetaStoreClient createClient(HiveConf hiveConf) {
+ static IMetaStoreClient createClient(HiveConf hiveConf, String clientClassName) {
IMetaStoreClient client;
try {
client =
RetryingMetaStoreClient.getProxy(
- hiveConf, tbl -> null, HiveMetaStoreClient.class.getName());
+ hiveConf,
Review Comment:
I've tested Hive 2.1, 2.2, 2.3 and 3.1. They are all supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #475: [FLINK-30620] Table Store Hive catalog supports specifying custom Hive metastore client
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on code in PR #475:
URL: https://github.com/apache/flink-table-store/pull/475#discussion_r1066684828
##########
flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalogOptions.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.hive;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+
+import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
+
+/** Specific config options for {@link HiveCatalogFactory}. */
+public class HiveCatalogOptions {
+
+ public static final ConfigOption<String> METASTORE_CLIENT_CLASS =
+ ConfigOptions.key("metastore.client.class")
+ .stringType()
+ .defaultValue(HiveMetaStoreClient.class.getName())
+ .withDescription("Class name of Hive metastore client");
Review Comment:
`HiveMetaStoreClient` have been shaded too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a diff in pull request #475: [FLINK-30620] Table Store Hive catalog supports specifying custom Hive metastore client
Posted by GitBox <gi...@apache.org>.
tsreaper commented on code in PR #475:
URL: https://github.com/apache/flink-table-store/pull/475#discussion_r1066610241
##########
flink-table-store-hive/flink-table-store-hive-catalog/src/main/java/org/apache/flink/table/store/hive/HiveCatalogOptions.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.hive;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+
+import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
+
+/** Specific config options for {@link HiveCatalogFactory}. */
+public class HiveCatalogOptions {
+
+ public static final ConfigOption<String> METASTORE_CLIENT_CLASS =
+ ConfigOptions.key("metastore.client.class")
+ .stringType()
+ .defaultValue(HiveMetaStoreClient.class.getName())
+ .withDescription("Class name of Hive metastore client");
Review Comment:
Not necessary. If a user has his own client, it is very likely not shaded. Only our default client is shaded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] tsreaper merged pull request #475: [FLINK-30620] Table Store Hive catalog supports specifying custom Hive metastore client
Posted by GitBox <gi...@apache.org>.
tsreaper merged PR #475:
URL: https://github.com/apache/flink-table-store/pull/475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org