You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/06/30 12:13:59 UTC
[GitHub] [hudi] cuibo01 opened a new pull request, #6013: [WIP]Support hoodie catalog
cuibo01 opened a new pull request, #6013:
URL: https://github.com/apache/hudi/pull/6013
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179054663
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5051b524f5e241103afbdd9394d0a14a660c3bd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779)
* 8032fcaf15f8b246c8652ed5df49fa01564e2649 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790)
* 185ec5470953fc3420fb85c8d72c40bbf317f995 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791)
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179655321
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
}, {
"hash" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808",
"triggerID" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 0c113baae795ddf7f884b868dcde0ddc3f8adb92 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792)
* 02ff7556a3d1324bdf66aea4fd28988c8233e987 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r916512804
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieCatalogUtil.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.flink.api.java.hadoop.mapred.utils.HadoopUtils;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.HIVE_SITE_FILE;
+
+/**
+ * Utilities for Hoodie Catalog.
+ */
+public class HoodieCatalogUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieCatalogUtil.class);
+
+ /**
+ * Returns a new hiveConfig.
+ *
Review Comment:
`hiveConfig` -> `{@code HiveConf}`.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf, String hadoopConf) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf, hadoopConf), false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created HiveCatalog '{}'", catalogName);
+ }
Review Comment:
HiveCatalog -> hoodie catalog in hms mode
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf, String hadoopConf) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf, hadoopConf), false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created HiveCatalog '{}'", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
Review Comment:
What about the table is not a hudi table at all, do we still need the translation ?
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf, String hadoopConf) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf, hadoopConf), false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created HiveCatalog '{}'", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
+ parameters.put(PATH.key(), path);
+ if (!parameters.containsKey(FlinkOptions.HIVE_STYLE_PARTITIONING.key())) {
+ Path hoodieTablePath = new Path(path);
+ boolean hiveStyle = Arrays.stream(FSUtils.getFs(hoodieTablePath, hiveConf).listStatus(hoodieTablePath))
+ .map(fileStatus -> fileStatus.getPath().getName())
+ .filter(f -> !f.equals(".hoodie") && !f.equals("default"))
+ .anyMatch(FilePathUtils::hiveStylePartitionMath);
+ parameters.put(FlinkOptions.HIVE_STYLE_PARTITIONING.key(), String.valueOf(hiveStyle));
+ }
+ client.alter_table(tablePath.getDatabaseName(), tablePath.getObjectName(), hiveTable);
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to update table schema", e);
+ }
+ }
+ return hiveTable;
+ }
+
+ @Override
+ public CatalogBaseTable getTable(ObjectPath tablePath) throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ Table hiveTable = getHiveTable(tablePath);
+ hiveTable = translateSparkTable2Flink(tablePath, hiveTable);
+ String path = hiveTable.getSd().getLocation();
+ Map<String, String> parameters = hiveTable.getParameters();
+ Schema latestTableSchema = getLatestTableSchema(path);
+ org.apache.flink.table.api.Schema schema;
+ if (latestTableSchema != null) {
+ org.apache.flink.table.api.Schema.Builder builder = org.apache.flink.table.api.Schema.newBuilder()
+ .fromRowDataType(AvroSchemaConverter.convertToDataType(latestTableSchema));
+ String pkConstraintName = parameters.get(PK_CONSTRAINT_NAME);
+ if (!StringUtils.isNullOrEmpty(pkConstraintName)) {
+ builder.primaryKeyNamed(pkConstraintName, StringUtils.split(parameters.get(PK_COLUMNS), ","));
+ }
+ schema = builder.build();
+ } else {
+ LOG.warn("{} does not have any hoodie schema, and use hive table to covert the catalogBaseTable", tablePath);
+ schema = TableOptionProperties.convertTableSchema(hiveTable);
Review Comment:
use hive table to covert the catalogBaseTable -> use hive table schema to infer the table schema instead
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf, String hadoopConf) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf, hadoopConf), false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created HiveCatalog '{}'", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
+ parameters.put(PATH.key(), path);
+ if (!parameters.containsKey(FlinkOptions.HIVE_STYLE_PARTITIONING.key())) {
+ Path hoodieTablePath = new Path(path);
+ boolean hiveStyle = Arrays.stream(FSUtils.getFs(hoodieTablePath, hiveConf).listStatus(hoodieTablePath))
+ .map(fileStatus -> fileStatus.getPath().getName())
+ .filter(f -> !f.equals(".hoodie") && !f.equals("default"))
+ .anyMatch(FilePathUtils::hiveStylePartitionMath);
Review Comment:
hiveStylePartitionMath -> did you mean isHiveStylePartitioning ?
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieCatalogUtil.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.flink.api.java.hadoop.mapred.utils.HadoopUtils;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.HIVE_SITE_FILE;
+
+/**
+ * Utilities for Hoodie Catalog.
+ */
+public class HoodieCatalogUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieCatalogUtil.class);
+
+ /**
+ * Returns a new hiveConfig.
+ *
+ * @param hiveConfDir Hive conf directory path.
+ * @param hadoopConfDir Hadoop conf directory path.
+ * @return A HiveConf instance.
+ */
+ public static HiveConf createHiveConf(@Nullable String hiveConfDir, @Nullable String hadoopConfDir) {
+ // create HiveConf from hadoop configuration with hadoop conf directory configured.
+ Configuration hadoopConf = null;
+ if (isNullOrWhitespaceOnly(hadoopConfDir)) {
+ for (String possibleHadoopConfPath :
+ HadoopUtils.possibleHadoopConfPaths(
+ new org.apache.flink.configuration.Configuration())) {
+ hadoopConf = getHadoopConfiguration(possibleHadoopConfPath);
+ if (hadoopConf != null) {
+ break;
+ }
+ }
+ } else {
+ hadoopConf = getHadoopConfiguration(hadoopConfDir);
+ if (hadoopConf == null) {
+ String possiableUsedConfFiles =
+ "core-site.xml | hdfs-site.xml | yarn-site.xml | mapred-site.xml";
+ throw new CatalogException(
+ "Failed to load the hadoop conf from specified path:" + hadoopConfDir,
+ new FileNotFoundException(
+ "Please check the path none of the conf files ("
+ + possiableUsedConfFiles
+ + ") exist in the folder."));
+ }
+ }
+ if (hadoopConf == null) {
+ hadoopConf = new Configuration();
+ }
+ // ignore all the static conf file URLs that HiveConf may have set
+ HiveConf.setHiveSiteLocation(null);
+ HiveConf.setLoadMetastoreConfig(false);
+ HiveConf.setLoadHiveServer2Config(false);
+ HiveConf hiveConf = new HiveConf(hadoopConf, HiveConf.class);
+
+ LOG.info("Setting hive conf dir as {}", hiveConfDir);
+
+ if (hiveConfDir != null) {
+ Path hiveSite = new Path(hiveConfDir, HIVE_SITE_FILE);
+ if (!hiveSite.toUri().isAbsolute()) {
+ // treat relative URI as local file to be compatible with previous behavior
+ hiveSite = new Path(new File(hiveSite.toString()).toURI());
+ }
+ try (InputStream inputStream = hiveSite.getFileSystem(hadoopConf).open(hiveSite)) {
+ hiveConf.addResource(inputStream, hiveSite.toString());
+ // trigger a read from the conf so that the input stream is read
+ isEmbeddedMetastore(hiveConf);
+ } catch (IOException e) {
+ throw new CatalogException(
+ "Failed to load hive-site.xml from specified path:" + hiveSite, e);
+ }
+ } else {
+ // user doesn't provide hive conf dir, we try to find it in classpath
+ URL hiveSite =
+ Thread.currentThread().getContextClassLoader().getResource(HIVE_SITE_FILE);
+ if (hiveSite != null) {
+ LOG.info("Found {} in classpath: {}", HIVE_SITE_FILE, hiveSite);
+ hiveConf.addResource(hiveSite);
+ }
+ }
+ return hiveConf;
+ }
+
+ /**
+ * Check whether the hive.metastore.uris is empty
+ */
+ public static boolean isEmbeddedMetastore(HiveConf hiveConf) {
+ return isNullOrWhitespaceOnly(hiveConf.getVar(HiveConf.ConfVars.METASTOREURIS));
+ }
+
+ /**
+ * Returns a new Hadoop Configuration object using the path to the hadoop conf configured.
+ *
+ * @param hadoopConfDir Hadoop conf directory path.
+ * @return A Hadoop configuration instance.
+ */
+ public static Configuration getHadoopConfiguration(String hadoopConfDir) {
+ if (new File(hadoopConfDir).exists()) {
+ List<File> possiableConfFiles = new ArrayList<File>();
+ File coreSite = new File(hadoopConfDir, "core-site.xml");
+ if (coreSite.exists()) {
+ possiableConfFiles.add(coreSite);
+ }
+ File hdfsSite = new File(hadoopConfDir, "hdfs-site.xml");
+ if (hdfsSite.exists()) {
+ possiableConfFiles.add(hdfsSite);
+ }
+ File yarnSite = new File(hadoopConfDir, "yarn-site.xml");
+ if (yarnSite.exists()) {
+ possiableConfFiles.add(yarnSite);
+ }
+ // Add mapred-site.xml. We need to read configurations like compression codec.
Review Comment:
Did you know that we already have tool clazz for hive/hadoop configurations in `HadoopConfugurations` ?
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -442,4 +442,8 @@ public static String[] extractHivePartitionFields(org.apache.flink.configuration
}
return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
+
+ public static boolean hiveStylePartitionMath(String path) {
+ return HIVE_PARTITION_NAME_PATTERN.matcher(path).matches();
Review Comment:
hiveStylePartitionMath -> isHiveStylePartitioning
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/catalog/TestHoodieHiveCatalog.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogTableImpl;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Test cases for {@link HoodieHiveCatalog}.
+ */
+public class TestHoodieHiveCatalog {
+ TableSchema schema =
+ TableSchema.builder()
+ .field("name", DataTypes.STRING())
+ .field("age", DataTypes.INT())
+ .build();
+ private static HoodieHiveCatalog hoodieCatalog;
+ private final ObjectPath tablePath = new ObjectPath("default", "test");
+
+ @BeforeAll
+ public static void createCatalog() {
+ hoodieCatalog = TestHoodieCatalogUtils.createHiveCatalog();
+ hoodieCatalog.open();
+ }
+
+ @AfterEach
+ public void dropTable() throws TableNotExistException {
+ hoodieCatalog.dropTable(tablePath, true);
+ }
+
+ @AfterAll
+ public static void closeCatalog() {
+ if (hoodieCatalog != null) {
+ hoodieCatalog.close();
+ }
+ }
+
+ @Test
+ public void testCreateAndGetHoodieMORTable() throws Exception {
+ Map<String, String> originOptions = new HashMap<>();
+ originOptions.put(FactoryUtil.CONNECTOR.key(), "hudi");
+ originOptions.put(FlinkOptions.TABLE_TYPE.key(), FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
+ CatalogTable table =
+ new CatalogTableImpl(schema, originOptions, "hudi table");
+ hoodieCatalog.createTable(tablePath, table, false);
+ CatalogBaseTable table1 = hoodieCatalog.getTable(tablePath);
+ assertEquals(table1.getOptions().get(CONNECTOR.key()), "hudi");
+ assertEquals(table1.getOptions().get(FlinkOptions.TABLE_TYPE.key()), FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
+ }
+
+ @Test
+ public void testCreateAndGetHoodieTable() throws Exception {
+ Map<String, String> originOptions =
+ Collections.singletonMap(FactoryUtil.CONNECTOR.key(), "hudi");
+ CatalogTable table =
+ new CatalogTableImpl(schema, originOptions, "hudi table");
+ hoodieCatalog.createTable(tablePath, table, false);
+ CatalogBaseTable table1 = hoodieCatalog.getTable(tablePath);
+ assertEquals(table1.getOptions().get(CONNECTOR.key()), "hudi");
+ }
+
+ @Test
+ public void testCreateNonHoodieTable() throws TableAlreadyExistException, DatabaseNotExistException {
+ CatalogTable table =
+ new CatalogTableImpl(schema, Collections.emptyMap(), "hudi table");
+ try {
+ hoodieCatalog.createTable(tablePath, table, false);
Review Comment:
We also need to add some test cases in `ITTestHoodieDataSource` to use the catalog to read and write.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf, String hadoopConf) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf, hadoopConf), false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created HiveCatalog '{}'", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
+ parameters.put(PATH.key(), path);
+ if (!parameters.containsKey(FlinkOptions.HIVE_STYLE_PARTITIONING.key())) {
+ Path hoodieTablePath = new Path(path);
+ boolean hiveStyle = Arrays.stream(FSUtils.getFs(hoodieTablePath, hiveConf).listStatus(hoodieTablePath))
+ .map(fileStatus -> fileStatus.getPath().getName())
+ .filter(f -> !f.equals(".hoodie") && !f.equals("default"))
+ .anyMatch(FilePathUtils::hiveStylePartitionMath);
+ parameters.put(FlinkOptions.HIVE_STYLE_PARTITIONING.key(), String.valueOf(hiveStyle));
+ }
+ client.alter_table(tablePath.getDatabaseName(), tablePath.getObjectName(), hiveTable);
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to update table schema", e);
+ }
+ }
+ return hiveTable;
+ }
+
+ @Override
+ public CatalogBaseTable getTable(ObjectPath tablePath) throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ Table hiveTable = getHiveTable(tablePath);
+ hiveTable = translateSparkTable2Flink(tablePath, hiveTable);
+ String path = hiveTable.getSd().getLocation();
+ Map<String, String> parameters = hiveTable.getParameters();
+ Schema latestTableSchema = getLatestTableSchema(path);
+ org.apache.flink.table.api.Schema schema;
+ if (latestTableSchema != null) {
+ org.apache.flink.table.api.Schema.Builder builder = org.apache.flink.table.api.Schema.newBuilder()
+ .fromRowDataType(AvroSchemaConverter.convertToDataType(latestTableSchema));
+ String pkConstraintName = parameters.get(PK_CONSTRAINT_NAME);
+ if (!StringUtils.isNullOrEmpty(pkConstraintName)) {
+ builder.primaryKeyNamed(pkConstraintName, StringUtils.split(parameters.get(PK_COLUMNS), ","));
+ }
+ schema = builder.build();
+ } else {
+ LOG.warn("{} does not have any hoodie schema, and use hive table to covert the catalogBaseTable", tablePath);
+ schema = TableOptionProperties.convertTableSchema(hiveTable);
+ }
+ return CatalogTable.of(schema, parameters.get(COMMENT),
+ TableOptionProperties.getFieldNames(hiveTable.getPartitionKeys()), parameters);
+ }
+
+ @Override
+ public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
+ throws TableAlreadyExistException, DatabaseNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkNotNull(table, "table cannot be null");
+
+ if (!databaseExists(tablePath.getDatabaseName())) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName());
+ }
+
+ if (!table.getOptions().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("The %s is not hoodie table", tablePath.getObjectName()));
+ }
+
+ if (table instanceof CatalogView) {
+ throw new HoodieCatalogException("Hoodie catalog does not support to CREATE VIEW.");
+ }
+
+ try {
+ boolean isMorTable = table.getOptions().getOrDefault(FlinkOptions.TABLE_TYPE.key(),
+ FlinkOptions.TABLE_TYPE.defaultValue()).equalsIgnoreCase(FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
Review Comment:
`getOrDefault(FlinkOptions.TABLE_TYPE.key(), FlinkOptions.TABLE_TYPE.defaultValue())` -> use get directly is ok if the option already has a default value.
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/catalog/TestHoodieHiveCatalog.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogTableImpl;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Test cases for {@link HoodieHiveCatalog}.
+ */
+public class TestHoodieHiveCatalog {
+ TableSchema schema =
+ TableSchema.builder()
+ .field("name", DataTypes.STRING())
+ .field("age", DataTypes.INT())
+ .build();
+ private static HoodieHiveCatalog hoodieCatalog;
+ private final ObjectPath tablePath = new ObjectPath("default", "test");
+
+ @BeforeAll
+ public static void createCatalog() {
+ hoodieCatalog = TestHoodieCatalogUtils.createHiveCatalog();
+ hoodieCatalog.open();
+ }
+
+ @AfterEach
+ public void dropTable() throws TableNotExistException {
+ hoodieCatalog.dropTable(tablePath, true);
+ }
+
+ @AfterAll
+ public static void closeCatalog() {
+ if (hoodieCatalog != null) {
+ hoodieCatalog.close();
+ }
+ }
+
+ @Test
+ public void testCreateAndGetHoodieMORTable() throws Exception {
+ Map<String, String> originOptions = new HashMap<>();
+ originOptions.put(FactoryUtil.CONNECTOR.key(), "hudi");
+ originOptions.put(FlinkOptions.TABLE_TYPE.key(), FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
+ CatalogTable table =
+ new CatalogTableImpl(schema, originOptions, "hudi table");
+ hoodieCatalog.createTable(tablePath, table, false);
+ CatalogBaseTable table1 = hoodieCatalog.getTable(tablePath);
+ assertEquals(table1.getOptions().get(CONNECTOR.key()), "hudi");
+ assertEquals(table1.getOptions().get(FlinkOptions.TABLE_TYPE.key()), FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
+ }
+
+ @Test
+ public void testCreateAndGetHoodieTable() throws Exception {
+ Map<String, String> originOptions =
+ Collections.singletonMap(FactoryUtil.CONNECTOR.key(), "hudi");
+ CatalogTable table =
+ new CatalogTableImpl(schema, originOptions, "hudi table");
+ hoodieCatalog.createTable(tablePath, table, false);
+ CatalogBaseTable table1 = hoodieCatalog.getTable(tablePath);
+ assertEquals(table1.getOptions().get(CONNECTOR.key()), "hudi");
Review Comment:
We may need to inspect key options like partition keys, primary keys, preCombine keys, and custom keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179749844
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
}, {
"hash" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808",
"triggerID" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"triggerType" : "PUSH"
}, {
"hash" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9819",
"triggerID" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 02ff7556a3d1324bdf66aea4fd28988c8233e987 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808)
* f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9819)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179059344
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5051b524f5e241103afbdd9394d0a14a660c3bd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779)
* 8032fcaf15f8b246c8652ed5df49fa01564e2649 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790)
* 185ec5470953fc3420fb85c8d72c40bbf317f995 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791)
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 0c113baae795ddf7f884b868dcde0ddc3f8adb92 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiaozhch5 commented on pull request #6013: [WIP][HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
xiaozhch5 commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1173127949
When I create the table in flink sql, I can query the data in spark sql directly. But when I create the table in spark sql, I can't query the data in flink sql. Does it possible for flink to query the data after spark sql create the table?
spark sql
flink sql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179741015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
}, {
"hash" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808",
"triggerID" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"triggerType" : "PUSH"
}, {
"hash" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 02ff7556a3d1324bdf66aea4fd28988c8233e987 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808)
* f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179001089
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5051b524f5e241103afbdd9394d0a14a660c3bd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779)
* 8032fcaf15f8b246c8652ed5df49fa01564e2649 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790)
* 185ec5470953fc3420fb85c8d72c40bbf317f995 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1178798507
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5051b524f5e241103afbdd9394d0a14a660c3bd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r916510581
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieCatalogFactoryOptions.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.table.catalog.CommonCatalogOptions;
+
+/** {@link ConfigOption}s for {@link HoodieHiveCatalog}. */
+public class HoodieCatalogFactoryOptions {
+ public static final String IDENTIFIER = "hudi-hive";
+ public static final String DEFAULT_DB = "default";
Review Comment:
Can we just name the catalog identifier as `hudi` and apply two modes: hms and dfs ? The new name `hudi-hive` is really confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1178923894
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5051b524f5e241103afbdd9394d0a14a660c3bd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779)
* 8032fcaf15f8b246c8652ed5df49fa01564e2649 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r917361595
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf), false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created Hoodie Catalog '{}' in hms mode", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
+ parameters.put(PATH.key(), path);
+ if (!parameters.containsKey(FlinkOptions.HIVE_STYLE_PARTITIONING.key())) {
+ Path hoodieTablePath = new Path(path);
+ boolean hiveStyle = Arrays.stream(FSUtils.getFs(hoodieTablePath, hiveConf).listStatus(hoodieTablePath))
+ .map(fileStatus -> fileStatus.getPath().getName())
+ .filter(f -> !f.equals(".hoodie") && !f.equals("default"))
+ .anyMatch(FilePathUtils::isHiveStylePartitioning);
+ parameters.put(FlinkOptions.HIVE_STYLE_PARTITIONING.key(), String.valueOf(hiveStyle));
+ }
+ client.alter_table(tablePath.getDatabaseName(), tablePath.getObjectName(), hiveTable);
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to update table schema", e);
+ }
+ }
+ return hiveTable;
+ }
+
+ @Override
+ public CatalogBaseTable getTable(ObjectPath tablePath) throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ Table hiveTable = getHiveTable(tablePath);
+ hiveTable = translateSparkTable2Flink(tablePath, hiveTable);
+ String path = hiveTable.getSd().getLocation();
+ Map<String, String> parameters = hiveTable.getParameters();
+ Schema latestTableSchema = getLatestTableSchema(path);
+ org.apache.flink.table.api.Schema schema;
+ if (latestTableSchema != null) {
+ org.apache.flink.table.api.Schema.Builder builder = org.apache.flink.table.api.Schema.newBuilder()
+ .fromRowDataType(AvroSchemaConverter.convertToDataType(latestTableSchema));
+ String pkConstraintName = parameters.get(PK_CONSTRAINT_NAME);
+ if (!StringUtils.isNullOrEmpty(pkConstraintName)) {
+ builder.primaryKeyNamed(pkConstraintName, StringUtils.split(parameters.get(PK_COLUMNS), ","));
+ }
+ schema = builder.build();
+ } else {
+ LOG.warn("{} does not have any hoodie schema, and use hive table schema to infer the table schema", tablePath);
+ schema = TableOptionProperties.convertTableSchema(hiveTable);
+ }
+ return CatalogTable.of(schema, parameters.get(COMMENT),
+ TableOptionProperties.getFieldNames(hiveTable.getPartitionKeys()), parameters);
+ }
+
+ @Override
+ public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
+ throws TableAlreadyExistException, DatabaseNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkNotNull(table, "table cannot be null");
+
+ if (!databaseExists(tablePath.getDatabaseName())) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName());
+ }
+
+ if (!table.getOptions().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("The %s is not hoodie table", tablePath.getObjectName()));
+ }
+
+ if (table instanceof CatalogView) {
+ throw new HoodieCatalogException("Hoodie catalog does not support to CREATE VIEW.");
+ }
+
+ try {
+ boolean isMorTable = table.getOptions().getOrDefault(FlinkOptions.TABLE_TYPE.key(),
+ FlinkOptions.TABLE_TYPE.defaultValue()).equalsIgnoreCase(FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
+ Table hiveTable = instantiateHiveTable(tablePath, table, inferTablePath(tablePath, table), isMorTable);
+ //create hive table
+ client.createTable(hiveTable);
+ //init hoodie metaClient
+ initTableIfNotExists(tablePath, (CatalogTable)table);
Review Comment:
Updated
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf), false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created Hoodie Catalog '{}' in hms mode", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
+ parameters.put(PATH.key(), path);
+ if (!parameters.containsKey(FlinkOptions.HIVE_STYLE_PARTITIONING.key())) {
+ Path hoodieTablePath = new Path(path);
+ boolean hiveStyle = Arrays.stream(FSUtils.getFs(hoodieTablePath, hiveConf).listStatus(hoodieTablePath))
+ .map(fileStatus -> fileStatus.getPath().getName())
+ .filter(f -> !f.equals(".hoodie") && !f.equals("default"))
+ .anyMatch(FilePathUtils::isHiveStylePartitioning);
+ parameters.put(FlinkOptions.HIVE_STYLE_PARTITIONING.key(), String.valueOf(hiveStyle));
+ }
+ client.alter_table(tablePath.getDatabaseName(), tablePath.getObjectName(), hiveTable);
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to update table schema", e);
+ }
+ }
+ return hiveTable;
+ }
+
+ @Override
+ public CatalogBaseTable getTable(ObjectPath tablePath) throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ Table hiveTable = getHiveTable(tablePath);
+ hiveTable = translateSparkTable2Flink(tablePath, hiveTable);
+ String path = hiveTable.getSd().getLocation();
+ Map<String, String> parameters = hiveTable.getParameters();
+ Schema latestTableSchema = getLatestTableSchema(path);
+ org.apache.flink.table.api.Schema schema;
+ if (latestTableSchema != null) {
+ org.apache.flink.table.api.Schema.Builder builder = org.apache.flink.table.api.Schema.newBuilder()
+ .fromRowDataType(AvroSchemaConverter.convertToDataType(latestTableSchema));
+ String pkConstraintName = parameters.get(PK_CONSTRAINT_NAME);
+ if (!StringUtils.isNullOrEmpty(pkConstraintName)) {
+ builder.primaryKeyNamed(pkConstraintName, StringUtils.split(parameters.get(PK_COLUMNS), ","));
+ }
+ schema = builder.build();
+ } else {
+ LOG.warn("{} does not have any hoodie schema, and use hive table schema to infer the table schema", tablePath);
+ schema = TableOptionProperties.convertTableSchema(hiveTable);
+ }
+ return CatalogTable.of(schema, parameters.get(COMMENT),
+ TableOptionProperties.getFieldNames(hiveTable.getPartitionKeys()), parameters);
+ }
+
+ @Override
+ public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
+ throws TableAlreadyExistException, DatabaseNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkNotNull(table, "table cannot be null");
+
+ if (!databaseExists(tablePath.getDatabaseName())) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName());
+ }
+
+ if (!table.getOptions().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("The %s is not hoodie table", tablePath.getObjectName()));
+ }
+
+ if (table instanceof CatalogView) {
+ throw new HoodieCatalogException("Hoodie catalog does not support to CREATE VIEW.");
+ }
+
+ try {
+ boolean isMorTable = table.getOptions().getOrDefault(FlinkOptions.TABLE_TYPE.key(),
+ FlinkOptions.TABLE_TYPE.defaultValue()).equalsIgnoreCase(FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
+ Table hiveTable = instantiateHiveTable(tablePath, table, inferTablePath(tablePath, table), isMorTable);
+ //create hive table
+ client.createTable(hiveTable);
+ //init hoodie metaClient
+ initTableIfNotExists(tablePath, (CatalogTable)table);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new TableAlreadyExistException(getName(), tablePath, e);
+ }
+ } catch (Exception e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create table %s", tablePath.getFullName()), e);
+ }
+ }
+
+ private void initTableIfNotExists(ObjectPath tablePath, CatalogTable catalogTable) {
+ Configuration flinkConf = Configuration.fromMap(applyOptionsHook(catalogTable.getOptions()));
+ flinkConf.addAllToProperties(hiveConf.getAllProperties());
+ final String avroSchema = AvroSchemaConverter.convertToSchema(catalogTable.getSchema().toPhysicalRowDataType().getLogicalType()).toString();
+ flinkConf.setString(FlinkOptions.SOURCE_AVRO_SCHEMA, avroSchema);
+
+ // stores two copies of options:
+ // - partition keys
+ // - primary keys
+ // because the HoodieTableMetaClient is a heavy impl, we try to avoid initializing it
+ // when calling #getTable.
+
+ if (catalogTable.getUnresolvedSchema().getPrimaryKey().isPresent()) {
+ final String pkColumns = String.join(",", catalogTable.getUnresolvedSchema().getPrimaryKey().get().getColumnNames());
+ String recordKey = flinkConf.get(FlinkOptions.RECORD_KEY_FIELD);
+ if (!Objects.equals(pkColumns, recordKey)) {
+ throw new HoodieCatalogException(String.format("%s and %s are the different", pkColumns, recordKey));
+ }
+ }
+
+ if (catalogTable.isPartitioned()) {
+ final String partitions = String.join(",", catalogTable.getPartitionKeys());
+ flinkConf.setString(FlinkOptions.PARTITION_PATH_FIELD, partitions);
+ }
+
+ if (!flinkConf.getOptional(PATH).isPresent()) {
+ flinkConf.setString(PATH, inferTablePath(tablePath, catalogTable));
+ }
+
+ flinkConf.setString(FlinkOptions.TABLE_NAME, tablePath.getObjectName());
+ try {
+ StreamerUtil.initTableIfNotExists(flinkConf);
+ } catch (IOException e) {
+ throw new HoodieCatalogException("Initialize table exception.", e);
+ }
+ }
+
+ private String inferTablePath(ObjectPath tablePath, CatalogBaseTable table) {
+ String location = table.getOptions().getOrDefault(PATH.key(), "");
+ if (StringUtils.isNullOrEmpty(location)) {
+ try {
+ Path dbLocation = new Path(client.getDatabase(tablePath.getDatabaseName()).getLocationUri());
+ location = new Path(dbLocation, tablePath.getObjectName()).toString();
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to infer hoodie table path for table %s", tablePath), e);
+ }
+ }
+ return location;
+ }
+
+ private Map<String, String> applyOptionsHook(Map<String, String> options) {
+ Map<String, String> properties = new HashMap<>(options);
+ if (!options.containsKey(FlinkOptions.RECORD_KEY_FIELD.key())) {
+ properties.put(FlinkOptions.RECORD_KEY_FIELD.key(), FlinkOptions.RECORD_KEY_FIELD.defaultValue());
+ }
+ if (!options.containsKey(FlinkOptions.PRECOMBINE_FIELD.key())) {
+ properties.put(FlinkOptions.PRECOMBINE_FIELD.key(), FlinkOptions.PRECOMBINE_FIELD.defaultValue());
+ }
+ if (!options.containsKey(FlinkOptions.TABLE_TYPE.key())) {
+ properties.put(FlinkOptions.TABLE_TYPE.key(), FlinkOptions.TABLE_TYPE.defaultValue());
+ }
+ return properties;
+ }
+
+ private Table instantiateHiveTable(ObjectPath tablePath, CatalogBaseTable table, String location, boolean useRealTimeInputFormat) throws IOException {
+ // let Hive set default parameters for us, e.g. serialization.format
+ Table hiveTable =
+ org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(
+ tablePath.getDatabaseName(), tablePath.getObjectName());
+ hiveTable.setOwner(UserGroupInformation.getCurrentUser().getUserName());
+ hiveTable.setCreateTime((int) (System.currentTimeMillis() / 1000));
+
+ Map<String, String> properties = applyOptionsHook(table.getOptions());
+ properties.put("EXTERNAL", "TRUE");
+ // Table comment
+ if (table.getComment() != null) {
+ properties.put(COMMENT, table.getComment());
+ }
+
+ //set sd
+ StorageDescriptor sd = new StorageDescriptor();
+ List<FieldSchema> allColumns = TableOptionProperties.createHiveColumns(table.getSchema());
+
+ // Table columns and partition keys
+ if (table instanceof CatalogTable) {
+ CatalogTable catalogTable = (CatalogTable) table;
+
+ if (catalogTable.isPartitioned()) {
+ int partitionKeySize = catalogTable.getPartitionKeys().size();
+ List<FieldSchema> regularColumns =
+ allColumns.subList(0, allColumns.size() - partitionKeySize);
+ List<FieldSchema> partitionColumns =
+ allColumns.subList(
+ allColumns.size() - partitionKeySize, allColumns.size());
+
+ sd.setCols(regularColumns);
+ hiveTable.setPartitionKeys(partitionColumns);
+ } else {
+ sd.setCols(allColumns);
+ hiveTable.setPartitionKeys(new ArrayList<>());
+ }
+ } else {
+ sd.setCols(allColumns);
+ }
+
+ HoodieFileFormat baseFileFormat = HoodieFileFormat.PARQUET;
+ //ignore uber input Format
+ String inputFormatClassName = HoodieInputFormatUtils.getInputFormatClassName(baseFileFormat, useRealTimeInputFormat);
+ String outputFormatClassName = HoodieInputFormatUtils.getOutputFormatClassName(baseFileFormat);
+ String serDeClassName = HoodieInputFormatUtils.getSerDeClassName(baseFileFormat);
+ sd.setInputFormat(inputFormatClassName);
+ sd.setOutputFormat(outputFormatClassName);
+ Map<String, String> serdeProperties = new HashMap<>();
+ serdeProperties.put("path", location);
+ serdeProperties.put(ConfigUtils.IS_QUERY_AS_RO_TABLE, String.valueOf(!useRealTimeInputFormat));
+ serdeProperties.put("serialization.format", "1");
+ sd.setSerdeInfo(new SerDeInfo(null, serDeClassName, serdeProperties));
+
+ sd.setLocation(location);
+
+ hiveTable.setSd(sd);
+
+ properties.putAll(TableOptionProperties.translateFlinkTableProperties2Spark((CatalogTable)table, hiveConf));
+
+ //set pk
+ if (table.getUnresolvedSchema().getPrimaryKey().isPresent()) {
+ String pkColumns = String.join(",", table.getUnresolvedSchema().getPrimaryKey().get().getColumnNames());
+ String recordKey = properties.getOrDefault(FlinkOptions.RECORD_KEY_FIELD.key(), FlinkOptions.RECORD_KEY_FIELD.defaultValue());
+ if (!Objects.equals(pkColumns, recordKey)) {
+ throw new HoodieCatalogException(
+ String.format("If the table has primaryKey, the primaryKey should be the the same as the recordKey, but pk %s and recordKey %s are the different",
+ pkColumns,
+ recordKey));
+ }
+ properties.put(PK_CONSTRAINT_NAME, table.getUnresolvedSchema().getPrimaryKey().get().getConstraintName());
+ properties.put(PK_COLUMNS, pkColumns);
+ }
+
+ if (!properties.containsKey(FlinkOptions.PATH.key())) {
+ properties.put(FlinkOptions.PATH.key(), location);
+ }
+
+ hiveTable.setParameters(properties);
+ return hiveTable;
+ }
+
+ @Override
+ public List<String> listTables(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+
+ try {
+ return client.getAllTables(databaseName);
+ } catch (UnknownDBException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list tables in database %s", databaseName), e);
+ }
+ }
+
+ @Override
+ public List<String> listViews(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ throw new HoodieCatalogException("Hoodie catalog does not support to listViews");
+ }
+
+ @Override
+ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+
+ try {
+ return client.tableExists(tablePath.getDatabaseName(), tablePath.getObjectName());
+ } catch (UnknownDBException e) {
+ return false;
+ } catch (TException e) {
+ throw new CatalogException(
+ String.format(
+ "Failed to check whether table %s exists or not.",
+ tablePath.getFullName()),
+ e);
+ }
+ }
+
+ @Override
+ public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
+ throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+
+ try {
+ client.dropTable(
+ tablePath.getDatabaseName(),
+ tablePath.getObjectName(),
+ // Indicate whether associated data should be deleted.
+ // Set to 'true' for now because Flink tables shouldn't have data in Hive. Can
+ // be changed later if necessary
+ true,
+ ignoreIfNotExists);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new TableNotExistException(getName(), tablePath);
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to drop table %s", tablePath.getFullName()), e);
+ }
+ }
+
+ @Override
+ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignoreIfNotExists)
+ throws TableNotExistException, TableAlreadyExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkArgument(
+ !isNullOrWhitespaceOnly(newTableName), "newTableName cannot be null or empty");
+
+ try {
+ // alter_table() doesn't throw a clear exception when target table doesn't exist.
+ // Thus, check the table existence explicitly
+ if (tableExists(tablePath)) {
+ ObjectPath newPath = new ObjectPath(tablePath.getDatabaseName(), newTableName);
+ // alter_table() doesn't throw a clear exception when new table already exists.
+ // Thus, check the table existence explicitly
+ if (tableExists(newPath)) {
+ throw new TableAlreadyExistException(getName(), newPath);
+ } else {
+ Table hiveTable = getHiveTable(tablePath);
+
+ //update hoodie
+ StorageDescriptor sd = hiveTable.getSd();
+ String location = sd.getLocation();
+ HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setBasePath(location).setConf(hiveConf).build();
+ //Init table with new name
+ HoodieTableMetaClient.withPropertyBuilder().fromProperties(metaClient.getTableConfig().getProps())
+ .setTableName(newTableName)
+ .initTable(hiveConf, location);
+
+ hiveTable.setTableName(newTableName);
+ client.alter_table(
+ tablePath.getDatabaseName(), tablePath.getObjectName(), hiveTable);
+ }
+ } else if (!ignoreIfNotExists) {
+ throw new TableNotExistException(getName(), tablePath);
+ }
+ } catch (Exception e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to rename table %s", tablePath.getFullName()), e);
+ }
+ }
+
+ @Override
+ public void alterTable(
+ ObjectPath tablePath, CatalogBaseTable newCatalogTable, boolean ignoreIfNotExists)
+ throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkNotNull(newCatalogTable, "newCatalogTable cannot be null");
+
+ if (!newCatalogTable.getOptions().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("The %s is not hoodie table", tablePath.getObjectName()));
+ }
+ if (newCatalogTable instanceof CatalogView) {
+ throw new HoodieCatalogException("Hoodie catalog does not support to ALTER VIEW");
+ }
+
+ try {
+ Table hiveTable = getHiveTable(tablePath);
+ } catch (TableNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw e;
+ }
+ return;
+ }
+ try {
+ boolean isMorTable = newCatalogTable.getOptions().getOrDefault(FlinkOptions.TABLE_TYPE.key(),
+ FlinkOptions.TABLE_TYPE.defaultValue()).equalsIgnoreCase(FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179668221
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
}, {
"hash" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808",
"triggerID" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 02ff7556a3d1324bdf66aea4fd28988c8233e987 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiaozhch5 commented on a diff in pull request #6013: [WIP][HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
xiaozhch5 commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r912508833
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/TableOptionProperties.java:
##########
@@ -53,9 +81,26 @@ public class TableOptionProperties {
public static final String PK_COLUMNS = "pk.columns";
public static final String COMMENT = "comment";
public static final String PARTITION_COLUMNS = "partition.columns";
+ public static final String SPARK_SOURCE_PROVIDER = "spark.sql.source.provider";
Review Comment:
SPARK_SOURCE_PROVIDER should be "spark.sql.sources.provider" so as to resolve the problem list before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179068194
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5051b524f5e241103afbdd9394d0a14a660c3bd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779)
* 8032fcaf15f8b246c8652ed5df49fa01564e2649 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790)
* 185ec5470953fc3420fb85c8d72c40bbf317f995 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791)
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 0c113baae795ddf7f884b868dcde0ddc3f8adb92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179227887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8032fcaf15f8b246c8652ed5df49fa01564e2649 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790)
* 185ec5470953fc3420fb85c8d72c40bbf317f995 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791)
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 0c113baae795ddf7f884b868dcde0ddc3f8adb92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1181063822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
}, {
"hash" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808",
"triggerID" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"triggerType" : "PUSH"
}, {
"hash" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9819",
"triggerID" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"triggerType" : "PUSH"
}, {
"hash" : "9619231aabbd36a02215098f1b329aa1ef44a581",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9824",
"triggerID" : "9619231aabbd36a02215098f1b329aa1ef44a581",
"triggerType" : "PUSH"
}, {
"hash" : "f09f982667113ed96e623a736a3b04d36ef8049b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f09f982667113ed96e623a736a3b04d36ef8049b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 9619231aabbd36a02215098f1b329aa1ef44a581 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9824)
* f09f982667113ed96e623a736a3b04d36ef8049b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179654955
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
}, {
"hash" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 0c113baae795ddf7f884b868dcde0ddc3f8adb92 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792)
* 02ff7556a3d1324bdf66aea4fd28988c8233e987 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiaozhch5 commented on a diff in pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
xiaozhch5 commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r917345825
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,907 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private final boolean iniTable;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf, boolean iniTable) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf), iniTable, false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean iniTable, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ this.iniTable = iniTable;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created Hoodie Catalog '{}' in hms mode", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
+ parameters.put(PATH.key(), path);
+ if (!parameters.containsKey(FlinkOptions.HIVE_STYLE_PARTITIONING.key())) {
+ Path hoodieTablePath = new Path(path);
+ boolean hiveStyle = Arrays.stream(FSUtils.getFs(hoodieTablePath, hiveConf).listStatus(hoodieTablePath))
+ .map(fileStatus -> fileStatus.getPath().getName())
+ .filter(f -> !f.equals(".hoodie") && !f.equals("default"))
+ .anyMatch(FilePathUtils::isHiveStylePartitioning);
+ parameters.put(FlinkOptions.HIVE_STYLE_PARTITIONING.key(), String.valueOf(hiveStyle));
+ }
+ client.alter_table(tablePath.getDatabaseName(), tablePath.getObjectName(), hiveTable);
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to update table schema", e);
+ }
+ }
+ return hiveTable;
+ }
+
+ @Override
+ public CatalogBaseTable getTable(ObjectPath tablePath) throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ Table hiveTable = getHiveTable(tablePath);
+ hiveTable = translateSparkTable2Flink(tablePath, hiveTable);
+ String path = hiveTable.getSd().getLocation();
+ Map<String, String> parameters = hiveTable.getParameters();
+ Schema latestTableSchema = getLatestTableSchema(path);
+ org.apache.flink.table.api.Schema schema;
+ if (latestTableSchema != null) {
+ org.apache.flink.table.api.Schema.Builder builder = org.apache.flink.table.api.Schema.newBuilder()
+ .fromRowDataType(AvroSchemaConverter.convertToDataType(latestTableSchema));
+ String pkConstraintName = parameters.get(PK_CONSTRAINT_NAME);
+ if (!StringUtils.isNullOrEmpty(pkConstraintName)) {
+ builder.primaryKeyNamed(pkConstraintName, StringUtils.split(parameters.get(PK_COLUMNS), ","));
+ }
+ schema = builder.build();
+ } else {
+ LOG.warn("{} does not have any hoodie schema, and use hive table schema to infer the table schema", tablePath);
+ schema = TableOptionProperties.convertTableSchema(hiveTable);
+ }
+ return CatalogTable.of(schema, parameters.get(COMMENT),
+ TableOptionProperties.getFieldNames(hiveTable.getPartitionKeys()), parameters);
+ }
+
+ @Override
+ public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
+ throws TableAlreadyExistException, DatabaseNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkNotNull(table, "table cannot be null");
+
+ if (!databaseExists(tablePath.getDatabaseName())) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName());
+ }
+
+ if (!table.getOptions().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("The %s is not hoodie table", tablePath.getObjectName()));
+ }
+
+ if (table instanceof CatalogView) {
+ throw new HoodieCatalogException("Hoodie catalog does not support to CREATE VIEW.");
+ }
+
+ try {
+ boolean isMorTable = table.getOptions().getOrDefault(FlinkOptions.TABLE_TYPE.key(),
+ FlinkOptions.TABLE_TYPE.defaultValue()).equalsIgnoreCase(FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
+ Table hiveTable = instantiateHiveTable(tablePath, table, inferTablePath(tablePath, table), isMorTable);
+ //create hive table
+ client.createTable(hiveTable);
+ //init hoodie metaClient
+ initTableIfNotExists(tablePath, (CatalogTable)table);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new TableAlreadyExistException(getName(), tablePath, e);
+ }
+ } catch (Exception e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create table %s", tablePath.getFullName()), e);
+ }
+ }
+
+ private void initTableIfNotExists(ObjectPath tablePath, CatalogTable catalogTable) {
+ if (!iniTable) {
+ LOG.info("Skip init table.");
+ return;
+ }
+ Configuration flinkConf = Configuration.fromMap(applyOptionsHook(catalogTable.getOptions()));
+ flinkConf.addAllToProperties(hiveConf.getAllProperties());
+ final String avroSchema = AvroSchemaConverter.convertToSchema(catalogTable.getSchema().toPhysicalRowDataType().getLogicalType()).toString();
+ flinkConf.setString(FlinkOptions.SOURCE_AVRO_SCHEMA, avroSchema);
+
+ // stores two copies of options:
+ // - partition keys
+ // - primary keys
+ // because the HoodieTableMetaClient is a heavy impl, we try to avoid initializing it
+ // when calling #getTable.
+
+ if (catalogTable.getUnresolvedSchema().getPrimaryKey().isPresent()) {
+ final String pkColumns = String.join(",", catalogTable.getUnresolvedSchema().getPrimaryKey().get().getColumnNames());
+ String recordKey = flinkConf.get(FlinkOptions.RECORD_KEY_FIELD);
+ if (!Objects.equals(pkColumns, recordKey)) {
+ throw new HoodieCatalogException(String.format("%s and %s are the different", pkColumns, recordKey));
+ }
+ }
+
+ if (catalogTable.isPartitioned()) {
+ final String partitions = String.join(",", catalogTable.getPartitionKeys());
+ flinkConf.setString(FlinkOptions.PARTITION_PATH_FIELD, partitions);
+ }
+
+ if (!flinkConf.getOptional(PATH).isPresent()) {
+ flinkConf.setString(PATH, inferTablePath(tablePath, catalogTable));
+ }
+
+ flinkConf.setString(FlinkOptions.TABLE_NAME, tablePath.getObjectName());
+ try {
+ StreamerUtil.initTableIfNotExists(flinkConf);
+ } catch (IOException e) {
+ throw new HoodieCatalogException("Initialize table exception.", e);
+ }
+ }
+
+ private String inferTablePath(ObjectPath tablePath, CatalogBaseTable table) {
+ String location = table.getOptions().getOrDefault(PATH.key(), "");
+ if (StringUtils.isNullOrEmpty(location)) {
+ try {
+ Path dbLocation = new Path(client.getDatabase(tablePath.getDatabaseName()).getLocationUri());
+ location = new Path(dbLocation, tablePath.getObjectName()).toString();
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to infer hoodie table path for table %s", tablePath), e);
+ }
+ }
+ return location;
+ }
+
+ private Map<String, String> applyOptionsHook(Map<String, String> options) {
+ Map<String, String> properties = new HashMap<>(options);
+ if (!options.containsKey(FlinkOptions.RECORD_KEY_FIELD.key())) {
+ properties.put(FlinkOptions.RECORD_KEY_FIELD.key(), FlinkOptions.RECORD_KEY_FIELD.defaultValue());
+ }
+ if (!options.containsKey(FlinkOptions.PRECOMBINE_FIELD.key())) {
+ properties.put(FlinkOptions.PRECOMBINE_FIELD.key(), FlinkOptions.PRECOMBINE_FIELD.defaultValue());
+ }
+ if (!options.containsKey(FlinkOptions.TABLE_TYPE.key())) {
+ properties.put(FlinkOptions.TABLE_TYPE.key(), FlinkOptions.TABLE_TYPE.defaultValue());
+ }
+ return properties;
+ }
+
+ private Table instantiateHiveTable(ObjectPath tablePath, CatalogBaseTable table, String location, boolean useRealTimeInputFormat) throws IOException {
+ // let Hive set default parameters for us, e.g. serialization.format
+ Table hiveTable =
+ org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(
+ tablePath.getDatabaseName(), tablePath.getObjectName());
+ hiveTable.setOwner(UserGroupInformation.getCurrentUser().getUserName());
+ hiveTable.setCreateTime((int) (System.currentTimeMillis() / 1000));
+
+ Map<String, String> properties = applyOptionsHook(table.getOptions());
+ properties.put("EXTERNAL", "TRUE");
Review Comment:
If it's better for user to customize the table type of "EXTERNAL" or "MANAGED"?
Using flink engine and hudi catalog, if we drop the table, the hdfs dir will be deleted.
Using spark engine, if we set 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog', spark create the "MANAGED" table by default in HMS. And if we drop the table, the hdfs dir will be deleted.
I think if we can customize the table type of "EXTERNAL" or "MANAGED", we can control the hdfs dir will be deleted or not.
@danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 closed pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
cuibo01 closed pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
URL: https://github.com/apache/hudi/pull/6013
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1181158606
https://github.com/apache/hudi/pull/6082
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiaozhch5 commented on pull request #6013: [WIP][HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
xiaozhch5 commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1172886885
Hello, I test the PR, but the following problem occurs.
compile command:
```bash
mvn clean install -DskipTests -Dhive.version=3.1.2 -Dflink.version=1.13.6 -Pflink1.13 -Dhadoop.version=3.2.2 -Pspark3 -Pflink-bundle-shade-hive3 -Dscala.binary.version=2.12 -Dscala.version=2.12.12
```
The full stacks are list below,
```java
2022-07-02 19:37:04,770 INFO org.apache.hudi.table.catalog.HoodieCatalogUtil [] - Setting hive conf dir as /data/hive/hive/conf
2022-07-02 19:37:04,776 INFO org.apache.hudi.table.catalog.HoodieHiveCatalog [] - Created HiveCatalog 'hudi'
2022-07-02 19:37:04,777 INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient [] - Trying to connect to metastore with URI thrift://bigdata:9083
2022-07-02 19:37:04,778 INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient [] - Opened a connection to metastore, current connections: 1
2022-07-02 19:37:04,802 INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient [] - Connected to metastore.
2022-07-02 19:37:04,802 INFO org.apache.hadoop.hive.metastore.RetryingMetaStoreClient [] - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0
2022-07-02 19:37:04,933 INFO org.apache.hudi.table.catalog.HoodieHiveCatalog [] - Connected to Hive metastore
2022-07-02 19:37:19,532 INFO org.apache.flink.table.catalog.CatalogManager [] - Set the current default catalog as [hudi] and the current default database as [default].
2022-07-02 19:37:47,587 INFO org.apache.flink.table.catalog.CatalogManager [] - Set the current default database as [hudidb] in the current default catalog [hudi].
2022-07-02 19:38:29,533 WARN org.apache.flink.table.client.cli.CliClient [] - Could not execute SQL statement.
org.apache.flink.table.client.gateway.SqlExecutionException: Could not execute SQL statement.
at org.apache.flink.table.client.gateway.local.LocalExecutor.executeOperation(LocalExecutor.java:215) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.cli.CliClient.executeOperation(CliClient.java:564) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.cli.CliClient.callOperation(CliClient.java:424) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.cli.CliClient.lambda$executeStatement$0(CliClient.java:327) [flink-sql-client_2.12-1.13.6.jar:1.13.6]
at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_241]
at org.apache.flink.table.client.cli.CliClient.executeStatement(CliClient.java:327) [flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.cli.CliClient.executeInteractive(CliClient.java:297) [flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:221) [flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:151) [flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:95) [flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187) [flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161) [flink-sql-client_2.12-1.13.6.jar:1.13.6]
Caused by: org.apache.flink.table.api.TableException: Could not execute CreateTable in path `hudi`.`hudidb`.`orders_hudi`
at org.apache.flink.table.catalog.CatalogManager.execute(CatalogManager.java:847) ~[flink-table_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.catalog.CatalogManager.createTable(CatalogManager.java:659) ~[flink-table_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:865) ~[flink-table_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeOperation$3(LocalExecutor.java:213) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.gateway.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:90) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.gateway.local.LocalExecutor.executeOperation(LocalExecutor.java:213) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
... 11 more
Caused by: org.apache.hudi.exception.HoodieCatalogException: Failed to create table hudidb.orders_hudi
at org.apache.hudi.table.catalog.HoodieHiveCatalog.createTable(HoodieHiveCatalog.java:446) ~[hudi-flink1.13-bundle_2.12-0.12.0-SNAPSHOT.jar:0.12.0-SNAPSHOT]
at org.apache.flink.table.catalog.CatalogManager.lambda$createTable$10(CatalogManager.java:661) ~[flink-table_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.catalog.CatalogManager.execute(CatalogManager.java:841) ~[flink-table_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.catalog.CatalogManager.createTable(CatalogManager.java:659) ~[flink-table_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:865) ~[flink-table_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeOperation$3(LocalExecutor.java:213) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.gateway.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:90) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.gateway.local.LocalExecutor.executeOperation(LocalExecutor.java:213) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
... 11 more
Caused by: org.apache.hudi.exception.HoodieCatalogException: id and uuid are the different
at org.apache.hudi.table.catalog.HoodieHiveCatalog.instantiateHiveTable(HoodieHiveCatalog.java:579) ~[hudi-flink1.13-bundle_2.12-0.12.0-SNAPSHOT.jar:0.12.0-SNAPSHOT]
at org.apache.hudi.table.catalog.HoodieHiveCatalog.createTable(HoodieHiveCatalog.java:435) ~[hudi-flink1.13-bundle_2.12-0.12.0-SNAPSHOT.jar:0.12.0-SNAPSHOT]
at org.apache.flink.table.catalog.CatalogManager.lambda$createTable$10(CatalogManager.java:661) ~[flink-table_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.catalog.CatalogManager.execute(CatalogManager.java:841) ~[flink-table_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.catalog.CatalogManager.createTable(CatalogManager.java:659) ~[flink-table_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:865) ~[flink-table_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeOperation$3(LocalExecutor.java:213) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.gateway.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:90) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
at org.apache.flink.table.client.gateway.local.LocalExecutor.executeOperation(LocalExecutor.java:213) ~[flink-sql-client_2.12-1.13.6.jar:1.13.6]
... 11 more
```
But it's ok if I use hudi catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on pull request #6013: [WIP][HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1173013058
> It's ok when I change the primary key to uuid or define the parameter of 'hoodie.datasource.write.recordkey.field'.
>
> But another problem arouse when I query the table in spark sql catalog.
👍 yes flink config was incorrectly used as a string, now u can retry it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiaozhch5 commented on pull request #6013: [WIP][HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
xiaozhch5 commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1173111069
> > It's ok when I change the primary key to uuid or define the parameter of 'hoodie.datasource.write.recordkey.field'.
> > But another problem arouse when I query the table in spark sql catalog.
>
> 👍 yes flink config was incorrectly used as a string, now u can retry it
Yeah, the problem is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179798835
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
}, {
"hash" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808",
"triggerID" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"triggerType" : "PUSH"
}, {
"hash" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9819",
"triggerID" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"triggerType" : "PUSH"
}, {
"hash" : "9619231aabbd36a02215098f1b329aa1ef44a581",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9619231aabbd36a02215098f1b329aa1ef44a581",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9819)
* 9619231aabbd36a02215098f1b329aa1ef44a581 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179313622
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 0c113baae795ddf7f884b868dcde0ddc3f8adb92 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1178499813
> @danny0405 could you pls help review this pr. thanks we will solve the comments as soon as possible.
You should not merge the branch, the code is a mess now, please rebase the patch with latest master and force push to the branch then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1178552069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759)
* c5051b524f5e241103afbdd9394d0a14a660c3bd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r916511643
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieCatalogFactoryOptions.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.table.catalog.CommonCatalogOptions;
+
+/** {@link ConfigOption}s for {@link HoodieHiveCatalog}. */
+public class HoodieCatalogFactoryOptions {
+ public static final String IDENTIFIER = "hudi-hive";
+ public static final String DEFAULT_DB = "default";
+ public static final String HIVE_SITE_FILE = "hive-site.xml";
+
+ public static final ConfigOption<String> DEFAULT_DATABASE =
+ ConfigOptions.key(CommonCatalogOptions.DEFAULT_DATABASE_KEY)
+ .stringType()
+ .defaultValue(DEFAULT_DB);
+
+ public static final ConfigOption<String> HIVE_CONF_DIR =
+ ConfigOptions.key("hive-conf-dir").stringType().noDefaultValue();
+
+ public static final ConfigOption<String> HADOOP_CONF_DIR =
+ ConfigOptions.key("hadoop-conf-dir").stringType().noDefaultValue();
Review Comment:
Not sure if we should give an option for hadoop conf dir, people usually fetch the path through the env `HADOOP_CONF_DIR` or `HADOOP_HOME/conf`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiaozhch5 commented on pull request #6013: [WIP][HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
xiaozhch5 commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1173004258
> > Hello, I test the PR, but the following problem occurs.
>
> thx for your review. If the primary key exists, it must be the same as the `RECORD_KEY_FIELD`. 
It's ok when I change the primary key to uuid or define the parameter of 'hoodie.datasource.write.recordkey.field'.
But another problem arouse when I query the table in spark sql catalog.
The table type I defined is MERGE_ON_READ, but when I use the command of "desc formatted xxx", It shows the table type is COPY_ON_WRITE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r917097007
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf), false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created Hoodie Catalog '{}' in hms mode", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
+ parameters.put(PATH.key(), path);
+ if (!parameters.containsKey(FlinkOptions.HIVE_STYLE_PARTITIONING.key())) {
+ Path hoodieTablePath = new Path(path);
+ boolean hiveStyle = Arrays.stream(FSUtils.getFs(hoodieTablePath, hiveConf).listStatus(hoodieTablePath))
+ .map(fileStatus -> fileStatus.getPath().getName())
+ .filter(f -> !f.equals(".hoodie") && !f.equals("default"))
+ .anyMatch(FilePathUtils::isHiveStylePartitioning);
+ parameters.put(FlinkOptions.HIVE_STYLE_PARTITIONING.key(), String.valueOf(hiveStyle));
+ }
+ client.alter_table(tablePath.getDatabaseName(), tablePath.getObjectName(), hiveTable);
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to update table schema", e);
+ }
+ }
+ return hiveTable;
+ }
+
+ @Override
+ public CatalogBaseTable getTable(ObjectPath tablePath) throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ Table hiveTable = getHiveTable(tablePath);
+ hiveTable = translateSparkTable2Flink(tablePath, hiveTable);
+ String path = hiveTable.getSd().getLocation();
+ Map<String, String> parameters = hiveTable.getParameters();
+ Schema latestTableSchema = getLatestTableSchema(path);
+ org.apache.flink.table.api.Schema schema;
+ if (latestTableSchema != null) {
+ org.apache.flink.table.api.Schema.Builder builder = org.apache.flink.table.api.Schema.newBuilder()
+ .fromRowDataType(AvroSchemaConverter.convertToDataType(latestTableSchema));
+ String pkConstraintName = parameters.get(PK_CONSTRAINT_NAME);
+ if (!StringUtils.isNullOrEmpty(pkConstraintName)) {
+ builder.primaryKeyNamed(pkConstraintName, StringUtils.split(parameters.get(PK_COLUMNS), ","));
+ }
+ schema = builder.build();
+ } else {
+ LOG.warn("{} does not have any hoodie schema, and use hive table schema to infer the table schema", tablePath);
+ schema = TableOptionProperties.convertTableSchema(hiveTable);
+ }
+ return CatalogTable.of(schema, parameters.get(COMMENT),
+ TableOptionProperties.getFieldNames(hiveTable.getPartitionKeys()), parameters);
+ }
+
+ @Override
+ public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
+ throws TableAlreadyExistException, DatabaseNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkNotNull(table, "table cannot be null");
+
+ if (!databaseExists(tablePath.getDatabaseName())) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName());
+ }
+
+ if (!table.getOptions().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("The %s is not hoodie table", tablePath.getObjectName()));
+ }
+
+ if (table instanceof CatalogView) {
+ throw new HoodieCatalogException("Hoodie catalog does not support to CREATE VIEW.");
+ }
+
+ try {
+ boolean isMorTable = table.getOptions().getOrDefault(FlinkOptions.TABLE_TYPE.key(),
+ FlinkOptions.TABLE_TYPE.defaultValue()).equalsIgnoreCase(FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
+ Table hiveTable = instantiateHiveTable(tablePath, table, inferTablePath(tablePath, table), isMorTable);
+ //create hive table
+ client.createTable(hiveTable);
+ //init hoodie metaClient
+ initTableIfNotExists(tablePath, (CatalogTable)table);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new TableAlreadyExistException(getName(), tablePath, e);
+ }
+ } catch (Exception e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create table %s", tablePath.getFullName()), e);
+ }
+ }
+
+ private void initTableIfNotExists(ObjectPath tablePath, CatalogTable catalogTable) {
+ Configuration flinkConf = Configuration.fromMap(applyOptionsHook(catalogTable.getOptions()));
+ flinkConf.addAllToProperties(hiveConf.getAllProperties());
+ final String avroSchema = AvroSchemaConverter.convertToSchema(catalogTable.getSchema().toPhysicalRowDataType().getLogicalType()).toString();
+ flinkConf.setString(FlinkOptions.SOURCE_AVRO_SCHEMA, avroSchema);
+
+ // stores two copies of options:
+ // - partition keys
+ // - primary keys
+ // because the HoodieTableMetaClient is a heavy impl, we try to avoid initializing it
+ // when calling #getTable.
+
+ if (catalogTable.getUnresolvedSchema().getPrimaryKey().isPresent()) {
+ final String pkColumns = String.join(",", catalogTable.getUnresolvedSchema().getPrimaryKey().get().getColumnNames());
+ String recordKey = flinkConf.get(FlinkOptions.RECORD_KEY_FIELD);
+ if (!Objects.equals(pkColumns, recordKey)) {
+ throw new HoodieCatalogException(String.format("%s and %s are the different", pkColumns, recordKey));
+ }
+ }
+
+ if (catalogTable.isPartitioned()) {
+ final String partitions = String.join(",", catalogTable.getPartitionKeys());
+ flinkConf.setString(FlinkOptions.PARTITION_PATH_FIELD, partitions);
+ }
+
+ if (!flinkConf.getOptional(PATH).isPresent()) {
+ flinkConf.setString(PATH, inferTablePath(tablePath, catalogTable));
+ }
+
+ flinkConf.setString(FlinkOptions.TABLE_NAME, tablePath.getObjectName());
+ try {
+ StreamerUtil.initTableIfNotExists(flinkConf);
+ } catch (IOException e) {
+ throw new HoodieCatalogException("Initialize table exception.", e);
+ }
+ }
+
+ private String inferTablePath(ObjectPath tablePath, CatalogBaseTable table) {
+ String location = table.getOptions().getOrDefault(PATH.key(), "");
+ if (StringUtils.isNullOrEmpty(location)) {
+ try {
+ Path dbLocation = new Path(client.getDatabase(tablePath.getDatabaseName()).getLocationUri());
+ location = new Path(dbLocation, tablePath.getObjectName()).toString();
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to infer hoodie table path for table %s", tablePath), e);
+ }
+ }
+ return location;
+ }
+
+ private Map<String, String> applyOptionsHook(Map<String, String> options) {
+ Map<String, String> properties = new HashMap<>(options);
+ if (!options.containsKey(FlinkOptions.RECORD_KEY_FIELD.key())) {
+ properties.put(FlinkOptions.RECORD_KEY_FIELD.key(), FlinkOptions.RECORD_KEY_FIELD.defaultValue());
+ }
+ if (!options.containsKey(FlinkOptions.PRECOMBINE_FIELD.key())) {
+ properties.put(FlinkOptions.PRECOMBINE_FIELD.key(), FlinkOptions.PRECOMBINE_FIELD.defaultValue());
+ }
+ if (!options.containsKey(FlinkOptions.TABLE_TYPE.key())) {
+ properties.put(FlinkOptions.TABLE_TYPE.key(), FlinkOptions.TABLE_TYPE.defaultValue());
+ }
+ return properties;
+ }
+
+ private Table instantiateHiveTable(ObjectPath tablePath, CatalogBaseTable table, String location, boolean useRealTimeInputFormat) throws IOException {
+ // let Hive set default parameters for us, e.g. serialization.format
+ Table hiveTable =
+ org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(
+ tablePath.getDatabaseName(), tablePath.getObjectName());
+ hiveTable.setOwner(UserGroupInformation.getCurrentUser().getUserName());
+ hiveTable.setCreateTime((int) (System.currentTimeMillis() / 1000));
+
+ Map<String, String> properties = applyOptionsHook(table.getOptions());
+ properties.put("EXTERNAL", "TRUE");
+ // Table comment
+ if (table.getComment() != null) {
+ properties.put(COMMENT, table.getComment());
+ }
+
+ //set sd
+ StorageDescriptor sd = new StorageDescriptor();
+ List<FieldSchema> allColumns = TableOptionProperties.createHiveColumns(table.getSchema());
+
+ // Table columns and partition keys
+ if (table instanceof CatalogTable) {
+ CatalogTable catalogTable = (CatalogTable) table;
+
+ if (catalogTable.isPartitioned()) {
+ int partitionKeySize = catalogTable.getPartitionKeys().size();
+ List<FieldSchema> regularColumns =
+ allColumns.subList(0, allColumns.size() - partitionKeySize);
+ List<FieldSchema> partitionColumns =
+ allColumns.subList(
+ allColumns.size() - partitionKeySize, allColumns.size());
+
+ sd.setCols(regularColumns);
+ hiveTable.setPartitionKeys(partitionColumns);
+ } else {
+ sd.setCols(allColumns);
+ hiveTable.setPartitionKeys(new ArrayList<>());
+ }
+ } else {
+ sd.setCols(allColumns);
+ }
+
+ HoodieFileFormat baseFileFormat = HoodieFileFormat.PARQUET;
+ //ignore uber input Format
+ String inputFormatClassName = HoodieInputFormatUtils.getInputFormatClassName(baseFileFormat, useRealTimeInputFormat);
+ String outputFormatClassName = HoodieInputFormatUtils.getOutputFormatClassName(baseFileFormat);
+ String serDeClassName = HoodieInputFormatUtils.getSerDeClassName(baseFileFormat);
+ sd.setInputFormat(inputFormatClassName);
+ sd.setOutputFormat(outputFormatClassName);
+ Map<String, String> serdeProperties = new HashMap<>();
+ serdeProperties.put("path", location);
+ serdeProperties.put(ConfigUtils.IS_QUERY_AS_RO_TABLE, String.valueOf(!useRealTimeInputFormat));
+ serdeProperties.put("serialization.format", "1");
+ sd.setSerdeInfo(new SerDeInfo(null, serDeClassName, serdeProperties));
+
+ sd.setLocation(location);
+
+ hiveTable.setSd(sd);
+
+ properties.putAll(TableOptionProperties.translateFlinkTableProperties2Spark((CatalogTable)table, hiveConf));
+
+ //set pk
+ if (table.getUnresolvedSchema().getPrimaryKey().isPresent()) {
+ String pkColumns = String.join(",", table.getUnresolvedSchema().getPrimaryKey().get().getColumnNames());
+ String recordKey = properties.getOrDefault(FlinkOptions.RECORD_KEY_FIELD.key(), FlinkOptions.RECORD_KEY_FIELD.defaultValue());
+ if (!Objects.equals(pkColumns, recordKey)) {
+ throw new HoodieCatalogException(
+ String.format("If the table has primaryKey, the primaryKey should be the the same as the recordKey, but pk %s and recordKey %s are the different",
+ pkColumns,
+ recordKey));
+ }
+ properties.put(PK_CONSTRAINT_NAME, table.getUnresolvedSchema().getPrimaryKey().get().getConstraintName());
+ properties.put(PK_COLUMNS, pkColumns);
+ }
+
+ if (!properties.containsKey(FlinkOptions.PATH.key())) {
+ properties.put(FlinkOptions.PATH.key(), location);
+ }
+
+ hiveTable.setParameters(properties);
+ return hiveTable;
+ }
+
+ @Override
+ public List<String> listTables(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+
+ try {
+ return client.getAllTables(databaseName);
+ } catch (UnknownDBException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list tables in database %s", databaseName), e);
+ }
+ }
+
+ @Override
+ public List<String> listViews(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ throw new HoodieCatalogException("Hoodie catalog does not support to listViews");
+ }
+
+ @Override
+ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+
+ try {
+ return client.tableExists(tablePath.getDatabaseName(), tablePath.getObjectName());
+ } catch (UnknownDBException e) {
+ return false;
+ } catch (TException e) {
+ throw new CatalogException(
+ String.format(
+ "Failed to check whether table %s exists or not.",
+ tablePath.getFullName()),
+ e);
+ }
+ }
+
+ @Override
+ public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
+ throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+
+ try {
+ client.dropTable(
+ tablePath.getDatabaseName(),
+ tablePath.getObjectName(),
+ // Indicate whether associated data should be deleted.
+ // Set to 'true' for now because Flink tables shouldn't have data in Hive. Can
+ // be changed later if necessary
+ true,
+ ignoreIfNotExists);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new TableNotExistException(getName(), tablePath);
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to drop table %s", tablePath.getFullName()), e);
+ }
+ }
+
+ @Override
+ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignoreIfNotExists)
+ throws TableNotExistException, TableAlreadyExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkArgument(
+ !isNullOrWhitespaceOnly(newTableName), "newTableName cannot be null or empty");
+
+ try {
+ // alter_table() doesn't throw a clear exception when target table doesn't exist.
+ // Thus, check the table existence explicitly
+ if (tableExists(tablePath)) {
+ ObjectPath newPath = new ObjectPath(tablePath.getDatabaseName(), newTableName);
+ // alter_table() doesn't throw a clear exception when new table already exists.
+ // Thus, check the table existence explicitly
+ if (tableExists(newPath)) {
+ throw new TableAlreadyExistException(getName(), newPath);
+ } else {
+ Table hiveTable = getHiveTable(tablePath);
+
+ //update hoodie
+ StorageDescriptor sd = hiveTable.getSd();
+ String location = sd.getLocation();
+ HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setBasePath(location).setConf(hiveConf).build();
+ //Init table with new name
+ HoodieTableMetaClient.withPropertyBuilder().fromProperties(metaClient.getTableConfig().getProps())
+ .setTableName(newTableName)
+ .initTable(hiveConf, location);
+
+ hiveTable.setTableName(newTableName);
+ client.alter_table(
+ tablePath.getDatabaseName(), tablePath.getObjectName(), hiveTable);
+ }
+ } else if (!ignoreIfNotExists) {
+ throw new TableNotExistException(getName(), tablePath);
+ }
+ } catch (Exception e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to rename table %s", tablePath.getFullName()), e);
+ }
+ }
+
+ @Override
+ public void alterTable(
+ ObjectPath tablePath, CatalogBaseTable newCatalogTable, boolean ignoreIfNotExists)
+ throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkNotNull(newCatalogTable, "newCatalogTable cannot be null");
+
+ if (!newCatalogTable.getOptions().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("The %s is not hoodie table", tablePath.getObjectName()));
+ }
+ if (newCatalogTable instanceof CatalogView) {
+ throw new HoodieCatalogException("Hoodie catalog does not support to ALTER VIEW");
+ }
+
+ try {
+ Table hiveTable = getHiveTable(tablePath);
+ } catch (TableNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw e;
+ }
+ return;
+ }
+ try {
+ boolean isMorTable = newCatalogTable.getOptions().getOrDefault(FlinkOptions.TABLE_TYPE.key(),
+ FlinkOptions.TABLE_TYPE.defaultValue()).equalsIgnoreCase(FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
Review Comment:
let‘s forbid follow behavior
1) forbidden change hoodie table type . eg: mor -> cow /cow -> mor
2) if 'index.type' = 'BUCKET' is set already set for hoodie table. forbidden change this property， since bucket index is incompatible with other hoodie index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1177288897
@danny0405 could you pls help review this pr. thanks
we will solve the comments as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1177440889
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1177425298
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r917361867
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,907 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private final boolean iniTable;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf, boolean iniTable) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf), iniTable, false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean iniTable, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ this.iniTable = iniTable;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created Hoodie Catalog '{}' in hms mode", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
+ parameters.put(PATH.key(), path);
+ if (!parameters.containsKey(FlinkOptions.HIVE_STYLE_PARTITIONING.key())) {
+ Path hoodieTablePath = new Path(path);
+ boolean hiveStyle = Arrays.stream(FSUtils.getFs(hoodieTablePath, hiveConf).listStatus(hoodieTablePath))
+ .map(fileStatus -> fileStatus.getPath().getName())
+ .filter(f -> !f.equals(".hoodie") && !f.equals("default"))
+ .anyMatch(FilePathUtils::isHiveStylePartitioning);
+ parameters.put(FlinkOptions.HIVE_STYLE_PARTITIONING.key(), String.valueOf(hiveStyle));
+ }
+ client.alter_table(tablePath.getDatabaseName(), tablePath.getObjectName(), hiveTable);
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to update table schema", e);
+ }
+ }
+ return hiveTable;
+ }
+
+ @Override
+ public CatalogBaseTable getTable(ObjectPath tablePath) throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ Table hiveTable = getHiveTable(tablePath);
+ hiveTable = translateSparkTable2Flink(tablePath, hiveTable);
+ String path = hiveTable.getSd().getLocation();
+ Map<String, String> parameters = hiveTable.getParameters();
+ Schema latestTableSchema = getLatestTableSchema(path);
+ org.apache.flink.table.api.Schema schema;
+ if (latestTableSchema != null) {
+ org.apache.flink.table.api.Schema.Builder builder = org.apache.flink.table.api.Schema.newBuilder()
+ .fromRowDataType(AvroSchemaConverter.convertToDataType(latestTableSchema));
+ String pkConstraintName = parameters.get(PK_CONSTRAINT_NAME);
+ if (!StringUtils.isNullOrEmpty(pkConstraintName)) {
+ builder.primaryKeyNamed(pkConstraintName, StringUtils.split(parameters.get(PK_COLUMNS), ","));
+ }
+ schema = builder.build();
+ } else {
+ LOG.warn("{} does not have any hoodie schema, and use hive table schema to infer the table schema", tablePath);
+ schema = TableOptionProperties.convertTableSchema(hiveTable);
+ }
+ return CatalogTable.of(schema, parameters.get(COMMENT),
+ TableOptionProperties.getFieldNames(hiveTable.getPartitionKeys()), parameters);
+ }
+
+ @Override
+ public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
+ throws TableAlreadyExistException, DatabaseNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkNotNull(table, "table cannot be null");
+
+ if (!databaseExists(tablePath.getDatabaseName())) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName());
+ }
+
+ if (!table.getOptions().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("The %s is not hoodie table", tablePath.getObjectName()));
+ }
+
+ if (table instanceof CatalogView) {
+ throw new HoodieCatalogException("Hoodie catalog does not support to CREATE VIEW.");
+ }
+
+ try {
+ boolean isMorTable = table.getOptions().getOrDefault(FlinkOptions.TABLE_TYPE.key(),
+ FlinkOptions.TABLE_TYPE.defaultValue()).equalsIgnoreCase(FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
+ Table hiveTable = instantiateHiveTable(tablePath, table, inferTablePath(tablePath, table), isMorTable);
+ //create hive table
+ client.createTable(hiveTable);
+ //init hoodie metaClient
+ initTableIfNotExists(tablePath, (CatalogTable)table);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new TableAlreadyExistException(getName(), tablePath, e);
+ }
+ } catch (Exception e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create table %s", tablePath.getFullName()), e);
+ }
+ }
+
+ private void initTableIfNotExists(ObjectPath tablePath, CatalogTable catalogTable) {
+ if (!iniTable) {
+ LOG.info("Skip init table.");
+ return;
+ }
+ Configuration flinkConf = Configuration.fromMap(applyOptionsHook(catalogTable.getOptions()));
+ flinkConf.addAllToProperties(hiveConf.getAllProperties());
+ final String avroSchema = AvroSchemaConverter.convertToSchema(catalogTable.getSchema().toPhysicalRowDataType().getLogicalType()).toString();
+ flinkConf.setString(FlinkOptions.SOURCE_AVRO_SCHEMA, avroSchema);
+
+ // stores two copies of options:
+ // - partition keys
+ // - primary keys
+ // because the HoodieTableMetaClient is a heavy impl, we try to avoid initializing it
+ // when calling #getTable.
+
+ if (catalogTable.getUnresolvedSchema().getPrimaryKey().isPresent()) {
+ final String pkColumns = String.join(",", catalogTable.getUnresolvedSchema().getPrimaryKey().get().getColumnNames());
+ String recordKey = flinkConf.get(FlinkOptions.RECORD_KEY_FIELD);
+ if (!Objects.equals(pkColumns, recordKey)) {
+ throw new HoodieCatalogException(String.format("%s and %s are the different", pkColumns, recordKey));
+ }
+ }
+
+ if (catalogTable.isPartitioned()) {
+ final String partitions = String.join(",", catalogTable.getPartitionKeys());
+ flinkConf.setString(FlinkOptions.PARTITION_PATH_FIELD, partitions);
+ }
+
+ if (!flinkConf.getOptional(PATH).isPresent()) {
+ flinkConf.setString(PATH, inferTablePath(tablePath, catalogTable));
+ }
+
+ flinkConf.setString(FlinkOptions.TABLE_NAME, tablePath.getObjectName());
+ try {
+ StreamerUtil.initTableIfNotExists(flinkConf);
+ } catch (IOException e) {
+ throw new HoodieCatalogException("Initialize table exception.", e);
+ }
+ }
+
+ private String inferTablePath(ObjectPath tablePath, CatalogBaseTable table) {
+ String location = table.getOptions().getOrDefault(PATH.key(), "");
+ if (StringUtils.isNullOrEmpty(location)) {
+ try {
+ Path dbLocation = new Path(client.getDatabase(tablePath.getDatabaseName()).getLocationUri());
+ location = new Path(dbLocation, tablePath.getObjectName()).toString();
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to infer hoodie table path for table %s", tablePath), e);
+ }
+ }
+ return location;
+ }
+
+ private Map<String, String> applyOptionsHook(Map<String, String> options) {
+ Map<String, String> properties = new HashMap<>(options);
+ if (!options.containsKey(FlinkOptions.RECORD_KEY_FIELD.key())) {
+ properties.put(FlinkOptions.RECORD_KEY_FIELD.key(), FlinkOptions.RECORD_KEY_FIELD.defaultValue());
+ }
+ if (!options.containsKey(FlinkOptions.PRECOMBINE_FIELD.key())) {
+ properties.put(FlinkOptions.PRECOMBINE_FIELD.key(), FlinkOptions.PRECOMBINE_FIELD.defaultValue());
+ }
+ if (!options.containsKey(FlinkOptions.TABLE_TYPE.key())) {
+ properties.put(FlinkOptions.TABLE_TYPE.key(), FlinkOptions.TABLE_TYPE.defaultValue());
+ }
+ return properties;
+ }
+
+ private Table instantiateHiveTable(ObjectPath tablePath, CatalogBaseTable table, String location, boolean useRealTimeInputFormat) throws IOException {
+ // let Hive set default parameters for us, e.g. serialization.format
+ Table hiveTable =
+ org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(
+ tablePath.getDatabaseName(), tablePath.getObjectName());
+ hiveTable.setOwner(UserGroupInformation.getCurrentUser().getUserName());
+ hiveTable.setCreateTime((int) (System.currentTimeMillis() / 1000));
+
+ Map<String, String> properties = applyOptionsHook(table.getOptions());
+ properties.put("EXTERNAL", "TRUE");
Review Comment:
OS Flink supports only `EXTERNAL`, does HUDI need to support `MANAGED` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179900840
Can we squash the commits locally and force push the branch, there are 10 commits on the branch and it's hard to review in total for the details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1178549891
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759)
* c5051b524f5e241103afbdd9394d0a14a660c3bd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiaozhch5 commented on a diff in pull request #6013: [WIP][HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
xiaozhch5 commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r912508833
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/TableOptionProperties.java:
##########
@@ -53,9 +81,26 @@ public class TableOptionProperties {
public static final String PK_COLUMNS = "pk.columns";
public static final String COMMENT = "comment";
public static final String PARTITION_COLUMNS = "partition.columns";
+ public static final String SPARK_SOURCE_PROVIDER = "spark.sql.source.provider";
Review Comment:
SPARK_SOURCE_PROVIDER should be "spark.sql.sources.provider"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1177353078
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r916826110
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf, String hadoopConf) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf, hadoopConf), false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created HiveCatalog '{}'", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
+ parameters.put(PATH.key(), path);
+ if (!parameters.containsKey(FlinkOptions.HIVE_STYLE_PARTITIONING.key())) {
+ Path hoodieTablePath = new Path(path);
+ boolean hiveStyle = Arrays.stream(FSUtils.getFs(hoodieTablePath, hiveConf).listStatus(hoodieTablePath))
+ .map(fileStatus -> fileStatus.getPath().getName())
+ .filter(f -> !f.equals(".hoodie") && !f.equals("default"))
+ .anyMatch(FilePathUtils::hiveStylePartitionMath);
+ parameters.put(FlinkOptions.HIVE_STYLE_PARTITIONING.key(), String.valueOf(hiveStyle));
+ }
+ client.alter_table(tablePath.getDatabaseName(), tablePath.getObjectName(), hiveTable);
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to update table schema", e);
+ }
+ }
+ return hiveTable;
+ }
+
+ @Override
+ public CatalogBaseTable getTable(ObjectPath tablePath) throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ Table hiveTable = getHiveTable(tablePath);
+ hiveTable = translateSparkTable2Flink(tablePath, hiveTable);
+ String path = hiveTable.getSd().getLocation();
+ Map<String, String> parameters = hiveTable.getParameters();
+ Schema latestTableSchema = getLatestTableSchema(path);
+ org.apache.flink.table.api.Schema schema;
+ if (latestTableSchema != null) {
+ org.apache.flink.table.api.Schema.Builder builder = org.apache.flink.table.api.Schema.newBuilder()
+ .fromRowDataType(AvroSchemaConverter.convertToDataType(latestTableSchema));
+ String pkConstraintName = parameters.get(PK_CONSTRAINT_NAME);
+ if (!StringUtils.isNullOrEmpty(pkConstraintName)) {
+ builder.primaryKeyNamed(pkConstraintName, StringUtils.split(parameters.get(PK_COLUMNS), ","));
+ }
+ schema = builder.build();
+ } else {
+ LOG.warn("{} does not have any hoodie schema, and use hive table to covert the catalogBaseTable", tablePath);
+ schema = TableOptionProperties.convertTableSchema(hiveTable);
+ }
+ return CatalogTable.of(schema, parameters.get(COMMENT),
+ TableOptionProperties.getFieldNames(hiveTable.getPartitionKeys()), parameters);
+ }
+
+ @Override
+ public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
+ throws TableAlreadyExistException, DatabaseNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkNotNull(table, "table cannot be null");
+
+ if (!databaseExists(tablePath.getDatabaseName())) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName());
+ }
+
+ if (!table.getOptions().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("The %s is not hoodie table", tablePath.getObjectName()));
+ }
+
+ if (table instanceof CatalogView) {
+ throw new HoodieCatalogException("Hoodie catalog does not support to CREATE VIEW.");
+ }
+
+ try {
+ boolean isMorTable = table.getOptions().getOrDefault(FlinkOptions.TABLE_TYPE.key(),
+ FlinkOptions.TABLE_TYPE.defaultValue()).equalsIgnoreCase(FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
Review Comment:
but `table.getOptions()` does not contain `TABLE_TYPE`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on a diff in pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r916825060
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf, String hadoopConf) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf, hadoopConf), false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created HiveCatalog '{}'", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
Review Comment:
the table is a Spark table but not a Flink table, so we need to refresh the table to the hms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1177295117
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1177447791
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179799491
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
}, {
"hash" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808",
"triggerID" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"triggerType" : "PUSH"
}, {
"hash" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9819",
"triggerID" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"triggerType" : "PUSH"
}, {
"hash" : "9619231aabbd36a02215098f1b329aa1ef44a581",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9824",
"triggerID" : "9619231aabbd36a02215098f1b329aa1ef44a581",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9819)
* 9619231aabbd36a02215098f1b329aa1ef44a581 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9824)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cuibo01 commented on pull request #6013: [WIP][HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
cuibo01 commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1172889336
> Hello, I test the PR, but the following problem occurs.
>
thx for your review. If the primary key exists, it must be the same as the `RECORD_KEY_FIELD`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1178563645
Instead of using command merge, you should use rebase, merge woulg generate a new commit on the commit history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1178996622
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5051b524f5e241103afbdd9394d0a14a660c3bd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779)
* 8032fcaf15f8b246c8652ed5df49fa01564e2649 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790)
* 185ec5470953fc3420fb85c8d72c40bbf317f995 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179818445
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
}, {
"hash" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808",
"triggerID" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"triggerType" : "PUSH"
}, {
"hash" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9819",
"triggerID" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"triggerType" : "PUSH"
}, {
"hash" : "9619231aabbd36a02215098f1b329aa1ef44a581",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9824",
"triggerID" : "9619231aabbd36a02215098f1b329aa1ef44a581",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 9619231aabbd36a02215098f1b329aa1ef44a581 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9824)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r916509043
##########
hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java:
##########
@@ -261,6 +262,11 @@ private MessageType getTableParquetSchemaFromDataFile() {
}
}
+ public static MessageType convertAvroSchemaToParquetMessageType(Schema schema, Configuration hadoopConf) {
+ AvroSchemaConverter avroSchemaConverter = new AvroSchemaConverter(hadoopConf);
Review Comment:
`convertAvroSchemaToParquetMessageType` -> `convertAvroSchemaToParquet`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on code in PR #6013:
URL: https://github.com/apache/hudi/pull/6013#discussion_r917090371
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.catalog;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.format.FilePathUtils;
+import org.apache.hudi.util.AvroSchemaConverter;
+import org.apache.hudi.util.StreamerUtil;
+
+import org.apache.avro.Schema;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase;
+import org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner;
+import org.apache.flink.sql.parser.hive.ddl.SqlCreateHiveDatabase;
+import org.apache.flink.table.catalog.AbstractCatalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogPartition;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.CatalogPropertiesUtil;
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.DatabaseAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException;
+import org.apache.flink.table.catalog.exceptions.DatabaseNotExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionAlreadyExistsException;
+import org.apache.flink.table.catalog.exceptions.PartitionNotExistException;
+import org.apache.flink.table.catalog.exceptions.PartitionSpecInvalidException;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotPartitionedException;
+import org.apache.flink.table.catalog.exceptions.TablePartitionedException;
+import org.apache.flink.table.catalog.stats.CatalogColumnStatistics;
+import org.apache.flink.table.catalog.stats.CatalogTableStatistics;
+import org.apache.flink.table.expressions.Expression;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.api.UnknownDBException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabase.ALTER_DATABASE_OP;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_NAME;
+import static org.apache.flink.sql.parser.hive.ddl.SqlAlterHiveDatabaseOwner.DATABASE_OWNER_TYPE;
+import static org.apache.flink.table.factories.FactoryUtil.CONNECTOR;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.StringUtils.isNullOrWhitespaceOnly;
+import static org.apache.hudi.configuration.FlinkOptions.PATH;
+import static org.apache.hudi.table.catalog.HoodieCatalogFactoryOptions.DEFAULT_DB;
+import static org.apache.hudi.table.catalog.TableOptionProperties.COMMENT;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_COLUMNS;
+import static org.apache.hudi.table.catalog.TableOptionProperties.PK_CONSTRAINT_NAME;
+import static org.apache.hudi.table.catalog.TableOptionProperties.SPARK_SOURCE_PROVIDER;
+
+/**
+ * A catalog implementation for Hoodie based on MetaStore.
+ */
+public class HoodieHiveCatalog extends AbstractCatalog {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieHiveCatalog.class);
+
+ private final HiveConf hiveConf;
+ private IMetaStoreClient client;
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, String hiveConf) {
+ this(catalogName, defaultDatabase, HoodieCatalogUtil.createHiveConf(hiveConf), false);
+ }
+
+ public HoodieHiveCatalog(String catalogName, String defaultDatabase, HiveConf hiveConf, boolean allowEmbedded) {
+ super(catalogName, defaultDatabase == null ? DEFAULT_DB : defaultDatabase);
+ this.hiveConf = hiveConf;
+ if (!allowEmbedded) {
+ checkArgument(
+ !HoodieCatalogUtil.isEmbeddedMetastore(this.hiveConf),
+ "Embedded metastore is not allowed. Make sure you have set a valid value for "
+ + HiveConf.ConfVars.METASTOREURIS.toString());
+ }
+ LOG.info("Created Hoodie Catalog '{}' in hms mode", catalogName);
+ }
+
+ @Override
+ public void open() throws CatalogException {
+ if (this.client == null) {
+ try {
+ this.client = Hive.get(hiveConf).getMSC();
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to create HiveMetaStoreClient", e);
+ }
+ LOG.info("Connected to Hive metastore");
+ }
+ if (!databaseExists(getDefaultDatabase())) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Configured default database %s doesn't exist in catalog %s.",
+ getDefaultDatabase(), getName()));
+ }
+ }
+
+ @Override
+ public void close() throws CatalogException {
+ if (client != null) {
+ client.close();
+ client = null;
+ LOG.info("Close connection to Hive metastore");
+ }
+ }
+
+ public HiveConf getHiveConf() {
+ return hiveConf;
+ }
+
+ // ------ databases ------
+
+ @Override
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ return client.getAllDatabases();
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to list all databases in %s", getName()), e);
+ }
+ }
+
+ public Database getHiveDatabase(String databaseName) throws DatabaseNotExistException {
+ try {
+ return client.getDatabase(databaseName);
+ } catch (NoSuchObjectException e) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to get database %s from %s", databaseName, getName()), e);
+ }
+ }
+
+ @Override
+ public CatalogDatabase getDatabase(String databaseName)
+ throws DatabaseNotExistException, CatalogException {
+ Database hiveDatabase = getHiveDatabase(databaseName);
+
+ Map<String, String> properties = new HashMap<>(hiveDatabase.getParameters());
+
+ properties.put(SqlCreateHiveDatabase.DATABASE_LOCATION_URI, hiveDatabase.getLocationUri());
+
+ return new CatalogDatabaseImpl(properties, hiveDatabase.getDescription());
+ }
+
+ @Override
+ public boolean databaseExists(String databaseName) throws CatalogException {
+ try {
+ return client.getDatabase(databaseName) != null;
+ } catch (NoSuchObjectException e) {
+ return false;
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format(
+ "Failed to determine whether database %s exists or not", databaseName),
+ e);
+ }
+ }
+
+ @Override
+ public void createDatabase(
+ String databaseName, CatalogDatabase database, boolean ignoreIfExists)
+ throws DatabaseAlreadyExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(database, "database cannot be null");
+
+ Map<String, String> properties = database.getProperties();
+
+ String dbLocationUri = properties.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+
+ Database hiveDatabase =
+ new Database(databaseName, database.getComment(), dbLocationUri, properties);
+
+ try {
+ client.createDatabase(hiveDatabase);
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new DatabaseAlreadyExistException(getName(), hiveDatabase.getName());
+ }
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to create database %s", hiveDatabase.getName()), e);
+ }
+ }
+
+ @Override
+ public void dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade)
+ throws DatabaseNotExistException, DatabaseNotEmptyException, CatalogException {
+ try {
+ client.dropDatabase(name, true, ignoreIfNotExists, cascade);
+ } catch (NoSuchObjectException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), name);
+ }
+ } catch (InvalidOperationException e) {
+ throw new DatabaseNotEmptyException(getName(), name);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to drop database %s", name), e);
+ }
+ }
+
+ @Override
+ public void alterDatabase(
+ String databaseName, CatalogDatabase newDatabase, boolean ignoreIfNotExists)
+ throws DatabaseNotExistException, CatalogException {
+ checkArgument(
+ !isNullOrWhitespaceOnly(databaseName), "databaseName cannot be null or empty");
+ checkNotNull(newDatabase, "newDatabase cannot be null");
+
+ // client.alterDatabase doesn't throw any exception if there is no existing database
+ Database hiveDB;
+ try {
+ hiveDB = getHiveDatabase(databaseName);
+ } catch (DatabaseNotExistException e) {
+ if (!ignoreIfNotExists) {
+ throw new DatabaseNotExistException(getName(), databaseName);
+ }
+
+ return;
+ }
+
+ try {
+ client.alterDatabase(databaseName, alterDatabase(hiveDB, newDatabase));
+ } catch (TException e) {
+ throw new HoodieCatalogException(
+ String.format("Failed to alter database %s", databaseName), e);
+ }
+ }
+
+ private static Database alterDatabase(Database hiveDB, CatalogDatabase newDatabase) {
+ Map<String, String> newParams = newDatabase.getProperties();
+ String opStr = newParams.remove(ALTER_DATABASE_OP);
+ if (opStr == null) {
+ // by default is to alter db properties
+ opStr = SqlAlterHiveDatabase.AlterHiveDatabaseOp.CHANGE_PROPS.name();
+ }
+ String newLocation = newParams.remove(SqlCreateHiveDatabase.DATABASE_LOCATION_URI);
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp op =
+ SqlAlterHiveDatabase.AlterHiveDatabaseOp.valueOf(opStr);
+ switch (op) {
+ case CHANGE_PROPS:
+ hiveDB.setParameters(newParams);
+ break;
+ case CHANGE_LOCATION:
+ hiveDB.setLocationUri(newLocation);
+ break;
+ case CHANGE_OWNER:
+ String ownerName = newParams.remove(DATABASE_OWNER_NAME);
+ String ownerType = newParams.remove(DATABASE_OWNER_TYPE);
+ hiveDB.setOwnerName(ownerName);
+ switch (ownerType) {
+ case SqlAlterHiveDatabaseOwner.ROLE_OWNER:
+ hiveDB.setOwnerType(PrincipalType.ROLE);
+ break;
+ case SqlAlterHiveDatabaseOwner.USER_OWNER:
+ hiveDB.setOwnerType(PrincipalType.USER);
+ break;
+ default:
+ throw new CatalogException("Unsupported database owner type: " + ownerType);
+ }
+ break;
+ default:
+ throw new CatalogException("Unsupported alter database op:" + opStr);
+ }
+ // is_generic is deprecated, remove it
+ if (hiveDB.getParameters() != null) {
+ hiveDB.getParameters().remove(CatalogPropertiesUtil.IS_GENERIC);
+ }
+ return hiveDB;
+ }
+
+ // ------ tables ------
+
+ private Table checkHoodieTable(Table hiveTable) {
+ if (!hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")
+ && !hiveTable.getParameters().getOrDefault(SPARK_SOURCE_PROVIDER, "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("the %s is not hoodie table", hiveTable.getTableName()));
+ }
+ return hiveTable;
+ }
+
+ private boolean isFlinkTable(Table hiveTable) {
+ return hiveTable.getParameters().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi");
+ }
+
+ private org.apache.avro.Schema getLatestTableSchema(String path) {
+ if (StringUtils.isNullOrEmpty(path) || !StreamerUtil.tableExists(path, hiveConf)) {
+ return null;
+ }
+
+ try {
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(path, hiveConf);
+ return new TableSchemaResolver(metaClient).getTableAvroSchema(false);
+ } catch (Exception e) {
+ LOG.warn("Error while resolving the latest table schema", e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ public Table getHiveTable(ObjectPath tablePath) throws TableNotExistException {
+ try {
+ Table hiveTable = client.getTable(tablePath.getDatabaseName(), tablePath.getObjectName());
+ return checkHoodieTable(hiveTable);
+ } catch (NoSuchObjectException e) {
+ throw new TableNotExistException(getName(), tablePath);
+ } catch (TException e) {
+ throw new HoodieCatalogException(String.format("Failed to get table %s from Hive metastore", tablePath.getObjectName()));
+ }
+ }
+
+ private Table translateSparkTable2Flink(ObjectPath tablePath, Table hiveTable) {
+ if (!isFlinkTable(hiveTable)) {
+ try {
+ Map<String, String> parameters = hiveTable.getParameters();
+ parameters.putAll(TableOptionProperties.translateSparkTableProperties2Flink(hiveTable));
+ String path = hiveTable.getSd().getLocation();
+ parameters.put(PATH.key(), path);
+ if (!parameters.containsKey(FlinkOptions.HIVE_STYLE_PARTITIONING.key())) {
+ Path hoodieTablePath = new Path(path);
+ boolean hiveStyle = Arrays.stream(FSUtils.getFs(hoodieTablePath, hiveConf).listStatus(hoodieTablePath))
+ .map(fileStatus -> fileStatus.getPath().getName())
+ .filter(f -> !f.equals(".hoodie") && !f.equals("default"))
+ .anyMatch(FilePathUtils::isHiveStylePartitioning);
+ parameters.put(FlinkOptions.HIVE_STYLE_PARTITIONING.key(), String.valueOf(hiveStyle));
+ }
+ client.alter_table(tablePath.getDatabaseName(), tablePath.getObjectName(), hiveTable);
+ } catch (Exception e) {
+ throw new HoodieCatalogException("Failed to update table schema", e);
+ }
+ }
+ return hiveTable;
+ }
+
+ @Override
+ public CatalogBaseTable getTable(ObjectPath tablePath) throws TableNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ Table hiveTable = getHiveTable(tablePath);
+ hiveTable = translateSparkTable2Flink(tablePath, hiveTable);
+ String path = hiveTable.getSd().getLocation();
+ Map<String, String> parameters = hiveTable.getParameters();
+ Schema latestTableSchema = getLatestTableSchema(path);
+ org.apache.flink.table.api.Schema schema;
+ if (latestTableSchema != null) {
+ org.apache.flink.table.api.Schema.Builder builder = org.apache.flink.table.api.Schema.newBuilder()
+ .fromRowDataType(AvroSchemaConverter.convertToDataType(latestTableSchema));
+ String pkConstraintName = parameters.get(PK_CONSTRAINT_NAME);
+ if (!StringUtils.isNullOrEmpty(pkConstraintName)) {
+ builder.primaryKeyNamed(pkConstraintName, StringUtils.split(parameters.get(PK_COLUMNS), ","));
+ }
+ schema = builder.build();
+ } else {
+ LOG.warn("{} does not have any hoodie schema, and use hive table schema to infer the table schema", tablePath);
+ schema = TableOptionProperties.convertTableSchema(hiveTable);
+ }
+ return CatalogTable.of(schema, parameters.get(COMMENT),
+ TableOptionProperties.getFieldNames(hiveTable.getPartitionKeys()), parameters);
+ }
+
+ @Override
+ public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
+ throws TableAlreadyExistException, DatabaseNotExistException, CatalogException {
+ checkNotNull(tablePath, "tablePath cannot be null");
+ checkNotNull(table, "table cannot be null");
+
+ if (!databaseExists(tablePath.getDatabaseName())) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName());
+ }
+
+ if (!table.getOptions().getOrDefault(CONNECTOR.key(), "").equalsIgnoreCase("hudi")) {
+ throw new HoodieCatalogException(String.format("The %s is not hoodie table", tablePath.getObjectName()));
+ }
+
+ if (table instanceof CatalogView) {
+ throw new HoodieCatalogException("Hoodie catalog does not support to CREATE VIEW.");
+ }
+
+ try {
+ boolean isMorTable = table.getOptions().getOrDefault(FlinkOptions.TABLE_TYPE.key(),
+ FlinkOptions.TABLE_TYPE.defaultValue()).equalsIgnoreCase(FlinkOptions.TABLE_TYPE_MERGE_ON_READ);
+ Table hiveTable = instantiateHiveTable(tablePath, table, inferTablePath(tablePath, table), isMorTable);
+ //create hive table
+ client.createTable(hiveTable);
+ //init hoodie metaClient
+ initTableIfNotExists(tablePath, (CatalogTable)table);
Review Comment:
Let's make this operation optional, some flink job on k8s may have no right to write dfs. @XuQianJin-Stars
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1178927681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5051b524f5e241103afbdd9394d0a14a660c3bd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779)
* 8032fcaf15f8b246c8652ed5df49fa01564e2649 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1177290322
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179233985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 185ec5470953fc3420fb85c8d72c40bbf317f995 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791)
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* 0c113baae795ddf7f884b868dcde0ddc3f8adb92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6013: [HUDI-4089] Support HMS for flink HoodieCatalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6013:
URL: https://github.com/apache/hudi/pull/6013#issuecomment-1179760086
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"triggerType" : "PUSH"
}, {
"hash" : "fbd973eb579a8a12b4e1cecde42c55aa7f2ba1e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9759",
"triggerID" : "1177425298",
"triggerType" : "MANUAL"
}, {
"hash" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9779",
"triggerID" : "c5051b524f5e241103afbdd9394d0a14a660c3bd",
"triggerType" : "PUSH"
}, {
"hash" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9790",
"triggerID" : "8032fcaf15f8b246c8652ed5df49fa01564e2649",
"triggerType" : "PUSH"
}, {
"hash" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9791",
"triggerID" : "185ec5470953fc3420fb85c8d72c40bbf317f995",
"triggerType" : "PUSH"
}, {
"hash" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cb269cf6fadc020ed8e512d673d3435a69e4740",
"triggerType" : "PUSH"
}, {
"hash" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9792",
"triggerID" : "0c113baae795ddf7f884b868dcde0ddc3f8adb92",
"triggerType" : "PUSH"
}, {
"hash" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9808",
"triggerID" : "02ff7556a3d1324bdf66aea4fd28988c8233e987",
"triggerType" : "PUSH"
}, {
"hash" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9819",
"triggerID" : "f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cb269cf6fadc020ed8e512d673d3435a69e4740 UNKNOWN
* f0b1f5b7f00ab7b294e907a6e1bc0c7b1ca3ef00 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9819)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org