You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/12/07 15:42:24 UTC

[GitHub] [inlong] ZuoFengZhang opened a new pull request, #6790: [INLONG-6785][Manager] Support register and manage hudi sort

ZuoFengZhang opened a new pull request, #6790:
URL: https://github.com/apache/inlong/pull/6790

   ### Prepare a Pull Request
   *[Manager] Support register and manage hudi sort*
   
   
   
   - Fixes #6785 
   
   ### Motivation
   
   *Explain here the context, and why you're making that change. What is the problem you're trying to solve?*
   
   ### Modifications
   
   *Describe the modifications you've done.*
   
   ### Verifying this change
   
   *(Please pick either of the following options)*
   
   - [x] This change is a trivial rework/code cleanup without any test coverage.
   
   - [x] This change is already covered by existing tests, such as:
     *(please describe tests)*
   
   - [x] This change added tests and can be verified as follows:
   
     *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (10MB)*
     - *Extended integration test for recovery after broker failure*
   
   ### Documentation
   
     - Does this pull request introduce a new feature? (yes / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
     - If a feature is not applicable for documentation, explain why?
     - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] dockerzhang commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
dockerzhang commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1042960165


##########
inlong-manager/manager-plugins/src/main/java/org/apache/inlong/manager/plugin/listener/RestartStreamListener.java:
##########
@@ -38,12 +43,6 @@
 import org.apache.inlong.manager.workflow.event.ListenerResult;
 import org.apache.inlong.manager.workflow.event.task.SortOperateListener;
 
-import java.util.HashMap;

Review Comment:
   > Please refer this: #3082, and reformat those packages' order, thanks.
   
   spotless support defines the order of the import packages.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] healchow commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1044003136


##########
inlong-manager/manager-plugins/src/main/java/org/apache/inlong/manager/plugin/listener/RestartStreamListener.java:
##########
@@ -89,9 +89,11 @@ public ListenerResult listen(WorkflowContext context) throws Exception {
 
         Map<String, String> kvConf = new HashMap<>();
         groupExtList.forEach(groupExtInfo -> kvConf.put(groupExtInfo.getKeyName(), groupExtInfo.getKeyValue()));
-        streamExtList.forEach(extInfo -> {
-            kvConf.put(extInfo.getKeyName(), extInfo.getKeyValue());
-        });
+        if (streamExtList != null) {

Review Comment:
   // import org.apache.commons.collections.CollectionUtils;
   ```suggestion
           if (!CollectionUtils.isEmpty(streamExtList)) {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] leosanqing commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
leosanqing commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1042918024


##########
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sink/hudi/HudiSink.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.pojo.sink.hudi;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import java.util.HashMap;
+import java.util.List;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.EqualsAndHashCode;
+import lombok.ToString;
+import lombok.experimental.SuperBuilder;
+import org.apache.inlong.manager.common.consts.SinkType;
+import org.apache.inlong.manager.common.util.CommonBeanUtils;
+import org.apache.inlong.manager.common.util.JsonTypeDefine;
+import org.apache.inlong.manager.pojo.sink.SinkRequest;
+import org.apache.inlong.manager.pojo.sink.StreamSink;
+
+/**
+ * Iceberg sink info

Review Comment:
   Hudi



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] healchow merged pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
healchow merged PR #6790:
URL: https://github.com/apache/inlong/pull/6790


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] ZuoFengZhang commented on pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
ZuoFengZhang commented on PR #6790:
URL: https://github.com/apache/inlong/pull/6790#issuecomment-1344199443

   > I don't seem to see unit tests. There are many modifications. It is recommended to add unit tests to key logic.
   
   I have added the UT about manager-server


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] healchow commented on pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
healchow commented on PR #6790:
URL: https://github.com/apache/inlong/pull/6790#issuecomment-1343767857

   I don't seem to see unit tests. There are many modifications. It is recommended to add unit tests to key logic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] healchow commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1044001261


##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/sink/hudi/HudiCatalogClient.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.sink.hudi;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Schema.UnresolvedColumn;
+import org.apache.flink.table.api.Schema.UnresolvedPhysicalColumn;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.org.apache.hbase.thirdparty.com.google.common.collect.Maps;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.hudi.table.catalog.HiveSchemaUtils;
+import org.apache.inlong.manager.pojo.sink.hudi.HudiColumnInfo;
+import org.apache.inlong.manager.pojo.sink.hudi.HudiTableInfo;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class HudiCatalogClient {

Review Comment:
   Please add Java docs for classes, public methods, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] gong commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1044062420


##########
inlong-manager/manager-service/pom.xml:
##########
@@ -557,5 +557,13 @@
             <artifactId>flink-core</artifactId>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.apache.hudi</groupId>
+            <artifactId>hudi-flink1.13-bundle</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-table-common</artifactId>
+        </dependency>

Review Comment:
   Why add it in here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] ZuoFengZhang commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
ZuoFengZhang commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1044353693


##########
inlong-manager/manager-service/pom.xml:
##########
@@ -557,5 +557,13 @@
             <artifactId>flink-core</artifactId>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.apache.hudi</groupId>
+            <artifactId>hudi-flink1.13-bundle</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-table-common</artifactId>
+        </dependency>

Review Comment:
   flink-table-common is used to parse the schema of the hive table, has been rewritten, and removed the reference



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] ZuoFengZhang commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
ZuoFengZhang commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1046599178


##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/sink/hudi/HudiCatalogClient.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.sink.hudi;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.org.apache.hbase.thirdparty.com.google.common.collect.Maps;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.inlong.manager.pojo.sink.hudi.HudiColumnInfo;
+import org.apache.inlong.manager.pojo.sink.hudi.HudiTableInfo;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The Catalog client for Hudi.
+ */
+public class HudiCatalogClient {
+
+    private static final Logger LOG = LoggerFactory.getLogger(HudiCatalogClient.class);
+
+    private final String uri;
+    private final String dbName;
+    private final String warehouse;
+    private IMetaStoreClient client;
+    private final HiveConf hiveConf;
+
+    public HudiCatalogClient(String uri, String warehouse, String dbName) throws MetaException {
+        this.uri = uri;
+        this.warehouse = warehouse;
+        this.dbName = dbName;
+        hiveConf = new HiveConf();
+        hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, uri);
+        hiveConf.setBoolVar(HiveConf.ConfVars.METASTORE_EXECUTE_SET_UGI, false);
+    }
+
+    /**
+     * Open the hive metastore connection
+     */
+    public void open() {
+        if (this.client == null) {
+            try {
+                this.client = Hive.get(hiveConf).getMSC();
+            } catch (Exception e) {
+                throw new HoodieCatalogException("Failed to create hive metastore client", e);
+            }
+            LOG.info("Connected to Hive metastore");
+        }
+    }
+
+    private void createDatabase(String warehouse, Map<String, String> meta, boolean ignoreIfExists) {
+        Database database = new Database();
+        Map<String, String> parameter = Maps.newHashMap();
+        database.setName(dbName);
+        database.setLocationUri((new Path(warehouse, dbName) + ".db"));
+        meta.forEach((key, value) -> {
+            if (key.equals("comment")) {
+                database.setDescription(value);
+            } else if (key.equals("location")) {
+                database.setLocationUri(value);
+            } else if (value != null) {
+                parameter.put(key, value);
+            }
+            database.setParameters(parameter);
+        });
+        try {
+            client.createDatabase(database);
+        } catch (AlreadyExistsException e) {
+            if (!ignoreIfExists) {
+                throw new RuntimeException("Database '" + dbName + "' already exist!");
+            }
+        } catch (TException e) {
+            throw new RuntimeException("Failed to create database '" + dbName
+                    + "'", e);
+        }
+    }
+
+    /**
+     * Create the hudi database
+     * @param warehouse the warehouse directory in dfs
+     * @param ignoreIfExists not create again if exist 
+     */
+    public void createDatabase(String warehouse, boolean ignoreIfExists) {
+        createDatabase(warehouse, Maps.newHashMap(), ignoreIfExists);
+    }
+
+    /**
+     * Check table if exist
+     * @param tableName the table name of hudi table
+     * @return return true if exist
+     */
+    public boolean tableExist(String tableName) throws TException {
+        return client.tableExists(dbName, tableName);
+    }
+
+    /**
+     * get column infos of exist hudi table
+     * @param dbName the database name
+     * @param tableName the table name
+     */
+    public List<HudiColumnInfo> getColumns(
+            String dbName,
+            String tableName)
+            throws TException {
+        Table hiveTable = client.getTable(dbName, tableName);
+        List<FieldSchema> allCols = hiveTable.getSd().getCols().stream()
+                // filter out the metadata columns
+                .filter(s -> !HoodieAvroUtils.isMetadataField(s.getName()))
+                .collect(Collectors.toList());
+        allCols.addAll(hiveTable.getPartitionKeys());
+
+        return allCols.stream()
+                .map((FieldSchema s) -> {
+                    HudiColumnInfo info = new HudiColumnInfo();
+                    info.setName(s.getName());
+                    info.setType(s.getType());
+                    return info;
+                })
+                .collect(Collectors.toList());
+    }
+
+    /**
+     * Add column to hudi table at the tail
+     */
+    public void addColumns(String tableName, List<HudiColumnInfo> columns) throws TException {
+        Table hiveTable = client.getTable(dbName, tableName);
+        Table newHiveTable = hiveTable.deepCopy();
+        List<FieldSchema> cols = newHiveTable.getSd().getCols();
+        for (HudiColumnInfo column : columns) {
+            FieldSchema fieldSchema = new FieldSchema();
+            fieldSchema.setName(column.getName());
+            fieldSchema.setType(column.getType());
+            fieldSchema.setComment(column.getDesc());
+            cols.add(fieldSchema);
+        }
+        newHiveTable.getSd().setCols(cols);
+        client.alter_table(dbName, tableName, newHiveTable);
+    }
+
+    /**
+     * Create hudi table and register to hive metastore
+     * @param tableName the hudi table name
+     * @param tableInfo the hudi table info
+     * @param useRealTimeInputFormat ignore uber input Format 
+     */
+    public void createTable(
+            String tableName,
+            HudiTableInfo tableInfo,
+            boolean useRealTimeInputFormat)
+            throws TException, IOException {
+        Table hiveTable = org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(dbName, tableName);
+        hiveTable.setOwner(UserGroupInformation.getCurrentUser().getUserName());
+        hiveTable.setCreateTime((int) (System.currentTimeMillis() / 1000));
+
+        Map<String, String> properties = new HashMap<>();
+        String location = this.warehouse + "/" + dbName + ".db" + "/" + tableName;
+        properties.put("path", location);
+
+        List<FieldSchema> cols = new ArrayList<>();
+        for (HudiColumnInfo column : tableInfo.getColumns()) {
+            FieldSchema fieldSchema = new FieldSchema();
+            fieldSchema.setName(column.getName());
+            fieldSchema.setType(column.getType());
+            fieldSchema.setComment(column.getDesc());
+            cols.add(fieldSchema);
+        }
+
+        // Build storage of hudi table
+        StorageDescriptor sd = new StorageDescriptor();
+        sd.setCols(cols);
+        hiveTable.setDbName(dbName);
+        hiveTable.setTableName(tableName);
+        // FIXME: splitSchemas need config by frontend
+
+        HoodieFileFormat baseFileFormat = HoodieFileFormat.PARQUET;
+        // ignore uber input Format
+        String inputFormatClassName =
+                HoodieInputFormatUtils.getInputFormatClassName(baseFileFormat, useRealTimeInputFormat);
+        String outputFormatClassName = HoodieInputFormatUtils.getOutputFormatClassName(baseFileFormat);
+        String serDeClassName = HoodieInputFormatUtils.getSerDeClassName(baseFileFormat);
+        sd.setInputFormat(inputFormatClassName);
+        sd.setOutputFormat(outputFormatClassName);
+        Map<String, String> serdeProperties = new HashMap<>();
+        serdeProperties.put("path", location);
+        serdeProperties.put(ConfigUtils.IS_QUERY_AS_RO_TABLE, String.valueOf(!useRealTimeInputFormat));
+        serdeProperties.put("serialization.format", "1");

Review Comment:
   `serialization.format` is the field delimiter character in file between two column fields of the table when the file is serialized. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] healchow commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1044000077


##########
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sink/hudi/HudiPartition.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.pojo.sink.hudi;
+
+import org.apache.inlong.manager.common.util.Preconditions;
+
+/**
+ * Iceberg partition type

Review Comment:
   Hudi



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] healchow commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1045251012


##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/sink/hudi/HudiCatalogClient.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.sink.hudi;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.org.apache.hbase.thirdparty.com.google.common.collect.Maps;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.inlong.manager.pojo.sink.hudi.HudiColumnInfo;
+import org.apache.inlong.manager.pojo.sink.hudi.HudiTableInfo;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The Catalog client for Hudi.
+ */
+public class HudiCatalogClient {
+
+    private static final Logger LOG = LoggerFactory.getLogger(HudiCatalogClient.class);
+
+    private final String uri;
+    private final String dbName;
+    private final String warehouse;
+    private IMetaStoreClient client;
+    private final HiveConf hiveConf;
+
+    public HudiCatalogClient(String uri, String warehouse, String dbName) throws MetaException {
+        this.uri = uri;
+        this.warehouse = warehouse;
+        this.dbName = dbName;
+        hiveConf = new HiveConf();
+        hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, uri);
+        hiveConf.setBoolVar(HiveConf.ConfVars.METASTORE_EXECUTE_SET_UGI, false);
+    }
+
+    /**
+     * Open the hive metastore connection
+     */
+    public void open() {
+        if (this.client == null) {
+            try {
+                this.client = Hive.get(hiveConf).getMSC();
+            } catch (Exception e) {
+                throw new HoodieCatalogException("Failed to create hive metastore client", e);
+            }
+            LOG.info("Connected to Hive metastore");
+        }
+    }
+
+    private void createDatabase(String warehouse, Map<String, String> meta, boolean ignoreIfExists) {
+        Database database = new Database();
+        Map<String, String> parameter = Maps.newHashMap();
+        database.setName(dbName);
+        database.setLocationUri((new Path(warehouse, dbName) + ".db"));
+        meta.forEach((key, value) -> {
+            if (key.equals("comment")) {
+                database.setDescription(value);
+            } else if (key.equals("location")) {
+                database.setLocationUri(value);
+            } else if (value != null) {
+                parameter.put(key, value);
+            }
+            database.setParameters(parameter);
+        });
+        try {
+            client.createDatabase(database);
+        } catch (AlreadyExistsException e) {
+            if (!ignoreIfExists) {
+                throw new RuntimeException("Database '" + dbName + "' already exist!");
+            }
+        } catch (TException e) {
+            throw new RuntimeException("Failed to create database '" + dbName
+                    + "'", e);
+        }
+    }
+
+    /**
+     * Create the hudi database
+     * @param warehouse the warehouse directory in dfs
+     * @param ignoreIfExists not create again if exist 
+     */
+    public void createDatabase(String warehouse, boolean ignoreIfExists) {
+        createDatabase(warehouse, Maps.newHashMap(), ignoreIfExists);
+    }
+
+    /**
+     * Check table if exist
+     * @param tableName the table name of hudi table
+     * @return return true if exist
+     */
+    public boolean tableExist(String tableName) throws TException {
+        return client.tableExists(dbName, tableName);
+    }
+
+    /**
+     * get column infos of exist hudi table
+     * @param dbName the database name
+     * @param tableName the table name
+     */
+    public List<HudiColumnInfo> getColumns(
+            String dbName,
+            String tableName)
+            throws TException {
+        Table hiveTable = client.getTable(dbName, tableName);
+        List<FieldSchema> allCols = hiveTable.getSd().getCols().stream()
+                // filter out the metadata columns
+                .filter(s -> !HoodieAvroUtils.isMetadataField(s.getName()))
+                .collect(Collectors.toList());
+        allCols.addAll(hiveTable.getPartitionKeys());
+
+        return allCols.stream()
+                .map((FieldSchema s) -> {
+                    HudiColumnInfo info = new HudiColumnInfo();
+                    info.setName(s.getName());
+                    info.setType(s.getType());
+                    return info;
+                })
+                .collect(Collectors.toList());
+    }
+
+    /**
+     * Add column to hudi table at the tail
+     */
+    public void addColumns(String tableName, List<HudiColumnInfo> columns) throws TException {
+        Table hiveTable = client.getTable(dbName, tableName);
+        Table newHiveTable = hiveTable.deepCopy();
+        List<FieldSchema> cols = newHiveTable.getSd().getCols();
+        for (HudiColumnInfo column : columns) {
+            FieldSchema fieldSchema = new FieldSchema();
+            fieldSchema.setName(column.getName());
+            fieldSchema.setType(column.getType());
+            fieldSchema.setComment(column.getDesc());
+            cols.add(fieldSchema);
+        }
+        newHiveTable.getSd().setCols(cols);
+        client.alter_table(dbName, tableName, newHiveTable);
+    }
+
+    /**
+     * Create hudi table and register to hive metastore
+     * @param tableName the hudi table name
+     * @param tableInfo the hudi table info
+     * @param useRealTimeInputFormat ignore uber input Format 
+     */
+    public void createTable(
+            String tableName,
+            HudiTableInfo tableInfo,
+            boolean useRealTimeInputFormat)
+            throws TException, IOException {
+        Table hiveTable = org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(dbName, tableName);
+        hiveTable.setOwner(UserGroupInformation.getCurrentUser().getUserName());
+        hiveTable.setCreateTime((int) (System.currentTimeMillis() / 1000));
+
+        Map<String, String> properties = new HashMap<>();
+        String location = this.warehouse + "/" + dbName + ".db" + "/" + tableName;
+        properties.put("path", location);
+
+        List<FieldSchema> cols = new ArrayList<>();
+        for (HudiColumnInfo column : tableInfo.getColumns()) {
+            FieldSchema fieldSchema = new FieldSchema();
+            fieldSchema.setName(column.getName());
+            fieldSchema.setType(column.getType());
+            fieldSchema.setComment(column.getDesc());
+            cols.add(fieldSchema);
+        }
+
+        // Build storage of hudi table
+        StorageDescriptor sd = new StorageDescriptor();
+        sd.setCols(cols);
+        hiveTable.setDbName(dbName);
+        hiveTable.setTableName(tableName);
+        // FIXME: splitSchemas need config by frontend
+
+        HoodieFileFormat baseFileFormat = HoodieFileFormat.PARQUET;
+        // ignore uber input Format
+        String inputFormatClassName =
+                HoodieInputFormatUtils.getInputFormatClassName(baseFileFormat, useRealTimeInputFormat);
+        String outputFormatClassName = HoodieInputFormatUtils.getOutputFormatClassName(baseFileFormat);
+        String serDeClassName = HoodieInputFormatUtils.getSerDeClassName(baseFileFormat);
+        sd.setInputFormat(inputFormatClassName);
+        sd.setOutputFormat(outputFormatClassName);
+        Map<String, String> serdeProperties = new HashMap<>();
+        serdeProperties.put("path", location);
+        serdeProperties.put(ConfigUtils.IS_QUERY_AS_RO_TABLE, String.valueOf(!useRealTimeInputFormat));
+        serdeProperties.put("serialization.format", "1");

Review Comment:
   Why set the `format` to `1`? Does it have any value else?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] ZuoFengZhang commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
ZuoFengZhang commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1046599178


##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/sink/hudi/HudiCatalogClient.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.sink.hudi;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.org.apache.hbase.thirdparty.com.google.common.collect.Maps;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.inlong.manager.pojo.sink.hudi.HudiColumnInfo;
+import org.apache.inlong.manager.pojo.sink.hudi.HudiTableInfo;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The Catalog client for Hudi.
+ */
+public class HudiCatalogClient {
+
+    private static final Logger LOG = LoggerFactory.getLogger(HudiCatalogClient.class);
+
+    private final String uri;
+    private final String dbName;
+    private final String warehouse;
+    private IMetaStoreClient client;
+    private final HiveConf hiveConf;
+
+    public HudiCatalogClient(String uri, String warehouse, String dbName) throws MetaException {
+        this.uri = uri;
+        this.warehouse = warehouse;
+        this.dbName = dbName;
+        hiveConf = new HiveConf();
+        hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, uri);
+        hiveConf.setBoolVar(HiveConf.ConfVars.METASTORE_EXECUTE_SET_UGI, false);
+    }
+
+    /**
+     * Open the hive metastore connection
+     */
+    public void open() {
+        if (this.client == null) {
+            try {
+                this.client = Hive.get(hiveConf).getMSC();
+            } catch (Exception e) {
+                throw new HoodieCatalogException("Failed to create hive metastore client", e);
+            }
+            LOG.info("Connected to Hive metastore");
+        }
+    }
+
+    private void createDatabase(String warehouse, Map<String, String> meta, boolean ignoreIfExists) {
+        Database database = new Database();
+        Map<String, String> parameter = Maps.newHashMap();
+        database.setName(dbName);
+        database.setLocationUri((new Path(warehouse, dbName) + ".db"));
+        meta.forEach((key, value) -> {
+            if (key.equals("comment")) {
+                database.setDescription(value);
+            } else if (key.equals("location")) {
+                database.setLocationUri(value);
+            } else if (value != null) {
+                parameter.put(key, value);
+            }
+            database.setParameters(parameter);
+        });
+        try {
+            client.createDatabase(database);
+        } catch (AlreadyExistsException e) {
+            if (!ignoreIfExists) {
+                throw new RuntimeException("Database '" + dbName + "' already exist!");
+            }
+        } catch (TException e) {
+            throw new RuntimeException("Failed to create database '" + dbName
+                    + "'", e);
+        }
+    }
+
+    /**
+     * Create the hudi database
+     * @param warehouse the warehouse directory in dfs
+     * @param ignoreIfExists not create again if exist 
+     */
+    public void createDatabase(String warehouse, boolean ignoreIfExists) {
+        createDatabase(warehouse, Maps.newHashMap(), ignoreIfExists);
+    }
+
+    /**
+     * Check table if exist
+     * @param tableName the table name of hudi table
+     * @return return true if exist
+     */
+    public boolean tableExist(String tableName) throws TException {
+        return client.tableExists(dbName, tableName);
+    }
+
+    /**
+     * get column infos of exist hudi table
+     * @param dbName the database name
+     * @param tableName the table name
+     */
+    public List<HudiColumnInfo> getColumns(
+            String dbName,
+            String tableName)
+            throws TException {
+        Table hiveTable = client.getTable(dbName, tableName);
+        List<FieldSchema> allCols = hiveTable.getSd().getCols().stream()
+                // filter out the metadata columns
+                .filter(s -> !HoodieAvroUtils.isMetadataField(s.getName()))
+                .collect(Collectors.toList());
+        allCols.addAll(hiveTable.getPartitionKeys());
+
+        return allCols.stream()
+                .map((FieldSchema s) -> {
+                    HudiColumnInfo info = new HudiColumnInfo();
+                    info.setName(s.getName());
+                    info.setType(s.getType());
+                    return info;
+                })
+                .collect(Collectors.toList());
+    }
+
+    /**
+     * Add column to hudi table at the tail
+     */
+    public void addColumns(String tableName, List<HudiColumnInfo> columns) throws TException {
+        Table hiveTable = client.getTable(dbName, tableName);
+        Table newHiveTable = hiveTable.deepCopy();
+        List<FieldSchema> cols = newHiveTable.getSd().getCols();
+        for (HudiColumnInfo column : columns) {
+            FieldSchema fieldSchema = new FieldSchema();
+            fieldSchema.setName(column.getName());
+            fieldSchema.setType(column.getType());
+            fieldSchema.setComment(column.getDesc());
+            cols.add(fieldSchema);
+        }
+        newHiveTable.getSd().setCols(cols);
+        client.alter_table(dbName, tableName, newHiveTable);
+    }
+
+    /**
+     * Create hudi table and register to hive metastore
+     * @param tableName the hudi table name
+     * @param tableInfo the hudi table info
+     * @param useRealTimeInputFormat ignore uber input Format 
+     */
+    public void createTable(
+            String tableName,
+            HudiTableInfo tableInfo,
+            boolean useRealTimeInputFormat)
+            throws TException, IOException {
+        Table hiveTable = org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(dbName, tableName);
+        hiveTable.setOwner(UserGroupInformation.getCurrentUser().getUserName());
+        hiveTable.setCreateTime((int) (System.currentTimeMillis() / 1000));
+
+        Map<String, String> properties = new HashMap<>();
+        String location = this.warehouse + "/" + dbName + ".db" + "/" + tableName;
+        properties.put("path", location);
+
+        List<FieldSchema> cols = new ArrayList<>();
+        for (HudiColumnInfo column : tableInfo.getColumns()) {
+            FieldSchema fieldSchema = new FieldSchema();
+            fieldSchema.setName(column.getName());
+            fieldSchema.setType(column.getType());
+            fieldSchema.setComment(column.getDesc());
+            cols.add(fieldSchema);
+        }
+
+        // Build storage of hudi table
+        StorageDescriptor sd = new StorageDescriptor();
+        sd.setCols(cols);
+        hiveTable.setDbName(dbName);
+        hiveTable.setTableName(tableName);
+        // FIXME: splitSchemas need config by frontend
+
+        HoodieFileFormat baseFileFormat = HoodieFileFormat.PARQUET;
+        // ignore uber input Format
+        String inputFormatClassName =
+                HoodieInputFormatUtils.getInputFormatClassName(baseFileFormat, useRealTimeInputFormat);
+        String outputFormatClassName = HoodieInputFormatUtils.getOutputFormatClassName(baseFileFormat);
+        String serDeClassName = HoodieInputFormatUtils.getSerDeClassName(baseFileFormat);
+        sd.setInputFormat(inputFormatClassName);
+        sd.setOutputFormat(outputFormatClassName);
+        Map<String, String> serdeProperties = new HashMap<>();
+        serdeProperties.put("path", location);
+        serdeProperties.put(ConfigUtils.IS_QUERY_AS_RO_TABLE, String.valueOf(!useRealTimeInputFormat));
+        serdeProperties.put("serialization.format", "1");

Review Comment:
   `serialization.format` is the field delimiter character in file between two column fields of the table when the file is serialized. This option does not take effect in the hudi table, so can cancel.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] ZuoFengZhang commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
ZuoFengZhang commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1042946640


##########
inlong-manager/manager-plugins/src/main/java/org/apache/inlong/manager/plugin/listener/RestartStreamListener.java:
##########
@@ -38,12 +43,6 @@
 import org.apache.inlong.manager.workflow.event.ListenerResult;
 import org.apache.inlong.manager.workflow.event.task.SortOperateListener;
 
-import java.util.HashMap;

Review Comment:
   Ok, I have a question why the spotless script doesn't work



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] healchow commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage hudi sort

Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1042845015


##########
inlong-manager/manager-plugins/src/main/java/org/apache/inlong/manager/plugin/listener/RestartStreamListener.java:
##########
@@ -38,12 +43,6 @@
 import org.apache.inlong.manager.workflow.event.ListenerResult;
 import org.apache.inlong.manager.workflow.event.task.SortOperateListener;
 
-import java.util.HashMap;

Review Comment:
   Please refer this: https://github.com/apache/inlong/discussions/3082, and reformat those packages' order, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [inlong] healchow commented on a diff in pull request #6790: [INLONG-6785][Manager] Support register and manage the resource of Apache Hudi

Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #6790:
URL: https://github.com/apache/inlong/pull/6790#discussion_r1045251078


##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/resource/sink/hudi/HudiCatalogClient.java:
##########
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.service.resource.sink.hudi;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.exception.HoodieCatalogException;
+import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils;
+import org.apache.hudi.org.apache.hbase.thirdparty.com.google.common.collect.Maps;
+import org.apache.hudi.sync.common.util.ConfigUtils;
+import org.apache.inlong.manager.pojo.sink.hudi.HudiColumnInfo;
+import org.apache.inlong.manager.pojo.sink.hudi.HudiTableInfo;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The Catalog client for Hudi.
+ */
+public class HudiCatalogClient {
+
+    private static final Logger LOG = LoggerFactory.getLogger(HudiCatalogClient.class);
+
+    private final String uri;
+    private final String dbName;
+    private final String warehouse;
+    private IMetaStoreClient client;
+    private final HiveConf hiveConf;
+
+    public HudiCatalogClient(String uri, String warehouse, String dbName) throws MetaException {
+        this.uri = uri;
+        this.warehouse = warehouse;
+        this.dbName = dbName;
+        hiveConf = new HiveConf();
+        hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, uri);
+        hiveConf.setBoolVar(HiveConf.ConfVars.METASTORE_EXECUTE_SET_UGI, false);
+    }
+
+    /**
+     * Open the hive metastore connection
+     */
+    public void open() {
+        if (this.client == null) {
+            try {
+                this.client = Hive.get(hiveConf).getMSC();
+            } catch (Exception e) {
+                throw new HoodieCatalogException("Failed to create hive metastore client", e);
+            }
+            LOG.info("Connected to Hive metastore");
+        }
+    }
+
+    private void createDatabase(String warehouse, Map<String, String> meta, boolean ignoreIfExists) {
+        Database database = new Database();
+        Map<String, String> parameter = Maps.newHashMap();
+        database.setName(dbName);
+        database.setLocationUri((new Path(warehouse, dbName) + ".db"));
+        meta.forEach((key, value) -> {
+            if (key.equals("comment")) {
+                database.setDescription(value);
+            } else if (key.equals("location")) {
+                database.setLocationUri(value);
+            } else if (value != null) {
+                parameter.put(key, value);
+            }
+            database.setParameters(parameter);
+        });
+        try {
+            client.createDatabase(database);
+        } catch (AlreadyExistsException e) {
+            if (!ignoreIfExists) {
+                throw new RuntimeException("Database '" + dbName + "' already exist!");
+            }
+        } catch (TException e) {
+            throw new RuntimeException("Failed to create database '" + dbName
+                    + "'", e);
+        }
+    }
+
+    /**
+     * Create the hudi database
+     * @param warehouse the warehouse directory in dfs
+     * @param ignoreIfExists not create again if exist 
+     */
+    public void createDatabase(String warehouse, boolean ignoreIfExists) {
+        createDatabase(warehouse, Maps.newHashMap(), ignoreIfExists);
+    }
+
+    /**
+     * Check table if exist
+     * @param tableName the table name of hudi table
+     * @return return true if exist
+     */
+    public boolean tableExist(String tableName) throws TException {
+        return client.tableExists(dbName, tableName);
+    }
+
+    /**
+     * get column infos of exist hudi table
+     * @param dbName the database name
+     * @param tableName the table name
+     */
+    public List<HudiColumnInfo> getColumns(
+            String dbName,
+            String tableName)
+            throws TException {
+        Table hiveTable = client.getTable(dbName, tableName);
+        List<FieldSchema> allCols = hiveTable.getSd().getCols().stream()
+                // filter out the metadata columns
+                .filter(s -> !HoodieAvroUtils.isMetadataField(s.getName()))
+                .collect(Collectors.toList());
+        allCols.addAll(hiveTable.getPartitionKeys());
+
+        return allCols.stream()
+                .map((FieldSchema s) -> {
+                    HudiColumnInfo info = new HudiColumnInfo();
+                    info.setName(s.getName());
+                    info.setType(s.getType());
+                    return info;
+                })
+                .collect(Collectors.toList());
+    }
+
+    /**
+     * Add column to hudi table at the tail
+     */
+    public void addColumns(String tableName, List<HudiColumnInfo> columns) throws TException {
+        Table hiveTable = client.getTable(dbName, tableName);
+        Table newHiveTable = hiveTable.deepCopy();
+        List<FieldSchema> cols = newHiveTable.getSd().getCols();
+        for (HudiColumnInfo column : columns) {
+            FieldSchema fieldSchema = new FieldSchema();
+            fieldSchema.setName(column.getName());
+            fieldSchema.setType(column.getType());
+            fieldSchema.setComment(column.getDesc());
+            cols.add(fieldSchema);
+        }
+        newHiveTable.getSd().setCols(cols);
+        client.alter_table(dbName, tableName, newHiveTable);
+    }
+
+    /**
+     * Create hudi table and register to hive metastore
+     * @param tableName the hudi table name
+     * @param tableInfo the hudi table info
+     * @param useRealTimeInputFormat ignore uber input Format 
+     */
+    public void createTable(
+            String tableName,
+            HudiTableInfo tableInfo,
+            boolean useRealTimeInputFormat)
+            throws TException, IOException {
+        Table hiveTable = org.apache.hadoop.hive.ql.metadata.Table.getEmptyTable(dbName, tableName);
+        hiveTable.setOwner(UserGroupInformation.getCurrentUser().getUserName());
+        hiveTable.setCreateTime((int) (System.currentTimeMillis() / 1000));
+
+        Map<String, String> properties = new HashMap<>();
+        String location = this.warehouse + "/" + dbName + ".db" + "/" + tableName;
+        properties.put("path", location);
+
+        List<FieldSchema> cols = new ArrayList<>();
+        for (HudiColumnInfo column : tableInfo.getColumns()) {
+            FieldSchema fieldSchema = new FieldSchema();
+            fieldSchema.setName(column.getName());
+            fieldSchema.setType(column.getType());
+            fieldSchema.setComment(column.getDesc());
+            cols.add(fieldSchema);
+        }
+
+        // Build storage of hudi table
+        StorageDescriptor sd = new StorageDescriptor();
+        sd.setCols(cols);
+        hiveTable.setDbName(dbName);
+        hiveTable.setTableName(tableName);
+        // FIXME: splitSchemas need config by frontend
+
+        HoodieFileFormat baseFileFormat = HoodieFileFormat.PARQUET;
+        // ignore uber input Format
+        String inputFormatClassName =
+                HoodieInputFormatUtils.getInputFormatClassName(baseFileFormat, useRealTimeInputFormat);
+        String outputFormatClassName = HoodieInputFormatUtils.getOutputFormatClassName(baseFileFormat);
+        String serDeClassName = HoodieInputFormatUtils.getSerDeClassName(baseFileFormat);
+        sd.setInputFormat(inputFormatClassName);
+        sd.setOutputFormat(outputFormatClassName);
+        Map<String, String> serdeProperties = new HashMap<>();
+        serdeProperties.put("path", location);
+        serdeProperties.put(ConfigUtils.IS_QUERY_AS_RO_TABLE, String.valueOf(!useRealTimeInputFormat));
+        serdeProperties.put("serialization.format", "1");
+
+        sd.setSerdeInfo(new SerDeInfo(null, serDeClassName, serdeProperties));
+

Review Comment:
   Unnecessary blank line.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org