[GitHub] [inlong] liaorui opened a new pull request, #6233: [INLONG-6232][Manager]Support Apache Doris load node management

[GitBox <gi...@apache.org>]

liaorui opened a new pull request, #6233:
URL: https://github.com/apache/inlong/pull/6233

   ### Prepare a Pull Request
   *(Change the title refer to the following example)*
   
   - Title Example: [INLONG-XYZ][Component] Title of the pull request
   
   *(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)*
   
   - Fixes #6232
   
   ### Motivation
   
   *Manager support apache doris load node managerment*
   
   ### Modifications
   
   *1. Add DORIS type in SinkType*
   *2. New classes DorisSink,  DorisSinkDTO, DorisSinkRequest, which contain doris-inlong sort parameters*
   *3. LoadNodeUtils provides DorisLoadNode instance*
   
   ### Verifying this change
   
   *(Please pick either of the following options)*
   
   - [ ] This change is a trivial rework/code cleanup without any test coverage.
   
   - [ ] This change is already covered by existing tests, such as:
     *(please describe tests)*
   
   - [ ] This change added tests and can be verified as follows:
   
     *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (10MB)*
     - *Extended integration test for recovery after broker failure*
   
   ### Documentation
   
     - Does this pull request introduce a new feature? (yes / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
     - If a feature is not applicable for documentation, explain why?
     - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] lucaspeng12138 commented on a diff in pull request #6233: [INLONG-6232][Manager]Support Apache Doris load node management

[GitBox <gi...@apache.org>]

lucaspeng12138 commented on code in PR #6233:
URL: https://github.com/apache/inlong/pull/6233#discussion_r1000087081


##########
inlong-manager/manager-common/src/main/java/org/apache/inlong/manager/common/consts/SinkType.java:
##########
@@ -36,5 +36,5 @@ public class SinkType {
     public static final String ORACLE = "ORACLE";
     public static final String TDSQLPOSTGRESQL = "TDSQLPOSTGRESQL";
     public static final String DLCICEBERG = "DLCICEBERG";
-
+    public static final String DORIS = "DORIS";

Review Comment:
   Shall followed by a blank line?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] dockerzhang merged pull request #6233: [INLONG-6232][Manager] Support Apache Doris load node management

[GitBox <gi...@apache.org>]

dockerzhang merged PR #6233:
URL: https://github.com/apache/inlong/pull/6233


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6233: [INLONG-6232][Manager] Support Apache Doris load node management

[GitBox <gi...@apache.org>]

yunqingmoswu commented on code in PR #6233:
URL: https://github.com/apache/inlong/pull/6233#discussion_r1005404793


##########
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sort/util/LoadNodeUtils.java:
##########
@@ -291,6 +295,44 @@ public static ClickHouseLoadNode createLoadNode(ClickHouseSink ckSink, List<Fiel
         );
     }
 
+    /**
+     * Create load node of Doris.
+     */
+    public static DorisLoadNode createLoadNode(DorisSink dorisSink, List<FieldInfo> fieldInfos,
+            List<FieldRelation> fieldRelations, Map<String, String> properties) {
+        DataTypeEnum dataType = DataTypeEnum.forName(dorisSink.getSinkMultipleFormat());
+        Format format;
+        switch (dataType) {

Review Comment:
   The format is only used for multiple sink.



##########
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/sink/doris/DorisSinkDTO.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.manager.pojo.sink.doris;
+
+import io.swagger.annotations.ApiModelProperty;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import javax.validation.constraints.NotNull;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.inlong.manager.common.enums.ErrorCodeEnum;
+import org.apache.inlong.manager.common.exceptions.BusinessException;
+import org.apache.inlong.manager.common.util.AESUtils;
+import org.apache.inlong.manager.common.util.JsonUtils;
+
+/**
+ * Sink info of Doris
+ */
+@Data
+@Builder
+@NoArgsConstructor
+@AllArgsConstructor
+public class DorisSinkDTO {
+
+    @ApiModelProperty("Doris FE http address")
+    private String feNodes;
+
+    @ApiModelProperty("Username for doris accessing")
+    private String username;
+
+    @ApiModelProperty("Password for doris accessing")
+    private String password;
+
+    @ApiModelProperty("Doris table name, such as: db.tbl")
+    private String tableIdentifier;
+
+    @ApiModelProperty("Label prefix for stream loading. Used for guaranteeing Flink EOS semantics, as global unique is "
+            + "needed in 2pc.")
+    private String labelPrefix;
+
+    @ApiModelProperty("The primary key of sink table")
+    private String primaryKey;
+
+    @ApiModelProperty("The multiple enable of sink")
+    private Boolean sinkMultipleEnable = false;
+
+    @ApiModelProperty("The multiple format of sink")
+    private String sinkMultipleFormat;
+
+    @ApiModelProperty("The multiple database-pattern of sink")
+    private String databasePattern;
+
+    @ApiModelProperty("The multiple table-pattern of sink")
+    private String tablePattern;
+
+    @ApiModelProperty("Password encrypt version")
+    private Integer encryptVersion;
+
+    @ApiModelProperty("Properties for clickhouse")

Review Comment:
   clickhouse -> Doris



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org