You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by zh...@apache.org on 2022/10/11 06:33:20 UTC
[rocketmq-connect] branch master updated: add doris sink connector (#325)
This is an automated email from the ASF dual-hosted git repository.
zhoubo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/rocketmq-connect.git
The following commit(s) were added to refs/heads/master by this push:
new 1e756a1f add doris sink connector (#325)
1e756a1f is described below
commit 1e756a1f63cb406841f83452d2e10e09513b75b8
Author: Lei Sun <87...@qq.com>
AuthorDate: Tue Oct 11 14:33:16 2022 +0800
add doris sink connector (#325)
* add doris sink connector
* code refactoring
* code refactoring
* update README.md
* pass checkstyle
* add logic to filter white table
* remove useless code
* update README.md
* update README.md
* update README.md
* add pom.xml
---
connectors/rocketmq-connect-doris/README.txt | 170 ++++++
connectors/rocketmq-connect-doris/pom.xml | 255 +++++++++
.../connect/doris/config/AbstractConfig.java | 132 +++++
.../connect/doris/connector/DorisSinkConfig.java | 331 ++++++++++++
.../doris/connector/DorisSinkConnector.java | 74 +++
.../connect/doris/connector/DorisSinkTask.java | 107 ++++
.../connect/doris/exception/ConfigException.java | 36 ++
.../connect/doris/exception/DorisException.java | 34 ++
.../exception/TableAlterOrCreateException.java | 34 ++
.../doris/schema/column/ColumnDefAdjuster.java | 72 +++
.../doris/schema/column/ColumnDefinition.java | 337 ++++++++++++
.../connect/doris/schema/column/ColumnId.java | 105 ++++
.../schema/column/parser/DateColumnParser.java | 61 +++
.../schema/column/parser/TimeColumnParser.java | 63 +++
.../column/parser/TimestampColumnParser.java | 54 ++
.../connect/doris/schema/db/DbStructure.java | 50 ++
.../doris/schema/table/TableDefinition.java | 63 +++
.../doris/schema/table/TableDefinitions.java | 79 +++
.../connect/doris/schema/table/TableId.java | 144 +++++
.../doris/serialization/ByteArrayDeserializer.java | 25 +
.../doris/serialization/ByteArraySerializer.java | 24 +
.../serialization/ByteBufferDeserializer.java | 30 ++
.../doris/serialization/ByteBufferSerializer.java | 41 ++
.../connect/doris/serialization/Deserializer.java | 63 +++
.../doris/serialization/DoubleDeserializer.java | 38 ++
.../doris/serialization/DoubleSerializer.java | 38 ++
.../doris/serialization/FloatDeserializer.java | 38 ++
.../doris/serialization/FloatSerializer.java | 34 ++
.../doris/serialization/IntegerDeserializer.java | 38 ++
.../doris/serialization/IntegerSerializer.java | 33 ++
.../doris/serialization/JsonDeserializer.java | 44 ++
.../doris/serialization/JsonSerializer.java | 50 ++
.../doris/serialization/LongDeserializer.java | 38 ++
.../doris/serialization/LongSerializer.java | 37 ++
.../connect/doris/serialization/Serializer.java | 61 +++
.../doris/serialization/ShortDeserializer.java | 39 ++
.../doris/serialization/ShortSerializer.java | 31 ++
.../doris/serialization/StringDeserializer.java | 49 ++
.../doris/serialization/StringSerializer.java | 51 ++
.../connect/doris/sink/BufferedRecords.java | 234 ++++++++
.../rocketmq/connect/doris/sink/DorisDialect.java | 125 +++++
.../connect/doris/sink/DorisStreamLoader.java | 145 +++++
.../connect/doris/sink/RecordValidator.java | 101 ++++
.../rocketmq/connect/doris/sink/Updater.java | 74 +++
.../doris/sink/metadata/FieldsMetadata.java | 304 +++++++++++
.../connect/doris/sink/metadata/SchemaPair.java | 64 +++
.../doris/sink/metadata/SinkRecordField.java | 90 ++++
.../rocketmq/connect/doris/util/BytesUtil.java | 30 ++
.../connect/doris/util/ConnectorGroupUtils.java | 53 ++
.../rocketmq/connect/doris/util/DateTimeUtils.java | 71 +++
.../connect/doris/util/ExpressionBuilder.java | 595 +++++++++++++++++++++
.../connect/doris/util/IdentifierRules.java | 186 +++++++
.../connect/doris/util/JdbcDriverInfo.java | 136 +++++
.../connect/doris/util/NumericMapping.java | 54 ++
.../rocketmq/connect/doris/util/QuoteMethod.java | 42 ++
.../rocketmq/connect/doris/util/TableType.java | 85 +++
.../rocketmq/connect/doris/util/TableUtil.java | 34 ++
connectors/rocketmq-connect-jdbc/README.md | 121 ++---
58 files changed, 5475 insertions(+), 72 deletions(-)
diff --git a/connectors/rocketmq-connect-doris/README.txt b/connectors/rocketmq-connect-doris/README.txt
new file mode 100644
index 00000000..d47eebb3
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/README.txt
@@ -0,0 +1,170 @@
+# rocketmq-connect-doris
+
+## rocketmq-connect-doirs packaging
+
+```
+mvn clean package -Dmaven.test.skip=true
+```
+
+## rocketmq-connect-doris
+Supporting for database languages:
+| Language | Support |
+|----------------|------------|
+| DDL | NO |
+| DML | YES |
+| DCL | NO |
+| TCL | NO |
+DDL:
+Currently, sink-connect-doris load data to Doris via Stream Load which does not support DDL, so DDL
+ is also not supported by sink-connect-doris.
+DML:
+Sink-connect-doris only support insert mode, and entries should have increment filed.
+DCL:
+Not support now.
+TCL:
+Transaction concept not different in Mysql and Doris, so TCL is not supported.
+
+## How to use
+0. Doris cluster and Rocketmq Cluster should be prepared.
+1. build rocketmq connect (https://github.com/apache/rocketmq-connect).
+2. package rocketmq-connect-jdbc and rocketmq-connect-doris.
+3. run rocketmq connect runtime.
+4. create source-connect-mysql.
+5. create sourcr-connect-doris.
+
+```
+
+* **doris-sink-connector** Start
+
+```
+POST http://${runtime-ip}:${runtime-port}/connectors/${rocketmq-jdbc-sink-connector-name}
+{
+ "connector.class":"org.apache.rocketmq.connect.doris.connector.DorisSinkConnector",
+ "max.task":"1",
+ "table.whitelist":"sink_test.doris_test_sink",
+ "connect.topicnames":"doris_test_sink",
+ "host":"xx.xx.xx.xx",
+ "port":"xxxx",
+ "user":"xx",
+ "passwd":"****",
+ "database":"database",
+ "insert.mode":"INSERT",
+ "db.timezone":"UTC",
+ "table.types":"TABLE",
+ "key.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter",
+ "value.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter"}'
+}
+```
+
+> **Note:
+1.rocketmq-jdbc-connect depends on the startup of the `rocketmq-connect-runtime` project. You will need to put all the `jar` packages into the path configured by `pluginPaths` in the `runtime` project and re-execute the above startup request. This value is configured in the `connect.conf` file under the `runtime` project.
+2.The data pulled by the source connector is written to the topic automatically created with the table name. If you need to write to a specific topic, you need to specify the "connect-topicname" parameter
+3.The table name used by sink-connect-doris is specified by schema in message, which is the same with original mysql table name, so doris table name created should be the same with mysql table name.
+
+## rocketmq-connect-jdbc Stop
+
+```
+http://${runtime-ip}:${runtime-port}/connectors/${rocketmq-doris-connector-name}/stop
+```
+
+* **jdbc-sink-connector 参数说明**
+
+| KEY | TYPE | Must be filled | Description | Example
+|------------------------|----|---------|---------------|-------------------|
+|host | String | YES |doris host | 192.168.0.1 |
+|port | String | YES |doris http port | 8030 |
+|user | String | YES |doris user name | root |
+|passwd | String | YES |doris passwd | passwd |
+|max.task | Integer | NO |task number | 2 |
+|key.converter | String | YES |data converter | org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter |
+|value.converter | String | YES |value converter | org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter |
+```
+
+## Quick Start
+Doris Cluster:
+Fe: 192.168.0.1
+Be: 192.168.0.2 192.168.0.3
+
+Rocketmq Cluster:
+Nameserver: 192.168.0.1
+Broker-a: 192.168.0.2
+Broker-b: 192.168.0.3
+
+Note: run rocketmq-connect and source mysql on 192.168.0.3
+
+1. create mysql source table and insert entries;
+```
+ssh root@192.168.0.3
+mysql -uroot –p
+use sink_test;
+DROP TABLE IF EXISTS `doris_test_sink`;
+CREATE TABLE `doris_test_sink` (
+`id` int not NULL AUTO_INCREMENT COMMENT "",
+`number` int NULL COMMENT "",
+`price` double NULL COMMENT "",
+`skuname` varchar(40) NULL COMMENT "",
+`skudesc` varchar(200) NULL COMMENT "",
+PRIMARY KEY (`id`)
+) ENGINE=InnoDB AUTO_INCREMENT=15 DEFAULT CHARSET=utf8;
+BEGIN;
+INSERT INTO `doris_test_sink` VALUES (10001, 12, 13.3, 'test1','this is atest');
+INSERT INTO `doris_test_sink` VALUES (10002 ,100, 15.3, 'test2', 'this is atest');
+INSERT INTO `doris_test_sink` VALUES (10003, 102, 16.3, 'test3', 'this is atest');
+INSERT INTO `doris_test_sink` VALUES (10004, 120, 17.3, 'test4', 'this is atest');
+COMMIT;
+select * from doris_test_sink;
+```
+
+2. create Doris table
+```
+ssh root@192.168.0.1
+mysql -uroot -p
+use db_1;
+DROP TABLE IF EXISTS `doris_test_sink`;
+CREATE TABLE `doris_test_sink` (
+ `id` int not NULL COMMENT "",
+ `number` int NULL COMMENT "",
+ `price` double NULL COMMENT "",
+ `skuname` varchar(40) NULL COMMENT "",
+ `skudesc` varchar(200) NULL COMMENT ""
+) ENGINE=OLAP
+DUPLICATE KEY(`id`)
+COMMENT "OLAP"
+DISTRIBUTED BY HASH(`id`) BUCKETS 1
+PROPERTIES (
+"replication_num" = "1",
+"in_memory" = "false",
+"storage_format" = "V2"
+);
+select * from doris_test_sink;
+```
+
+3. run Rocketmq Connect
+```
+ssh root@192.168.0.3
+cd rocketmq-connect
+builde rocketmq-connect
+mvn -Prelease-connect -DskipTests clean install –U
+```
+build rocketmq-connect-jdbc
+build rocketmq-connect-doris
+modify connect-standalone.conf specialize nameserver ip and plugins path
+```
+cd distribution/target/rocketmq-connect-0.0.1-SNAPSHOT/rocketmq-connect-0.0.1-SNAPSHOT
+sh bin/connect-standalone.sh -c path/to/connect-standalone.conf
+```
+4. create source and sink connector
+```
+ssh root@192.168.0.3
+curl -X POST -H "Content-Type: application/json" http://127.0.0.1:8082/connectors/jdbc-mysql-source-test -d '{"connector.class":"org.apache.rocketmq.connect.jdbc.connector.JdbcSourceConnector","max-task":"1","connect.topicname":"doris_test_sink","connection.url":"jdbc:mysql://127.0.0.1:3306","connection.user":"****","connection.password":"*****","table.whitelist":"sink_test.doris_test_sink","mode": "incrementing","incrementing.column.name":"id","source-record-converter":"org.apache.rock [...]
+curl -X POST -H "Content-Type: application/json" http://127.0.0.1:8082/connectors/jdbc-doris-sink-test -d '{"connector.class":"org.apache.rocketmq.connect.doris.connector.DorisSinkConnector","max-task":"1","table.whitelist":"sink_test.doris_test_sink","connect.topicname":"doris_test_sink","connect.topicnames":"doris_test_sink","host":"192.168.0.1","port":"7030","user":"****","passwd":"*****","database":"db_1","insert.mode":"INSERT","db.timezone":"UTC","table.types":"TABLE","auto.create": [...]
+```
+
+5. verify result on Doris
+```
+ssh root@192.168.0.1
+mysql –uroot -p
+use db_1;
+select * from doris_test_sink;
+```
+
diff --git a/connectors/rocketmq-connect-doris/pom.xml b/connectors/rocketmq-connect-doris/pom.xml
new file mode 100644
index 00000000..7ed0e952
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/pom.xml
@@ -0,0 +1,255 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor
+ license agreements. See the NOTICE file distributed with this work for additional
+ information regarding copyright ownership. The ASF licenses this file to
+ You under the Apache License, Version 2.0 (the "License"); you may not use
+ this file except in compliance with the License. You may obtain a copy of
+ the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
+ by applicable law or agreed to in writing, software distributed under the
+ License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
+ OF ANY KIND, either express or implied. See the License for the specific
+ language governing permissions and limitations under the License. -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.rocketmq</groupId>
+ <artifactId>rocketmq-connect-doris</artifactId>
+ <packaging>jar</packaging>
+ <version>0.0.1-SNAPSHOT</version>
+
+ <name>rocketmq-connect-doris</name>
+ <url>https://github.com/ingdex/rocketmq-connect-doris.git</url>
+
+ <licenses>
+ <license>
+ <name>The Apache Software License, Version 2.0</name>
+ <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
+ </license>
+ </licenses>
+
+ <issueManagement>
+ <system>jira</system>
+ <url>https://issues.apache.org/jira/browse/RocketMQ</url>
+ </issueManagement>
+
+ <properties>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
+ <!-- Compiler settings properties -->
+ <maven.compiler.source>1.8</maven.compiler.source>
+ <maven.compiler.target>1.8</maven.compiler.target>
+ <rocketmq.version>4.7.0</rocketmq.version>
+
+ <!--test jar-->
+ <junit.version>4.13.1</junit.version>
+ <assertj.version>2.6.0</assertj.version>
+ <mockito.version>2.6.3</mockito.version>
+ <!--rocket connect api-->
+ <openmessaging-connector.version>0.1.4</openmessaging-connector.version>
+ <openmessaging-api.version>0.3.1-alpha</openmessaging-api.version>
+
+ <!--fast json-->
+ <fastjson.version>1.2.83</fastjson.version>
+
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-lang3</artifactId>
+ <version>3.12.0</version>
+ </dependency>
+ <!--rocketmq connect api-->
+ <dependency>
+ <groupId>io.openmessaging</groupId>
+ <artifactId>openmessaging-connector</artifactId>
+ <version>${openmessaging-connector.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>io.openmessaging</groupId>
+ <artifactId>openmessaging-api</artifactId>
+ <version>${openmessaging-api.version}</version>
+ </dependency>
+
+ <!--junit -->
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>${junit.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.assertj</groupId>
+ <artifactId>assertj-core</artifactId>
+ <version>${assertj.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-core</artifactId>
+ <version>${mockito.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <!--fast json version-->
+ <dependency>
+ <groupId>com.alibaba</groupId>
+ <artifactId>fastjson</artifactId>
+ <version>${fastjson.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-classic</artifactId>
+ <version>1.2.0</version>
+ </dependency>
+ <dependency>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-core</artifactId>
+ <version>1.2.9</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <version>4.5.13</version>
+ </dependency>
+ </dependencies>
+
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>versions-maven-plugin</artifactId>
+ <version>2.3</version>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>clirr-maven-plugin</artifactId>
+ <version>2.7</version>
+ </plugin>
+ <plugin>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <configuration>
+ <outputDirectory>${project.build.directory}/lib</outputDirectory>
+ <excludeTransitive>false</excludeTransitive>
+ <stripVersion>true</stripVersion>
+ </configuration>
+ </plugin>
+ <plugin>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <version>3.6.1</version>
+ <configuration>
+ <source>${maven.compiler.source}</source>
+ <target>${maven.compiler.target}</target>
+ <compilerVersion>${maven.compiler.source}</compilerVersion>
+ <showDeprecation>true</showDeprecation>
+ <showWarnings>true</showWarnings>
+ </configuration>
+ </plugin>
+ <plugin>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <version>2.19.1</version>
+ <configuration>
+ <argLine>-Xms512m -Xmx1024m</argLine>
+ <forkMode>always</forkMode>
+ <includes>
+ <include>**/*Test.java</include>
+ </includes>
+ </configuration>
+ </plugin>
+ <plugin>
+ <artifactId>maven-site-plugin</artifactId>
+ <version>3.6</version>
+ <configuration>
+ <locales>en_US</locales>
+ <outputEncoding>UTF-8</outputEncoding>
+ <inputEncoding>UTF-8</inputEncoding>
+ </configuration>
+ </plugin>
+ <plugin>
+ <artifactId>maven-source-plugin</artifactId>
+ <version>3.0.1</version>
+ <executions>
+ <execution>
+ <id>attach-sources</id>
+ <goals>
+ <goal>jar</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <artifactId>maven-javadoc-plugin</artifactId>
+ <version>2.10.4</version>
+ <configuration>
+ <charset>UTF-8</charset>
+ <locale>en_US</locale>
+ <excludePackageNames>io.openmessaging.internal</excludePackageNames>
+ </configuration>
+ <executions>
+ <execution>
+ <id>aggregate</id>
+ <goals>
+ <goal>aggregate</goal>
+ </goals>
+ <phase>site</phase>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <artifactId>maven-resources-plugin</artifactId>
+ <version>3.0.2</version>
+ <configuration>
+ <encoding>${project.build.sourceEncoding}</encoding>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>findbugs-maven-plugin</artifactId>
+ <version>3.0.4</version>
+ </plugin>
+ <plugin>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <version>3.0.0</version>
+ <configuration>
+ <descriptorRefs>
+ <descriptorRef>jar-with-dependencies</descriptorRef>
+ </descriptorRefs>
+ </configuration>
+ <executions>
+ <execution>
+ <id>make-assembly</id>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <artifactId>maven-checkstyle-plugin</artifactId>
+ <version>2.17</version>
+ <executions>
+ <execution>
+ <id>verify</id>
+ <phase>verify</phase>
+ <configuration>
+ <configLocation>../../style/rmq_checkstyle.xml</configLocation>
+ <encoding>UTF-8</encoding>
+ <consoleOutput>true</consoleOutput>
+ <failsOnError>true</failsOnError>
+ <includeTestSourceDirectory>false</includeTestSourceDirectory>
+ <includeTestResources>false</includeTestResources>
+ </configuration>
+ <goals>
+ <goal>check</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/config/AbstractConfig.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/config/AbstractConfig.java
new file mode 100644
index 00000000..5ff2b2ae
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/config/AbstractConfig.java
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.config;
+import io.openmessaging.KeyValue;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/**
+ * abstract config
+ */
+public abstract class AbstractConfig {
+
+ private static final Pattern COMMA_WITH_WHITESPACE = Pattern.compile("\\s*,\\s*");
+
+ // connection url
+ public static final String CONNECTION_PREFIX = "connection.";
+ public static final String CONNECTION_URL_CONFIG = CONNECTION_PREFIX + "url";
+ // connection user
+ public static final String CONNECTION_USER_CONFIG = CONNECTION_PREFIX + "user";
+ private static final String CONNECTION_USER_DOC = "JDBC connection user.";
+ // connection password
+ public static final String CONNECTION_PASSWORD_CONFIG = CONNECTION_PREFIX + "password";
+ private static final String CONNECTION_PASSWORD_DOC = "JDBC connection password.";
+ // connection attempts
+ public static final String CONNECTION_ATTEMPTS_CONFIG = CONNECTION_PREFIX + "attempts";
+ public static final String CONNECTION_ATTEMPTS_DOC = "Maximum number of attempts to retrieve a valid JDBC connection.Must be a positive integer.";
+ public static final int CONNECTION_ATTEMPTS_DEFAULT = 3;
+ // backoff ms
+ public static final String CONNECTION_BACKOFF_CONFIG = CONNECTION_PREFIX + "backoff.ms";
+ public static final String CONNECTION_BACKOFF_DOC = "Backoff time in milliseconds between connection attempts.";
+ public static final long CONNECTION_BACKOFF_DEFAULT = 10000L;
+ /**
+ * quote.sql.identifiers
+ */
+ public static final String QUOTE_SQL_IDENTIFIERS_CONFIG = "quote.sql.identifiers";
+ public static final String QUOTE_SQL_IDENTIFIERS_DOC =
+ "When to quote table names, column names, and other identifiers in SQL statements. "
+ + "For backward compatibility, the default is ``always``.";
+
+
+ private String connectionDbUrl;
+ private String connectionDbUser;
+ private String connectionDbPassword;
+ private Integer attempts;
+ private Long backoffMs;
+ private String quoteSqlIdentifiers;
+
+ public AbstractConfig(KeyValue config) {
+ connectionDbUrl = config.getString(CONNECTION_URL_CONFIG);
+ connectionDbUser = config.getString(CONNECTION_USER_CONFIG);
+ connectionDbPassword = config.getString(CONNECTION_PASSWORD_CONFIG);
+ attempts = config.getInt(CONNECTION_ATTEMPTS_CONFIG, CONNECTION_ATTEMPTS_DEFAULT);
+ backoffMs = config.getLong(CONNECTION_BACKOFF_CONFIG, CONNECTION_BACKOFF_DEFAULT);
+ }
+
+
+ public String getConnectionDbUrl() {
+ return connectionDbUrl;
+ }
+
+ public String getConnectionDbUser() {
+ return connectionDbUser;
+ }
+
+ public String getConnectionDbPassword() {
+ return connectionDbPassword;
+ }
+
+ public Integer getAttempts() {
+ return attempts;
+ }
+
+ public Long getBackoffMs() {
+ return backoffMs;
+ }
+
+ public String getQuoteSqlIdentifiers() {
+ return quoteSqlIdentifiers;
+ }
+
+ /**
+ * get list
+ *
+ * @param config
+ * @param key
+ * @return
+ */
+ protected List<String> getList(KeyValue config, String key) {
+ if (!config.containsKey(key) || Objects.isNull(config.getString(key))) {
+ return new ArrayList<>();
+ }
+ return Arrays.asList(COMMA_WITH_WHITESPACE.split(config.getString(key), -1));
+ }
+
+ /**
+ * get list
+ *
+ * @param config
+ * @param key
+ * @return
+ */
+ protected List<String> getList(KeyValue config, String key, String defaultValue) {
+ if (config.containsKey(key) || Objects.isNull(config.getString(key))) {
+ return Collections.singletonList(defaultValue);
+ }
+ return Arrays.asList(COMMA_WITH_WHITESPACE.split(config.getString(key), -1));
+ }
+
+ protected Boolean getBoolean(KeyValue config, String key, Boolean defaultValue) {
+ return config.containsKey(key) ? Boolean.parseBoolean(config.getString(key)) : defaultValue;
+ }
+
+}
+
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/connector/DorisSinkConfig.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/connector/DorisSinkConfig.java
new file mode 100644
index 00000000..5cbe294d
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/connector/DorisSinkConfig.java
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.connector;
+
+import io.openmessaging.KeyValue;
+import org.apache.rocketmq.connect.doris.config.AbstractConfig;
+import org.apache.rocketmq.connect.doris.exception.ConfigException;
+import org.apache.rocketmq.connect.doris.schema.table.TableId;
+import org.apache.rocketmq.connect.doris.util.TableType;
+import org.apache.rocketmq.connect.doris.util.TableUtil;
+
+import java.time.ZoneId;
+import java.util.EnumSet;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TimeZone;
+import java.util.stream.Collectors;
+
+/**
+ * jdbc sink config
+ */
+public class DorisSinkConfig extends AbstractConfig {
+
+ public enum InsertMode {
+ INSERT,
+ UPSERT,
+ UPDATE;
+
+ }
+
+ public enum PrimaryKeyMode {
+ NONE,
+ RECORD_KEY,
+ RECORD_VALUE;
+ }
+ public static final String HOST = "host";
+ public static final String PORT = "port";
+ public static final String DATABASE = "database";
+ public static final String USER = "user";
+ public static final String PASSWD = "passwd";
+ public static final String TABLE_NAME_FORMAT = "table.name.format";
+ public static final String TABLE_NAME_FORMAT_DEFAULT = "${topic}";
+ private static final String TABLE_NAME_FORMAT_DISPLAY = "Table Name Format";
+
+ /**
+ * table name from header
+ */
+ public static final String TABLE_NAME_FROM_HEADER = "table.name.from.header";
+ private static final boolean TABLE_NAME_FROM_HEADER_DEFAULT = Boolean.FALSE;
+ private static final String TABLE_NAME_FROM_HEADER_DISPLAY = "Table from header";
+
+ /**
+ * max retries
+ */
+ public static final String MAX_RETRIES = "max.retries";
+ private static final int MAX_RETRIES_DEFAULT = 10;
+ private static final String MAX_RETRIES_DOC =
+ "The maximum number of times to retry on errors before failing the task.";
+
+ public static final String RETRY_BACKOFF_MS = "retry.backoff.ms";
+ private static final int RETRY_BACKOFF_MS_DEFAULT = 3000;
+
+ public static final String BATCH_SIZE = "batch.size";
+ private static final int BATCH_SIZE_DEFAULT = 100;
+
+
+ public static final String DELETE_ENABLED = "delete.enabled";
+ private static final boolean DELETE_ENABLED_DEFAULT = false;
+
+
+ public static final String AUTO_CREATE = "auto.create";
+ private static final boolean AUTO_CREATE_DEFAULT = false;
+
+ public static final String AUTO_EVOLVE = "auto.evolve";
+ private static final boolean AUTO_EVOLVE_DEFAULT = false;
+ private static final String AUTO_EVOLVE_DOC =
+ "Whether to automatically add columns in the table schema when found to be missing relative "
+ + "to the record schema by issuing ``ALTER``.";
+
+ public static final String INSERT_MODE = "insert.mode";
+ private static final String INSERT_MODE_DEFAULT = "insert";
+ private static final String INSERT_MODE_DOC =
+ "The insertion mode to use. Supported modes are:\n"
+ + "``insert``\n"
+ + " Use standard SQL ``INSERT`` statements.\n"
+ + "``upsert``\n"
+ + " Use the appropriate upsert semantics for the target database if it is supported by "
+ + "the connector, e.g. ``INSERT OR IGNORE``.\n"
+ + "``update``\n"
+ + " Use the appropriate update semantics for the target database if it is supported by "
+ + "the connector, e.g. ``UPDATE``.";
+ private static final String INSERT_MODE_DISPLAY = "Insert Mode";
+
+
+ public static final String PK_FIELDS = "pk.fields";
+ private static final String PK_FIELDS_DEFAULT = "";
+ private static final String PK_FIELDS_DOC =
+ "List of comma-separated primary key field names. The runtime interpretation of this config"
+ + " depends on the ``pk.mode``:\n"
+ + "``none``\n"
+ + " Ignored as no fields are used as primary key in this mode.\n"
+ + "``record_key``\n"
+ + " If empty, all fields from the key struct will be used, otherwise used to extract the"
+ + " desired fields - for primitive key only a single field name must be configured.\n"
+ + "``record_value``\n"
+ + " If empty, all fields from the value struct will be used, otherwise used to extract "
+ + "the desired fields.";
+ private static final String PK_FIELDS_DISPLAY = "Primary Key Fields";
+
+ public static final String PK_MODE = "pk.mode";
+ private static final String PK_MODE_DEFAULT = "none";
+ private static final String PK_MODE_DOC =
+ "The primary key mode, also refer to ``" + PK_FIELDS + "`` documentation for interplay. "
+ + "Supported modes are:\n"
+ + "``none``\n"
+ + " No keys utilized.\n"
+ + "``record_value``\n"
+ + " Field(s) from the record value are used, which must be a struct.";
+ private static final String PK_MODE_DISPLAY = "Primary Key Mode";
+
+ public static final String FIELDS_WHITELIST = "fields.whitelist";
+ private static final String FIELDS_WHITELIST_DEFAULT = "";
+ private static final String FIELDS_WHITELIST_DOC =
+ "List of comma-separated record value field names. If empty, all fields from the record "
+ + "value are utilized, otherwise used to filter to the desired fields.\n"
+ + "Note that ``" + PK_FIELDS + "`` is applied independently in the context of which field"
+ + "(s) form the primary key columns in the destination database,"
+ + " while this configuration is applicable for the other columns.";
+ private static final String FIELDS_WHITELIST_DISPLAY = "Fields Whitelist";
+
+
+ public static final String DIALECT_NAME_CONFIG = "dialect.name";
+ public static final String DIALECT_NAME_DEFAULT = "";
+
+
+ public static final String DB_TIMEZONE_CONFIG = "db.timezone";
+ public static final String DB_TIMEZONE_DEFAULT = "UTC";
+
+ // table types
+ public static final String TABLE_TYPES_CONFIG = "table.types";
+ public static final String TABLE_TYPES_DEFAULT = TableType.TABLE.toString();
+ private static final String TABLE_TYPES_DOC =
+ "The comma-separated types of database tables to which the sink connector can write. "
+ + "By default this is ``" + TableType.TABLE + "``, but any combination of ``"
+ + TableType.TABLE + "`` and ``" + TableType.VIEW + "`` is allowed. Not all databases "
+ + "support writing to views, and when they do the the sink connector will fail if the "
+ + "view definition does not match the records' schemas (regardless of ``"
+ + AUTO_EVOLVE + "``).";
+
+ // white list tables
+ public static final String TABLE_WHITE_LIST_CONFIG = "tables.whitelist";
+ public static final String TABLE_NAME = "tableName";
+ public static final String TABLE_WHITE_LIST_DEFAULT = "";
+ private static final String TABLE_WHITE_LIST_DOC =
+ "Table white list.<br>db1.table01,db1.table02</br>";
+
+
+ private String tableNameFormat;
+ private boolean tableFromHeader;
+ private int maxRetries;
+ private int retryBackoffMs;
+ private int batchSize;
+ private boolean deleteEnabled;
+ private boolean autoCreate;
+ private boolean autoEvolve;
+ private InsertMode insertMode;
+ public final PrimaryKeyMode pkMode;
+ private List<String> pkFields;
+ private Set<String> fieldsWhitelist;
+ private Set<String> tableWhitelist;
+ private TimeZone timeZone;
+ private EnumSet<TableType> tableTypes;
+ public String getHost() {
+ return host;
+ }
+
+ public int getPort() {
+ return port;
+ }
+
+ public String getDatabase() {
+ return database;
+ }
+
+ public String getUser() {
+ return user;
+ }
+
+ public String getPasswd() {
+ return passwd;
+ }
+
+ private String host;
+ private int port;
+ private String database;
+ private String user;
+ private String passwd;
+
+ public DorisSinkConfig(KeyValue config) {
+ super(config);
+ host = config.getString(HOST).trim();
+ port = config.getInt(PORT);
+ database = config.getString(DATABASE).trim();
+ user = config.getString(USER).trim();
+ passwd = config.getString(PASSWD).trim();
+
+ tableNameFormat = config.getString(TABLE_NAME_FORMAT, TABLE_NAME_FORMAT_DEFAULT).trim();
+ tableFromHeader = getBoolean(config, TABLE_NAME_FROM_HEADER, false);
+ batchSize = config.getInt(BATCH_SIZE, BATCH_SIZE_DEFAULT);
+
+ maxRetries = config.getInt(MAX_RETRIES, MAX_RETRIES_DEFAULT);
+ retryBackoffMs = config.getInt(RETRY_BACKOFF_MS, RETRY_BACKOFF_MS_DEFAULT);
+ autoCreate = getBoolean(config, AUTO_CREATE, AUTO_CREATE_DEFAULT);
+ autoEvolve = getBoolean(config, AUTO_EVOLVE, AUTO_EVOLVE_DEFAULT);
+ if (Objects.nonNull(config.getString(INSERT_MODE))) {
+ insertMode = InsertMode.valueOf(config.getString(INSERT_MODE, INSERT_MODE_DEFAULT).toUpperCase());
+ }
+ deleteEnabled = getBoolean(config, DELETE_ENABLED, DELETE_ENABLED_DEFAULT);
+ pkMode = PrimaryKeyMode.valueOf(config.getString(PK_MODE, PK_MODE_DEFAULT).toUpperCase());
+ pkFields = getList(config, PK_FIELDS);
+ if (deleteEnabled && pkMode != PrimaryKeyMode.RECORD_KEY) {
+ throw new ConfigException(
+ "Primary key mode must be 'record_key' when delete support is enabled");
+ }
+ fieldsWhitelist = new HashSet<>(getList(config, FIELDS_WHITELIST));
+ // table white list
+ tableWhitelist = new HashSet<>(getList(config, TABLE_WHITE_LIST_CONFIG));
+ String dbTimeZone = config.getString(DB_TIMEZONE_CONFIG, DB_TIMEZONE_DEFAULT);
+ timeZone = TimeZone.getTimeZone(ZoneId.of(dbTimeZone));
+ tableTypes = TableType.parse(getList(config, TABLE_TYPES_CONFIG, TABLE_TYPES_DEFAULT));
+ }
+
+ public String getTableNameFormat() {
+ return tableNameFormat;
+ }
+
+ public boolean isTableFromHeader() {
+ return tableFromHeader;
+ }
+
+ public int getMaxRetries() {
+ return maxRetries;
+ }
+
+ public int getRetryBackoffMs() {
+ return retryBackoffMs;
+ }
+
+ public int getBatchSize() {
+ return batchSize;
+ }
+
+ public boolean isDeleteEnabled() {
+ return deleteEnabled;
+ }
+
+ public boolean isAutoCreate() {
+ return autoCreate;
+ }
+
+ public boolean isAutoEvolve() {
+ return autoEvolve;
+ }
+
+ public InsertMode getInsertMode() {
+ return insertMode;
+ }
+
+ public PrimaryKeyMode getPkMode() {
+ return pkMode;
+ }
+
+ public List<String> getPkFields() {
+ return pkFields;
+ }
+
+ public Set<String> getFieldsWhitelist() {
+ return fieldsWhitelist;
+ }
+
+ public Set<String> getTableWhitelist() {
+ return tableWhitelist;
+ }
+
+ public TimeZone getTimeZone() {
+ return timeZone;
+ }
+
+ public EnumSet<TableType> getTableTypes() {
+ return tableTypes;
+ }
+
+ public boolean filterWhiteTable(TableId tableId) {
+ // not filter table
+ if (tableWhitelist.isEmpty()) {
+ return true;
+ }
+ for (String tableName : tableWhitelist) {
+ TableId table = TableUtil.parseToTableId(tableName);
+ if (table.catalogName() != null && table.catalogName().equals(tableId.catalogName())) {
+ return true;
+ }
+ if (table.tableName().equals(tableId.tableName())) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ public Set<String> tableTypeNames() {
+ return tableTypes.stream().map(TableType::toString).collect(Collectors.toSet());
+ }
+
+}
+
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/connector/DorisSinkConnector.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/connector/DorisSinkConnector.java
new file mode 100644
index 00000000..a2e3119f
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/connector/DorisSinkConnector.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.connector;
+
+import io.openmessaging.KeyValue;
+import io.openmessaging.connector.api.component.task.Task;
+import io.openmessaging.connector.api.component.task.sink.SinkConnector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class DorisSinkConnector extends SinkConnector {
+
+ private static final Logger log = LoggerFactory.getLogger(DorisSinkConnector.class);
+ private KeyValue connectConfig;
+
+ public void start(KeyValue config) {
+ this.connectConfig = config;
+ }
+
+ /**
+ * Should invoke before start the connector.
+ *
+ * @param config
+ * @return error message
+ */
+ @Override
+ public void validate(KeyValue config) {
+ // do validate config
+ }
+
+ @Override
+ public void stop() {
+ this.connectConfig = null;
+ }
+
+ /**
+ * Returns a set of configurations for Tasks based on the current configuration,
+ * producing at most count configurations.
+ *
+ * @param maxTasks maximum number of configurations to generate
+ * @return configurations for Tasks
+ */
+ @Override
+ public List<KeyValue> taskConfigs(int maxTasks) {
+ log.info("Starting task config !!! ");
+ List<KeyValue> configs = new ArrayList<>();
+ for (int i = 0; i < maxTasks; i++) {
+ configs.add(this.connectConfig);
+ }
+ return configs;
+ }
+
+ @Override
+ public Class<? extends Task> taskClass() {
+ return DorisSinkTask.class;
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/connector/DorisSinkTask.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/connector/DorisSinkTask.java
new file mode 100644
index 00000000..f775b092
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/connector/DorisSinkTask.java
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.connect.doris.connector;
+
+import io.openmessaging.KeyValue;
+import io.openmessaging.connector.api.component.task.sink.ErrorRecordReporter;
+import io.openmessaging.connector.api.component.task.sink.SinkTask;
+import io.openmessaging.connector.api.component.task.sink.SinkTaskContext;
+import io.openmessaging.connector.api.data.ConnectRecord;
+import io.openmessaging.connector.api.errors.ConnectException;
+import io.openmessaging.connector.api.errors.RetriableException;
+import org.apache.rocketmq.connect.doris.exception.TableAlterOrCreateException;
+import org.apache.rocketmq.connect.doris.sink.Updater;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.SQLException;
+import java.util.List;
+
+/**
+ * jdbc sink task
+ */
+public class DorisSinkTask extends SinkTask {
+
+ private static final Logger log = LoggerFactory.getLogger(DorisSinkTask.class);
+ private SinkTaskContext context;
+ private ErrorRecordReporter errorRecordReporter;
+ private KeyValue originalConfig;
+ private DorisSinkConfig config;
+ int remainingRetries;
+ private Updater updater;
+
+
+ /**
+ * Put the records to the sink
+ *
+ * @param records
+ */
+ @Override
+ public void put(List<ConnectRecord> records) throws ConnectException {
+ if (records.isEmpty()) {
+ return;
+ }
+ final int recordsCount = records.size();
+ log.debug("Received {} records.", recordsCount);
+ try {
+ updater.write(records);
+ } catch (TableAlterOrCreateException tace) {
+ throw tace;
+ } catch (SQLException sqle) {
+ SQLException sqlAllMessagesException = getAllMessagesException(sqle);
+ if (remainingRetries > 0) {
+// updater.closeQuietly();
+ start(originalConfig);
+ remainingRetries--;
+ throw new RetriableException(sqlAllMessagesException);
+ }
+
+ }
+ remainingRetries = config.getMaxRetries();
+ }
+
+ private SQLException getAllMessagesException(SQLException sqle) {
+ String sqleAllMessages = "Exception chain:" + System.lineSeparator();
+ for (Throwable e : sqle) {
+ sqleAllMessages += e + System.lineSeparator();
+ }
+ SQLException sqlAllMessagesException = new SQLException(sqleAllMessages);
+ sqlAllMessagesException.setNextException(sqle);
+ return sqlAllMessagesException;
+ }
+
+ /**
+ * Start the component
+ *
+ * @param keyValue
+ */
+ @Override
+ public void start(KeyValue keyValue) {
+ originalConfig = keyValue;
+ config = new DorisSinkConfig(keyValue);
+ remainingRetries = config.getMaxRetries();
+ log.info("Initializing doris writer");
+ this.updater = new Updater(config);
+ }
+
+ @Override
+ public void stop() {
+ log.info("Stopping task");
+ }
+}
+
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/exception/ConfigException.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/exception/ConfigException.java
new file mode 100644
index 00000000..f74873f5
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/exception/ConfigException.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.exception;
+
+/**
+ * config exception
+ */
+public class ConfigException extends RuntimeException {
+ private static final long serialVersionUID = 1L;
+
+ public ConfigException(String message) {
+ super(message);
+ }
+
+ public ConfigException(String name, Object value) {
+ this(name, value, null);
+ }
+
+ public ConfigException(String name, Object value, String message) {
+ super("Invalid value " + value + " for configuration " + name + (message == null ? "" : ": " + message));
+ }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/exception/DorisException.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/exception/DorisException.java
new file mode 100644
index 00000000..0c991a91
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/exception/DorisException.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.connect.doris.exception;
+
+public class DorisException extends RuntimeException {
+ private static final long serialVersionUID = 2L;
+
+ public DorisException(String message) {
+ super(message);
+ }
+
+ public DorisException(String name, Object value) {
+ this(name, value, null);
+ }
+
+ public DorisException(String name, Object value, String message) {
+ super("Invalid value " + value + " for configuration " + name + (message == null ? "" : ": " + message));
+ }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/exception/TableAlterOrCreateException.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/exception/TableAlterOrCreateException.java
new file mode 100644
index 00000000..288873d2
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/exception/TableAlterOrCreateException.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.exception;
+
+import io.openmessaging.connector.api.errors.ConnectException;
+
+/**
+ * table alter or create exception
+ */
+public class TableAlterOrCreateException extends ConnectException {
+
+ public TableAlterOrCreateException(String reason) {
+ super(reason);
+ }
+
+ public TableAlterOrCreateException(String reason, Throwable throwable) {
+ super(reason, throwable);
+ }
+}
+
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/ColumnDefAdjuster.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/ColumnDefAdjuster.java
new file mode 100644
index 00000000..12b01b94
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/ColumnDefAdjuster.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.schema.column;
+
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.HashMap;
+import java.util.Map;
+
+public class ColumnDefAdjuster {
+ Map<String, ColumnDefinition.Nullability> nullable = new HashMap<>();
+
+ public ColumnDefAdjuster() {
+ }
+
+ public static ColumnDefAdjuster create(Connection conn,
+ String catalogPattern,
+ String schemaPattern,
+ String tablePattern,
+ String columnPattern) {
+ ColumnDefAdjuster adjuster = new ColumnDefAdjuster();
+ try (ResultSet rs = conn.getMetaData().getColumns(
+ catalogPattern, schemaPattern, tablePattern, columnPattern)) {
+ final int rsColumnCount = rs.getMetaData().getColumnCount();
+ while (rs.next()) {
+ final String columnName = rs.getString(4);
+ ColumnDefinition.Nullability nullability;
+ final int nullableValue = rs.getInt(11);
+ switch (nullableValue) {
+ case DatabaseMetaData.columnNoNulls:
+ nullability = ColumnDefinition.Nullability.NOT_NULL;
+ break;
+ case DatabaseMetaData.columnNullable:
+ nullability = ColumnDefinition.Nullability.NULL;
+ break;
+ case DatabaseMetaData.columnNullableUnknown:
+ default:
+ nullability = ColumnDefinition.Nullability.UNKNOWN;
+ break;
+ }
+ adjuster.nullable.put(columnName, nullability);
+ }
+ } catch (SQLException e) {
+ //pass
+ }
+
+ return adjuster;
+ }
+
+ public ColumnDefinition.Nullability nullable(String columnName) {
+ if (nullable == null || !nullable.containsKey(columnName)) {
+ return null;
+ }
+ return nullable.get(columnName);
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/ColumnDefinition.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/ColumnDefinition.java
new file mode 100644
index 00000000..3bcd3b8c
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/ColumnDefinition.java
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.schema.column;
+
+import java.sql.Types;
+import java.util.Objects;
+
+/**
+ * column definition
+ */
+public class ColumnDefinition {
+
+ /**
+ * The nullability of a column.
+ */
+ public enum Nullability {
+ NULL, NOT_NULL, UNKNOWN
+ }
+
+ /**
+ * The mutability of a column.
+ */
+ public enum Mutability {
+ READ_ONLY, MAYBE_WRITABLE, WRITABLE, UNKNOWN
+ }
+
+ private final ColumnId id;
+ private final String typeName;
+ private final int jdbcType;
+ private final int displaySize;
+ private final int precision;
+ private final int scale;
+ private final boolean autoIncremented;
+ private final boolean caseSensitive;
+ private final boolean searchable;
+ private final boolean currency;
+ private final boolean signedNumbers;
+ private final boolean isPrimaryKey;
+ private final Nullability nullability;
+ private final Mutability mutability;
+ private final String classNameForType;
+
+ public ColumnDefinition(
+ ColumnId id,
+ int jdbcType,
+ String typeName,
+ String classNameForType,
+ Nullability nullability,
+ Mutability mutability,
+ int precision,
+ int scale,
+ boolean signedNumbers,
+ int displaySize,
+ boolean autoIncremented,
+ boolean caseSensitive,
+ boolean searchable,
+ boolean currency,
+ boolean isPrimaryKey
+ ) {
+ this.id = id;
+ this.typeName = typeName;
+ this.jdbcType = jdbcType;
+ this.displaySize = displaySize;
+ this.precision = precision;
+ this.scale = scale;
+ this.autoIncremented = autoIncremented;
+ this.caseSensitive = caseSensitive;
+ this.searchable = searchable;
+ this.currency = currency;
+ this.signedNumbers = signedNumbers;
+ this.nullability = nullability != null ? nullability : Nullability.UNKNOWN;
+ this.mutability = mutability != null ? mutability : Mutability.MAYBE_WRITABLE;
+ this.classNameForType = classNameForType;
+ this.isPrimaryKey = isPrimaryKey;
+ }
+
+
+ /**
+ * Indicates whether the column is automatically numbered.
+ *
+ * @return <code>true</code> if so; <code>false</code> otherwise
+ */
+ public boolean isAutoIncrement() {
+ return autoIncremented;
+ }
+
+ /**
+ * Indicates whether the column's case matters.
+ *
+ * @return <code>true</code> if so; <code>false</code> otherwise
+ */
+ public boolean isCaseSensitive() {
+ return caseSensitive;
+ }
+
+ /**
+ * Indicates whether the column can be used in a where clause.
+ *
+ * @return <code>true</code> if so; <code>false</code> otherwise
+ */
+ public boolean isSearchable() {
+ return searchable;
+ }
+
+ /**
+ * Indicates whether the column is a cash value.
+ *
+ * @return <code>true</code> if so; <code>false</code> otherwise
+ */
+ public boolean isCurrency() {
+ return currency;
+ }
+
+ /**
+ * Indicates whether the column is part of the table's primary key.
+ *
+ * @return <code>true</code> if so; <code>false</code> otherwise
+ */
+ public boolean isPrimaryKey() {
+ return isPrimaryKey;
+ }
+
+ /**
+ * Indicates the nullability of values in the column.
+ *
+ * @return the nullability status of the given column; never null
+ */
+ public Nullability nullability() {
+ return nullability;
+ }
+
+ /**
+ * Indicates whether values in the column are optional. This is equivalent to calling:
+ * <pre>
+ * nullability() == Nullability.NULL || nullability() == Nullability.UNKNOWN
+ * </pre>
+ *
+ * @return <code>true</code> if so; <code>false</code> otherwise
+ */
+ public boolean isOptional() {
+ return nullability == Nullability.NULL || nullability == Nullability.UNKNOWN;
+ }
+
+ /**
+ * Indicates whether values in the column are signed numbers.
+ *
+ * @return <code>true</code> if so; <code>false</code> otherwise
+ */
+ public boolean isSignedNumber() {
+ return signedNumbers;
+ }
+
+ /**
+ * Indicates the column's normal maximum width in characters.
+ *
+ * @return the normal maximum number of characters allowed as the width of the designated column
+ */
+ public int displaySize() {
+ return displaySize;
+ }
+
+ /**
+ * Get the column's identifier.
+ *
+ * @return column identifier; never null
+ */
+ public ColumnId id() {
+ return id;
+ }
+
+ /**
+ * Get the column's table identifier.
+ *
+ * @return the table identifier; never null
+ */
+// public TableId tableId() {
+// return id.tableId();
+// }
+
+ /**
+ * Get the column's specified column size. For numeric data, this is the maximum precision. For
+ * character data, this is the length in characters. For datetime datatypes, this is the length in
+ * characters of the String representation (assuming the maximum allowed precision of the
+ * fractional seconds component). For binary data, this is the length in bytes. For the ROWID
+ * datatype, this is the length in bytes. 0 is returned for data types where the column size is
+ * not applicable.
+ *
+ * @return precision
+ */
+ public int precision() {
+ return precision;
+ }
+
+ /**
+ * Gets the column's number of digits to right of the decimal point. 0 is returned for data types
+ * where the scale is not applicable.
+ *
+ * @return scale
+ */
+ public int scale() {
+ return scale;
+ }
+
+ /**
+ * Retrieves the column's JDBC type.
+ *
+ * @return SQL type from java.sql.Types
+ * @see Types
+ */
+ public int type() {
+ return jdbcType;
+ }
+
+ /**
+ * Retrieves the designated column's database-specific type name.
+ *
+ * @return type name used by the database. If the column type is a user-defined type, then a
+ * fully-qualified type name is returned.
+ */
+ public String typeName() {
+ return typeName;
+ }
+
+ /**
+ * Indicates whether the designated column is mutable.
+ *
+ * @return the mutability; never null
+ */
+ public Mutability mutability() {
+ return mutability;
+ }
+
+ /**
+ * Returns the fully-qualified name of the Java class whose instances are manufactured if the
+ * method {@link java.sql.ResultSet#getObject(int)} is called to retrieve a value from the column.
+ * {@link java.sql.ResultSet#getObject(int)} may return a subclass of the class returned by this
+ * method.
+ *
+ * @return the fully-qualified name of the class in the Java programming language that would be
+ * used by the method <code>ResultSet.getObject</code> to retrieve the value in the specified
+ * column. This is the class name used for custom mapping.
+ */
+ public String classNameForType() {
+ return classNameForType;
+ }
+
+
+ @Override
+ public int hashCode() {
+ return id.hashCode();
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (obj == this) {
+ return true;
+ }
+ if (obj instanceof ColumnDefinition) {
+ ColumnDefinition that = (ColumnDefinition) obj;
+ return Objects.equals(this.id, that.id)
+ && this.jdbcType == that.jdbcType
+ && this.displaySize == that.displaySize
+ && this.precision == that.precision
+ && this.scale == that.scale
+ && this.autoIncremented == that.autoIncremented
+ && this.caseSensitive == that.caseSensitive
+ && this.searchable == that.searchable
+ && this.currency == that.currency
+ && this.signedNumbers == that.signedNumbers
+ && this.nullability == that.nullability
+ && Objects.equals(this.typeName, that.typeName)
+ && Objects.equals(this.classNameForType, that.classNameForType)
+ && Objects.equals(this.isPrimaryKey, that.isPrimaryKey);
+ }
+ return false;
+ }
+
+ @Override
+ public String toString() {
+ return "Column{'" + id.name() + '\'' + ", isPrimaryKey=" + isPrimaryKey + ", allowsNull="
+ + isOptional() + ", sqlType=" + typeName + '}';
+ }
+
+ /**
+ * Obtain a {@link ColumnDefinition} that has all the same characteristics as this column except
+ * that belongs to the specified table.
+ *
+ * @param tableId the new table ID; may not be null
+// * @return this definition if the specified table ID matches this definition's {@link #tableId()},
+ * or a new definition that is a copy of this definition except with an {@link #id() ID} that
+ * uses the specified table; never null
+ */
+// public ColumnDefinition forTable(TableId tableId) {
+// if (tableId().equals(tableId)) {
+// return this;
+// }
+// ColumnId newId = new ColumnId(tableId, this.id().name());
+// return new ColumnDefinition(newId, jdbcType, typeName, classNameForType, nullability,
+// mutability, precision, scale, signedNumbers, displaySize,
+// autoIncremented, caseSensitive, searchable, currency, isPrimaryKey
+// );
+// }
+
+ /**
+ * Obtain a {@link ColumnDefinition} that has all the same characteristics as this column except
+ * that it does or does not belong to the table's primary key
+ *
+ * @param isPrimaryKey true if the resulting column definition should be part of the table's
+ * primary key, or false otherwise
+ * @return a definition that is the same as this definition except it is or is not part of the
+ * tables primary key, or may be this object if {@link #isPrimaryKey()} already matches the
+ * supplied parameter; never null
+ */
+ public ColumnDefinition asPartOfPrimaryKey(boolean isPrimaryKey) {
+ if (isPrimaryKey == isPrimaryKey()) {
+ return this;
+ }
+ return new ColumnDefinition(id, jdbcType, typeName, classNameForType, nullability, mutability,
+ precision, scale, signedNumbers, displaySize, autoIncremented,
+ caseSensitive, searchable, currency, isPrimaryKey
+ );
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/ColumnId.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/ColumnId.java
new file mode 100644
index 00000000..39310c98
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/ColumnId.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.schema.column;
+
+import org.apache.rocketmq.connect.doris.schema.table.TableId;
+import org.apache.rocketmq.connect.doris.util.ExpressionBuilder;
+import org.apache.rocketmq.connect.doris.util.QuoteMethod;
+
+import java.util.Objects;
+
+/**
+ * column id
+ */
+public class ColumnId implements ExpressionBuilder.Expressable {
+
+ private final TableId tableId;
+ private final String name;
+ private final String alias;
+ private final int hash;
+
+ public ColumnId(
+ TableId tableId,
+ String columnName
+ ) {
+ this(tableId, columnName, null);
+ }
+
+ public ColumnId(
+ TableId tableId,
+ String columnName,
+ String alias
+ ) {
+ assert columnName != null;
+ this.tableId = tableId;
+ this.name = columnName;
+ this.alias = alias != null && !alias.trim().isEmpty() ? alias : name;
+ this.hash = Objects.hash(this.tableId, this.name);
+ }
+
+ public TableId tableId() {
+ return tableId;
+ }
+
+ public String name() {
+ return name;
+ }
+
+ public String aliasOrName() {
+ return alias;
+ }
+
+ @Override
+ public void appendTo(ExpressionBuilder builder, boolean useQuotes) {
+ appendTo(builder, useQuotes ? QuoteMethod.ALWAYS : QuoteMethod.NEVER);
+ }
+
+ @Override
+ public void appendTo(
+ ExpressionBuilder builder,
+ QuoteMethod useQuotes
+ ) {
+ if (tableId != null) {
+ builder.append(tableId);
+ builder.appendIdentifierDelimiter();
+ }
+ builder.appendColumnName(this.name, useQuotes);
+ }
+
+ @Override
+ public int hashCode() {
+ return hash;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (obj == this) {
+ return true;
+ }
+ if (obj instanceof ColumnId) {
+ ColumnId that = (ColumnId) obj;
+ return Objects.equals(this.name, that.name) && Objects.equals(this.alias, that.alias)
+ && Objects.equals(this.tableId, that.tableId);
+ }
+ return false;
+ }
+
+ @Override
+ public String toString() {
+ return ExpressionBuilder.create().append(this).toString();
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/parser/DateColumnParser.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/parser/DateColumnParser.java
new file mode 100644
index 00000000..a089f621
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/parser/DateColumnParser.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.schema.column.parser;
+
+import io.openmessaging.connector.api.data.Schema;
+import io.openmessaging.connector.api.data.SchemaBuilder;
+import io.openmessaging.connector.api.errors.ConnectException;
+
+import java.util.Calendar;
+import java.util.TimeZone;
+
+public class DateColumnParser {
+ public static final String LOGICAL_NAME = "org.apache.rocketmq.connect.doris.schema.column.parser.DateColumnParser";
+ private static final long MILLIS_PER_DAY = 86400000L;
+ private static final TimeZone UTC = TimeZone.getTimeZone("UTC");
+ public static final Schema SCHEMA = builder().build();
+
+ public DateColumnParser() {
+ }
+
+ public static SchemaBuilder builder() {
+ return SchemaBuilder.int32().name("org.apache.kafka.connect.data.Date");
+ }
+
+ public static int fromLogical(Schema schema, java.util.Date value) {
+ if (!LOGICAL_NAME.equals(schema.getName())) {
+ throw new ConnectException("Requested conversion of Date object but the schema does not match.");
+ } else {
+ Calendar calendar = Calendar.getInstance(UTC);
+ calendar.setTime(value);
+ if (calendar.get(11) == 0 && calendar.get(12) == 0 && calendar.get(13) == 0 && calendar.get(14) == 0) {
+ long unixMillis = calendar.getTimeInMillis();
+ return (int) (unixMillis / 86400000L);
+ } else {
+ throw new ConnectException("Kafka Connect Date type should not have any time fields set to non-zero values.");
+ }
+ }
+ }
+
+ public static java.util.Date toLogical(Schema schema, int value) {
+ if (!LOGICAL_NAME.equals(schema.getName())) {
+ throw new ConnectException("Requested conversion of Date object but the schema does not match.");
+ } else {
+ return new java.util.Date((long) value * 86400000L);
+ }
+ }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/parser/TimeColumnParser.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/parser/TimeColumnParser.java
new file mode 100644
index 00000000..d1606739
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/parser/TimeColumnParser.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.connect.doris.schema.column.parser;
+
+import io.openmessaging.connector.api.data.Schema;
+import io.openmessaging.connector.api.data.SchemaBuilder;
+import io.openmessaging.connector.api.errors.ConnectException;
+
+import java.util.Calendar;
+import java.util.Date;
+import java.util.TimeZone;
+
+public class TimeColumnParser {
+ public static final String LOGICAL_NAME = "org.apache.rocketmq.connect.doris.schema.column.parser.TimeColumnParser";
+ private static final long MILLIS_PER_DAY = 86400000L;
+ private static final TimeZone UTC = TimeZone.getTimeZone("UTC");
+ public static final Schema SCHEMA = builder().build();
+
+ public static SchemaBuilder builder() {
+ return SchemaBuilder.int32().name("org.apache.rocketmq.connect.doris.schema.column.parser.TimeColumnParser");
+ }
+
+ public static int fromLogical(Schema schema, Date value) {
+ if (!LOGICAL_NAME.equals(schema.getName())) {
+ throw new ConnectException("Requested conversion of Time object but the schema does not match.");
+ } else {
+ Calendar calendar = Calendar.getInstance(UTC);
+ calendar.setTime(value);
+ long unixMillis = calendar.getTimeInMillis();
+ if (unixMillis >= 0L && unixMillis <= 86400000L) {
+ return (int) unixMillis;
+ } else {
+ throw new ConnectException("Kafka Connect Time type should not have any date fields set to non-zero values.");
+ }
+ }
+ }
+
+ public static Date toLogical(Schema schema, int value) {
+ if (!LOGICAL_NAME.equals(schema.getName())) {
+ throw new ConnectException("Requested conversion of Date object but the schema does not match.");
+ } else if (value >= 0 && (long) value <= 86400000L) {
+ return new Date((long) value);
+ } else {
+ throw new ConnectException("Time values must use number of milliseconds greater than 0 and less than 86400000");
+ }
+ }
+
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/parser/TimestampColumnParser.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/parser/TimestampColumnParser.java
new file mode 100644
index 00000000..a095bded
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/column/parser/TimestampColumnParser.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.schema.column.parser;
+
+import io.openmessaging.connector.api.data.Schema;
+import io.openmessaging.connector.api.data.SchemaBuilder;
+import io.openmessaging.connector.api.errors.ConnectException;
+
+import java.util.Date;
+
+/**
+ * time stamp column parser
+ */
+public class TimestampColumnParser {
+ public static final String LOGICAL_NAME = "org.apache.rocketmq.connect.doris.schema.column.parser.TimestampColumnParser";
+ public static final Schema SCHEMA = builder().build();
+
+ public TimestampColumnParser() {
+ }
+
+ public static SchemaBuilder builder() {
+ return SchemaBuilder.int64().name("org.apache.kafka.connect.data.Timestamp");
+ }
+
+ public static long fromLogical(Schema schema, Date value) {
+ if (!LOGICAL_NAME.equals(schema.getName())) {
+ throw new ConnectException("Requested conversion of Timestamp object but the schema does not match.");
+ } else {
+ return value.getTime();
+ }
+ }
+
+ public static Date toLogical(Schema schema, long value) {
+ if (!LOGICAL_NAME.equals(schema.getName())) {
+ throw new ConnectException("Requested conversion of Timestamp object but the schema does not match.");
+ } else {
+ return new Date(value);
+ }
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/db/DbStructure.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/db/DbStructure.java
new file mode 100644
index 00000000..bcc14879
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/db/DbStructure.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.schema.db;
+
+import org.apache.rocketmq.connect.doris.connector.DorisSinkConfig;
+import org.apache.rocketmq.connect.doris.schema.table.TableId;
+import org.apache.rocketmq.connect.doris.sink.metadata.FieldsMetadata;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.sql.SQLException;
+
+/**
+ *
+ */
+public class DbStructure {
+ private static final Logger log = LoggerFactory.getLogger(DbStructure.class);
+
+ /**
+ * Create or amend table.
+ *
+ * @param config the connector configuration
+// * @param connection the database connection handle
+ * @param tableId the table ID
+ * @param fieldsMetadata the fields metadata
+ * @return whether a DDL operation was performed
+ * @throws SQLException if a DDL operation was deemed necessary but failed
+ */
+ public boolean createOrAmendIfNecessary(
+ final DorisSinkConfig config,
+ final TableId tableId,
+ final FieldsMetadata fieldsMetadata
+ ) throws SQLException {
+ // It seems that doris don't support create or amend table via stream load, so do nothing
+ return false;
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/table/TableDefinition.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/table/TableDefinition.java
new file mode 100644
index 00000000..68ca5aaf
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/table/TableDefinition.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.schema.table;
+
+import org.apache.rocketmq.connect.doris.schema.column.ColumnDefinition;
+import org.apache.rocketmq.connect.doris.util.TableType;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+
+/**
+ * A description of a table.
+ */
+public class TableDefinition {
+ private final TableId id;
+ private final Map<String, String> pkColumnNames = new LinkedHashMap<>();
+
+ public TableDefinition(TableId id, Iterable<ColumnDefinition> columns, TableType type) {
+ this.id = id;
+ }
+
+ public TableId id() {
+ return id;
+ }
+
+ @Override
+ public int hashCode() {
+ return id.hashCode();
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (obj == null) {
+ return true;
+ }
+ if (obj instanceof TableDefinition) {
+ TableDefinition that = (TableDefinition) obj;
+ return Objects.equals(this.id(), that.id());
+ }
+ return false;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("Table{name='%s', type=%s columns=%s}", id);
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/table/TableDefinitions.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/table/TableDefinitions.java
new file mode 100644
index 00000000..51a0d153
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/table/TableDefinitions.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.schema.table;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.SQLException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * A simple cache of {@link TableDefinition} keyed.
+ */
+public class TableDefinitions {
+
+ private static final Logger log = LoggerFactory.getLogger(TableDefinitions.class);
+
+ private final Map<TableId, TableDefinition> cache = new HashMap<>();
+// private final DatabaseDialect dialect;
+
+ /**
+ * Create an instance that uses the specified database dialect.
+ *
+ * @param dialect the database dialect; may not be null
+ */
+// public TableDefinitions(DatabaseDialect dialect) {
+// this.dialect = dialect;
+// }
+
+ /**
+ * Get the {@link TableDefinition} for the given table.
+ *
+ * @param tableId the table identifier; may not be null
+ * @return the cached {@link TableDefinition}, or null if there is no such table
+ * @throws SQLException if there is any problem using the connection
+ */
+ public TableDefinition get(
+ final TableId tableId
+ ) {
+ return cache.get(tableId);
+ }
+
+ public boolean contains(
+ final TableId tableId
+ ) {
+ return cache.containsKey(tableId);
+ }
+
+ /**
+ * Refresh the cached {@link TableDefinition} for the given table.
+ *
+ * @param tableId the table identifier; may not be null
+ * @return the refreshed {@link TableDefinition}, or null if there is no such table
+ * @throws SQLException if there is any problem using the connection
+ */
+ public TableDefinition refresh(
+ TableId tableId
+ ) throws SQLException {
+ TableDefinition dbTable = null;
+ log.info("Refreshing metadata for table {} to {}", tableId, dbTable);
+ cache.put(dbTable.id(), dbTable);
+ return dbTable;
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/table/TableId.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/table/TableId.java
new file mode 100644
index 00000000..2b0e821f
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/schema/table/TableId.java
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.schema.table;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.rocketmq.connect.doris.util.ExpressionBuilder;
+import org.apache.rocketmq.connect.doris.util.QuoteMethod;
+
+import java.util.Objects;
+
+public class TableId implements Comparable<TableId>, ExpressionBuilder.Expressable {
+
+ private final String catalogName;
+ private final String schemaName;
+ private final String tableName;
+ private final int hash;
+
+ public TableId(
+ String catalogName,
+ String schemaName,
+ String tableName
+ ) {
+ this.catalogName = catalogName == null || catalogName.isEmpty() ? null : catalogName;
+ this.schemaName = schemaName == null || schemaName.isEmpty() ? null : schemaName;
+ this.tableName = tableName;
+ this.hash = Objects.hash(catalogName, schemaName, tableName);
+ }
+
+ public String catalogName() {
+ return catalogName;
+ }
+
+ public String schemaName() {
+ return schemaName;
+ }
+
+ public String dbName() {
+ if (StringUtils.isNotBlank(catalogName)) {
+ return catalogName;
+ }
+ return schemaName;
+ }
+
+ public String tableName() {
+ return tableName;
+ }
+
+ @Override
+ public void appendTo(ExpressionBuilder builder, boolean useQuotes) {
+ appendTo(builder, useQuotes ? QuoteMethod.ALWAYS : QuoteMethod.NEVER);
+ }
+
+ @Override
+ public void appendTo(
+ ExpressionBuilder builder,
+ QuoteMethod useQuotes
+ ) {
+ if (catalogName != null) {
+ builder.appendIdentifier(catalogName, useQuotes);
+ builder.appendIdentifierDelimiter();
+ }
+ if (schemaName != null) {
+ builder.appendIdentifier(schemaName, useQuotes);
+ builder.appendIdentifierDelimiter();
+ }
+ builder.appendTableName(tableName, useQuotes);
+ }
+
+ @Override
+ public int hashCode() {
+ return hash;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (obj == this) {
+ return true;
+ }
+ if (obj instanceof TableId) {
+ TableId that = (TableId) obj;
+ return Objects.equals(this.catalogName, that.catalogName)
+ && Objects.equals(this.schemaName, that.schemaName)
+ && Objects.equals(this.tableName, that.tableName);
+ }
+ return false;
+ }
+
+ @Override
+ public int compareTo(TableId that) {
+ if (that == this) {
+ return 0;
+ }
+ int diff = this.tableName.compareTo(that.tableName);
+ if (diff != 0) {
+ return diff;
+ }
+ if (this.schemaName == null) {
+ if (that.schemaName != null) {
+ return -1;
+ }
+ } else {
+ if (that.schemaName == null) {
+ return 1;
+ }
+ diff = this.schemaName.compareTo(that.schemaName);
+ if (diff != 0) {
+ return diff;
+ }
+ }
+ if (this.catalogName == null) {
+ if (that.catalogName != null) {
+ return -1;
+ }
+ } else {
+ if (that.catalogName == null) {
+ return 1;
+ }
+ diff = this.catalogName.compareTo(that.catalogName);
+ if (diff != 0) {
+ return diff;
+ }
+ }
+ return 0;
+ }
+
+ @Override
+ public String toString() {
+ return ExpressionBuilder.create().append(this).toString();
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteArrayDeserializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteArrayDeserializer.java
new file mode 100644
index 00000000..f0f79214
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteArrayDeserializer.java
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+public class ByteArrayDeserializer implements Deserializer<byte[]> {
+
+ @Override
+ public byte[] deserialize(String topic, byte[] data) {
+ return data;
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteArraySerializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteArraySerializer.java
new file mode 100644
index 00000000..e7863742
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteArraySerializer.java
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+public class ByteArraySerializer implements Serializer<byte[]> {
+ @Override
+ public byte[] serialize(String topic, byte[] data) {
+ return data;
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteBufferDeserializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteBufferDeserializer.java
new file mode 100644
index 00000000..3aa5996f
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteBufferDeserializer.java
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import java.nio.ByteBuffer;
+
+public class ByteBufferDeserializer implements Deserializer<ByteBuffer> {
+ @Override
+ public ByteBuffer deserialize(String topic, byte[] data) {
+ if (data == null) {
+ return null;
+ }
+
+ return ByteBuffer.wrap(data);
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteBufferSerializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteBufferSerializer.java
new file mode 100644
index 00000000..807eed3a
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ByteBufferSerializer.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import java.nio.ByteBuffer;
+
+public class ByteBufferSerializer implements Serializer<ByteBuffer> {
+ @Override
+ public byte[] serialize(String topic, ByteBuffer data) {
+ if (data == null) {
+ return null;
+ }
+
+ data.rewind();
+ if (data.hasArray()) {
+ byte[] arr = data.array();
+ if (data.arrayOffset() == 0 && arr.length == data.remaining()) {
+ return arr;
+ }
+ }
+
+ byte[] ret = new byte[data.remaining()];
+ data.get(ret, 0, ret.length);
+ data.rewind();
+ return ret;
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/Deserializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/Deserializer.java
new file mode 100644
index 00000000..69e91891
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/Deserializer.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import io.openmessaging.KeyValue;
+
+import java.io.Closeable;
+import java.util.Map;
+
+/**
+ * An interface for converting bytes to objects.
+ *
+ * @param <T> Type to be deserialized into.
+ */
+public interface Deserializer<T> extends Closeable {
+
+ /**
+ * Configure this class.
+ * @param configs configs in key/value pairs
+ */
+ default void configure(Map<String, ?> configs) {
+ // intentionally left blank
+ }
+
+ /**
+ * Deserialize a record value from a byte array into a value or object.
+ * @param topic topic associated with the data
+ * @param data serialized bytes; may be null; implementations are recommended to handle null by returning a value or null rather than throwing an exception.
+ * @return deserialized typed data; may be null
+ */
+ T deserialize(String topic, byte[] data);
+
+ /**
+ * Deserialize a record value from a byte array into a value or object.
+ */
+ default T deserialize(String topic, KeyValue extensions, byte[] data) {
+ return deserialize(topic, data);
+ }
+
+ /**
+ * Close this deserializer.
+ * <p>
+ * This method must be idempotent as it may be called multiple times.
+ */
+ @Override
+ default void close() {
+ // intentionally left blank
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/DoubleDeserializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/DoubleDeserializer.java
new file mode 100644
index 00000000..23a5447a
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/DoubleDeserializer.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import org.apache.commons.lang3.SerializationException;
+
+public class DoubleDeserializer implements Deserializer<Double> {
+
+ @Override
+ public Double deserialize(String topic, byte[] data) {
+ if (data == null)
+ return null;
+ if (data.length != 8) {
+ throw new SerializationException("Size of data received by Deserializer is not 8");
+ }
+
+ long value = 0;
+ for (byte b : data) {
+ value <<= 8;
+ value |= b & 0xFF;
+ }
+ return Double.longBitsToDouble(value);
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/DoubleSerializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/DoubleSerializer.java
new file mode 100644
index 00000000..86ca81f0
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/DoubleSerializer.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+public class DoubleSerializer implements Serializer<Double> {
+ @Override
+ public byte[] serialize(String topic, Double data) {
+ if (data == null) {
+ return null;
+ }
+
+ long bits = Double.doubleToLongBits(data);
+ return new byte[] {
+ (byte) (bits >>> 56),
+ (byte) (bits >>> 48),
+ (byte) (bits >>> 40),
+ (byte) (bits >>> 32),
+ (byte) (bits >>> 24),
+ (byte) (bits >>> 16),
+ (byte) (bits >>> 8),
+ (byte) bits
+ };
+ }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/FloatDeserializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/FloatDeserializer.java
new file mode 100644
index 00000000..b2da4faf
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/FloatDeserializer.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import org.apache.commons.lang3.SerializationException;
+
+public class FloatDeserializer implements Deserializer<Float> {
+ @Override
+ public Float deserialize(final String topic, final byte[] data) {
+ if (data == null) {
+ return null;
+ }
+ if (data.length != 4) {
+ throw new SerializationException("Size of data received by Deserializer is not 4");
+ }
+
+ int value = 0;
+ for (byte b : data) {
+ value <<= 8;
+ value |= b & 0xFF;
+ }
+ return Float.intBitsToFloat(value);
+ }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/FloatSerializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/FloatSerializer.java
new file mode 100644
index 00000000..fc06f2be
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/FloatSerializer.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+public class FloatSerializer implements Serializer<Float> {
+ @Override
+ public byte[] serialize(final String topic, final Float data) {
+ if (data == null) {
+ return null;
+ }
+
+ long bits = Float.floatToRawIntBits(data);
+ return new byte[] {
+ (byte) (bits >>> 24),
+ (byte) (bits >>> 16),
+ (byte) (bits >>> 8),
+ (byte) bits
+ };
+ }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/IntegerDeserializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/IntegerDeserializer.java
new file mode 100644
index 00000000..b9ab9eea
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/IntegerDeserializer.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import org.apache.commons.lang3.SerializationException;
+
+public class IntegerDeserializer implements Deserializer<Integer> {
+ @Override
+ public Integer deserialize(String topic, byte[] data) {
+ if (data == null) {
+ return null;
+ }
+ if (data.length != 4) {
+ throw new SerializationException("Size of data received by IntegerDeserializer is not 4");
+ }
+
+ int value = 0;
+ for (byte b : data) {
+ value <<= 8;
+ value |= b & 0xFF;
+ }
+ return value;
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/IntegerSerializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/IntegerSerializer.java
new file mode 100644
index 00000000..6203ddac
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/IntegerSerializer.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+public class IntegerSerializer implements Serializer<Integer> {
+ @Override
+ public byte[] serialize(String topic, Integer data) {
+ if (data == null) {
+ return null;
+ }
+
+ return new byte[] {
+ (byte) (data >>> 24),
+ (byte) (data >>> 16),
+ (byte) (data >>> 8),
+ data.byteValue()
+ };
+ }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/JsonDeserializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/JsonDeserializer.java
new file mode 100644
index 00000000..dd1930c7
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/JsonDeserializer.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import com.alibaba.fastjson.JSON;
+import io.openmessaging.connector.api.errors.ConnectException;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+
+/**
+ * json deserializer
+ */
+public class JsonDeserializer implements Deserializer<Object> {
+
+ @Override
+ public Object deserialize(String topic, byte[] bytes) {
+ if (Objects.isNull(bytes)) {
+ return null;
+ }
+ Object data;
+ try {
+ String json = new String(bytes, StandardCharsets.UTF_8);
+ data = JSON.parse(json);
+ } catch (Exception e) {
+ throw new ConnectException(e);
+ }
+ return data;
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/JsonSerializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/JsonSerializer.java
new file mode 100644
index 00000000..ca794a8e
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/JsonSerializer.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson.serializer.SerializerFeature;
+import io.openmessaging.connector.api.errors.ConnectException;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+
+/**
+ * json serializer
+ */
+public class JsonSerializer implements Serializer<Object> {
+
+
+ /**
+ * Convert {@code data} into a byte array.
+ *
+ * @param topic topic associated with data
+ * @param data typed data
+ * @return serialized bytes
+ */
+ @Override
+ public byte[] serialize(String topic, Object data) {
+ if (Objects.isNull(data)) {
+ return null;
+ }
+ try {
+ return JSON.toJSONString(data, SerializerFeature.DisableCircularReferenceDetect, SerializerFeature.WriteMapNullValue).getBytes(StandardCharsets.UTF_8);
+ } catch (Exception e) {
+ throw new ConnectException("Error serializing JSON message", e);
+ }
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/LongDeserializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/LongDeserializer.java
new file mode 100644
index 00000000..5022b454
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/LongDeserializer.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import org.apache.commons.lang3.SerializationException;
+
+public class LongDeserializer implements Deserializer<Long> {
+ @Override
+ public Long deserialize(String topic, byte[] data) {
+ if (data == null) {
+ return null;
+ }
+ if (data.length != 8) {
+ throw new SerializationException("Size of data received by LongDeserializer is not 8");
+ }
+
+ long value = 0;
+ for (byte b : data) {
+ value <<= 8;
+ value |= b & 0xFF;
+ }
+ return value;
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/LongSerializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/LongSerializer.java
new file mode 100644
index 00000000..150e5bdc
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/LongSerializer.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+public class LongSerializer implements Serializer<Long> {
+ @Override
+ public byte[] serialize(String topic, Long data) {
+ if (data == null) {
+ return null;
+ }
+
+ return new byte[] {
+ (byte) (data >>> 56),
+ (byte) (data >>> 48),
+ (byte) (data >>> 40),
+ (byte) (data >>> 32),
+ (byte) (data >>> 24),
+ (byte) (data >>> 16),
+ (byte) (data >>> 8),
+ data.byteValue()
+ };
+ }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/Serializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/Serializer.java
new file mode 100644
index 00000000..c96145e4
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/Serializer.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import io.openmessaging.KeyValue;
+
+import java.io.Closeable;
+import java.util.Map;
+
+/**
+ * An interface for converting objects to bytes.
+ */
+public interface Serializer<T> extends Closeable {
+
+ /**
+ * Configure this class.
+ * @param configs configs in key/value pairs
+ */
+ default void configure(Map<String, ?> configs) {
+ // intentionally left blank
+ }
+
+ /**
+ * Convert {@code data} into a byte array.
+ *
+ * @param topic topic associated with data
+ * @param data typed data
+ * @return serialized bytes
+ */
+ byte[] serialize(String topic, T data);
+
+ /**
+ * Convert data into a byte array.
+ * @return serialized bytes
+ */
+ default byte[] serialize(String topic, KeyValue extensions, T data) {
+ return serialize(topic, data);
+ }
+
+ /**
+ * Close this serializer.
+ */
+ @Override
+ default void close() {
+ // intentionally left blank
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ShortDeserializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ShortDeserializer.java
new file mode 100644
index 00000000..f5d4fd75
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ShortDeserializer.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import org.apache.commons.lang3.SerializationException;
+
+public class ShortDeserializer implements Deserializer<Short> {
+
+ @Override
+ public Short deserialize(String topic, byte[] data) {
+ if (data == null) {
+ return null;
+ }
+ if (data.length != 2) {
+ throw new SerializationException("Size of data received by ShortDeserializer is not 2");
+ }
+
+ short value = 0;
+ for (byte b : data) {
+ value <<= 8;
+ value |= b & 0xFF;
+ }
+ return value;
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ShortSerializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ShortSerializer.java
new file mode 100644
index 00000000..1e40b38f
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/ShortSerializer.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+public class ShortSerializer implements Serializer<Short> {
+ @Override
+ public byte[] serialize(String topic, Short data) {
+ if (data == null) {
+ return null;
+ }
+
+ return new byte[] {
+ (byte) (data >>> 8),
+ data.byteValue()
+ };
+ }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/StringDeserializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/StringDeserializer.java
new file mode 100644
index 00000000..d7a42e6b
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/StringDeserializer.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import java.io.UnsupportedEncodingException;
+import java.util.Map;
+
+/**
+ * String encoding defaults to UTF8 and can be customized by setting the property key.deserializer.encoding,
+ * value.deserializer.encoding or deserializer.encoding. The first two take precedence over the last.
+ */
+public class StringDeserializer implements Deserializer<String> {
+ private String encoding = "UTF8";
+
+ @Override
+ public void configure(Map<String, ?> configs) {
+ Object encodingValue = configs.get("deserializer.encoding");
+ if (encodingValue instanceof String) {
+ encoding = (String) encodingValue;
+ }
+ }
+
+ @Override
+ public String deserialize(String topic, byte[] data) {
+ try {
+ if (data == null) {
+ return null;
+ } else {
+ return new String(data, encoding);
+ }
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException("Error when deserializing byte[] to string due to unsupported encoding " + encoding);
+ }
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/StringSerializer.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/StringSerializer.java
new file mode 100644
index 00000000..f9b569a6
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/serialization/StringSerializer.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.serialization;
+
+import java.io.UnsupportedEncodingException;
+import java.util.Map;
+
+/**
+ * string serializer
+ */
+public class StringSerializer implements Serializer<String> {
+ private String encoding = "UTF8";
+
+ @Override
+ public void configure(Map<String, ?> configs) {
+ String propertyName = "serializer.encoding";
+ if (configs.containsKey(propertyName)) {
+ Object encodingValue = configs.get(propertyName);
+ if (encodingValue instanceof String) {
+ encoding = (String) encodingValue;
+ }
+ }
+ }
+
+ @Override
+ public byte[] serialize(String topic, String data) {
+ try {
+ if (data == null) {
+ return null;
+ } else {
+ return data.getBytes(encoding);
+ }
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException("Error when serializing string to byte[] due to unsupported encoding " + encoding);
+ }
+ }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/BufferedRecords.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/BufferedRecords.java
new file mode 100644
index 00000000..3058a894
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/BufferedRecords.java
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.sink;
+
+import io.openmessaging.connector.api.data.ConnectRecord;
+import io.openmessaging.connector.api.data.Schema;
+import io.openmessaging.connector.api.errors.ConnectException;
+import org.apache.rocketmq.connect.doris.exception.DorisException;
+import org.apache.rocketmq.connect.doris.schema.table.TableId;
+import org.apache.rocketmq.connect.doris.connector.DorisSinkConfig;
+import org.apache.rocketmq.connect.doris.schema.db.DbStructure;
+import org.apache.rocketmq.connect.doris.sink.metadata.FieldsMetadata;
+import org.apache.rocketmq.connect.doris.sink.metadata.SchemaPair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import static java.util.Objects.isNull;
+import static java.util.Objects.nonNull;
+
+/**
+ * buffered records
+ */
+public class BufferedRecords {
+ private static final Logger log = LoggerFactory.getLogger(BufferedRecords.class);
+
+ private final TableId tableId;
+ private final DorisSinkConfig config;
+ private final DbStructure dbStructure;
+ private List<ConnectRecord> records = new ArrayList<>();
+ private Schema keySchema;
+ private Schema schema;
+ private FieldsMetadata fieldsMetadata;
+ private RecordValidator recordValidator;
+ private List<ConnectRecord> updatePreparedRecords = new ArrayList<>();
+ private List<ConnectRecord> deletePreparedRecords = new ArrayList<>();
+ private boolean deletesInBatch = false;
+ private DorisStreamLoader loader;
+
+ public BufferedRecords(
+ DorisSinkConfig config,
+ TableId tableId,
+ DbStructure dbStructure
+ ) {
+ this.tableId = tableId;
+ this.config = config;
+ this.dbStructure = dbStructure;
+ this.recordValidator = RecordValidator.create(config);
+ this.loader = DorisStreamLoader.create(config);
+ }
+
+ /**
+ * add record
+ *
+ * @param record
+ * @return
+ * @throws SQLException
+ */
+ public List<ConnectRecord> add(ConnectRecord record) throws SQLException {
+ recordValidator.validate(record);
+ final List<ConnectRecord> flushed = new ArrayList<>();
+ boolean schemaChanged = false;
+ if (!Objects.equals(keySchema, record.getKeySchema())) {
+ keySchema = record.getKeySchema();
+ schemaChanged = true;
+ }
+ if (isNull(record.getSchema())) {
+ // For deletes, value and optionally value schema come in as null.
+ // We don't want to treat this as a schema change if key schemas is the same
+ // otherwise we flush unnecessarily.
+ if (config.isDeleteEnabled()) {
+ deletesInBatch = true;
+ }
+ } else if (Objects.equals(schema, record.getSchema())) {
+ if (config.isDeleteEnabled() && deletesInBatch) {
+ // flush so an insert after a delete of same record isn't lost
+ flushed.addAll(flush());
+ }
+ } else {
+ // value schema is not null and has changed. This is a real schema change.
+ schema = record.getSchema();
+ schemaChanged = true;
+ }
+
+ if (schemaChanged) {
+ // Each batch needs to have the same schemas, so get the buffered records out
+ flushed.addAll(flush());
+ // re-initialize everything that depends on the record schema
+ final SchemaPair schemaPair = new SchemaPair(
+ record.getKeySchema(),
+ record.getSchema(),
+ record.getExtensions()
+ );
+ // extract field
+ fieldsMetadata = FieldsMetadata.extract(
+ tableId.tableName(),
+ config.pkMode,
+ config.getPkFields(),
+ config.getFieldsWhitelist(),
+ schemaPair
+ );
+ }
+
+ // set deletesInBatch if schema value is not null
+ if (isNull(record.getData()) && config.isDeleteEnabled()) {
+ deletesInBatch = true;
+ }
+
+ records.add(record);
+ if (records.size() >= config.getBatchSize()) {
+ flushed.addAll(flush());
+ }
+ return flushed;
+ }
+
+ public List<ConnectRecord> flush() throws SQLException {
+ if (records.isEmpty()) {
+ log.debug("Records is empty");
+ return new ArrayList<>();
+ }
+ log.debug("Flushing {} buffered records", records.size());
+ for (ConnectRecord record : records) {
+ if (isNull(record.getData())) {
+ deletePreparedRecords.add(record);
+ } else {
+ updatePreparedRecords.add(record);
+ }
+ }
+ Optional<Long> totalUpdateCount = executeUpdates();
+ Optional<Long> totalDeleteCount = executeDeletes();
+ final long expectedCount = updateRecordCount();
+ log.trace("{} records:{} resulting in totalUpdateCount:{} totalDeleteCount:{}",
+ config.getInsertMode(),
+ records.size(),
+ totalUpdateCount,
+ totalDeleteCount
+ );
+ if (totalUpdateCount.filter(total -> total != expectedCount).isPresent()
+ && config.getInsertMode() == DorisSinkConfig.InsertMode.INSERT) {
+ throw new ConnectException(
+ String.format(
+ "Update count (%d) did not sum up to total number of records inserted (%d)",
+ totalUpdateCount.get(),
+ expectedCount
+ )
+ );
+ }
+ if (!totalUpdateCount.isPresent()) {
+ log.info(
+ "{} records:{} , but no count of the number of rows it affected is available",
+ config.getInsertMode(),
+ records.size()
+ );
+ }
+
+ final List<ConnectRecord> flushedRecords = records;
+ records = new ArrayList<>();
+ return flushedRecords;
+ }
+
+ /**
+ * @return an optional count of all updated rows or an empty optional if no info is available
+ */
+ private Optional<Long> executeUpdates() throws DorisException {
+ Optional<Long> count = Optional.empty();
+ if (updatePreparedRecords.isEmpty()) {
+ return count;
+ }
+ for (ConnectRecord record : updatePreparedRecords) {
+ String jsonData = DorisDialect.convertToUpdateJsonString(record);
+ try {
+ log.info("[executeUpdates]" + jsonData);
+ loader.loadJson(jsonData, record.getSchema().getName());
+ } catch (DorisException e) {
+ log.error("executeUpdates failed");
+ throw e;
+ } catch (Exception e) {
+ throw new DorisException("doris error");
+ }
+ count = count.isPresent()
+ ? count.map(total -> total + 1)
+ : Optional.of(1L);
+ }
+ return count;
+ }
+
+ private Optional<Long> executeDeletes() throws SQLException {
+ Optional<Long> totalDeleteCount = Optional.empty();
+ if (deletePreparedRecords.isEmpty()) {
+ return totalDeleteCount;
+ }
+ for (ConnectRecord record : updatePreparedRecords) {
+ String jsonData = DorisDialect.convertToDeleteJsonString(record);
+ try {
+ log.info("[executeDelete]" + jsonData);
+ loader.loadJson(jsonData, record.getSchema().getName());
+ } catch (DorisException e) {
+ log.error("executeDelete failed");
+ throw e;
+ } catch (Exception e) {
+ throw new DorisException("doris error");
+ }
+ totalDeleteCount = totalDeleteCount.isPresent()
+ ? totalDeleteCount.map(total -> total + 1)
+ : Optional.of(1L);
+ }
+ return totalDeleteCount;
+ }
+
+ private long updateRecordCount() {
+ return records
+ .stream()
+ // ignore deletes
+ .filter(record -> nonNull(record.getData()) || !config.isDeleteEnabled())
+ .count();
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/DorisDialect.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/DorisDialect.java
new file mode 100644
index 00000000..6d17bab4
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/DorisDialect.java
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.connect.doris.sink;
+
+import io.openmessaging.connector.api.data.ConnectRecord;
+import io.openmessaging.connector.api.data.Field;
+import io.openmessaging.connector.api.data.Struct;
+import org.apache.rocketmq.connect.doris.exception.TableAlterOrCreateException;
+import com.alibaba.fastjson.JSON;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+
+public class DorisDialect {
+ public static String convertToUpdateJsonString(ConnectRecord record, boolean isFirst) {
+ try {
+ Struct struct = (Struct) record.getData();
+ Map<String, String> keyValue = new HashMap<>();
+ for (Field field: struct.getSchema().getFields()) {
+ bindValue(keyValue, field, struct.getValues()[field.getIndex()]);
+ }
+ return isFirst ? "" : "," + JSON.toJSON(keyValue).toString();
+ } catch (TableAlterOrCreateException tace) {
+ throw tace;
+ }
+ }
+
+ public static String convertToUpdateJsonString(ConnectRecord record) {
+ try {
+ Struct struct = (Struct) record.getData();
+ Map<String, String> keyValue = new HashMap<>();
+ for (Field field: struct.getSchema().getFields()) {
+ bindValue(keyValue, field, struct.getValues()[field.getIndex()]);
+ }
+ return JSON.toJSON(keyValue).toString();
+ } catch (TableAlterOrCreateException tace) {
+ throw tace;
+ }
+ }
+
+ public static String convertToDeleteJsonString(ConnectRecord record) {
+ try {
+ // it seems that doris doesn't support delete via stream load
+ return "";
+ } catch (TableAlterOrCreateException tace) {
+ throw tace;
+ }
+ }
+
+ private static void bindValue(Map<String, String> keyValue, Field field, Object value) {
+ switch (field.getSchema().getFieldType()) {
+ case INT8:
+ case BOOLEAN:
+ case FLOAT64:
+ case INT32:
+ case INT64:
+ case FLOAT32:
+ if (value == null) {
+ keyValue.put(field.getName(), "null");
+ } else {
+ keyValue.put(field.getName(), value.toString());
+ }
+ break;
+ case STRING:
+ if (value == null) {
+ keyValue.put(field.getName(), "null");
+ } else {
+ keyValue.put(field.getName(), (String) value);
+ }
+ break;
+ case BYTES:
+ if (value == null) {
+ keyValue.put(field.getName(), "null");
+ } else {
+ final byte[] bytes;
+ if (value instanceof ByteBuffer) {
+ final ByteBuffer buffer = ((ByteBuffer) value).slice();
+ bytes = new byte[buffer.remaining()];
+ buffer.get(bytes);
+ } else if (value instanceof BigDecimal) {
+ keyValue.put(field.getName(), value.toString());
+ break;
+ } else {
+ bytes = (byte[]) value;
+ }
+ keyValue.put(field.getName(), Arrays.toString(bytes));
+ }
+ break;
+ case DATETIME:
+ if (value == null) {
+ keyValue.put(field.getName(), "null");
+ } else {
+ java.sql.Date date;
+ if (value instanceof java.util.Date) {
+ date = new java.sql.Date(((java.util.Date) value).getTime());
+ } else {
+ date = new java.sql.Date((int) value);
+ }
+ keyValue.put(
+ field.getName(), date.toString()
+ );
+ }
+ break;
+ default:
+ throw new TableAlterOrCreateException("Field type not found " + field);
+ }
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/DorisStreamLoader.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/DorisStreamLoader.java
new file mode 100644
index 00000000..26eb4262
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/DorisStreamLoader.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.connect.doris.sink;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.http.HttpHeaders;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.DefaultRedirectStrategy;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+import org.apache.rocketmq.connect.doris.connector.DorisSinkConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.UUID;
+
+
+/**
+ * This example mainly demonstrates how to use stream load to import data
+ * Including file type (CSV) and data in JSON format
+ */
+public class DorisStreamLoader {
+ private static final Logger log = LoggerFactory.getLogger(DorisStreamLoader.class);
+ // FE IP Address
+ private String host;
+ // FE port
+ private int port;
+ // db name
+ private String database;
+ // table name
+ private String table;
+ //user name
+ private String user;
+ //user password
+ private String passwd;
+ //http path of stream load task submission
+ private final String loadUrlWithoutTable;
+
+ private DorisStreamLoader(String host, int port, String database, String user, String passwd) {
+ this.host = host;
+ this.port = port;
+ this.database = database;
+ this.user = user;
+ this.passwd = passwd;
+ this.loadUrlWithoutTable = String.format("http://%s:%s/api/%s", host, port, database);
+ }
+
+ public static DorisStreamLoader create(DorisSinkConfig config) {
+ return new DorisStreamLoader(config.getHost(), config.getPort(), config.getDatabase(), config.getUser(), config.getPasswd());
+ }
+
+ //Build http client builder
+ private final HttpClientBuilder httpClientBuilder = HttpClients.custom().setRedirectStrategy(new DefaultRedirectStrategy() {
+ @Override
+ protected boolean isRedirectable(String method) {
+ // If the connection target is FE, you need to deal with 307 redirect
+ return true;
+ }
+ });
+
+ private String getLoadURL(String table) {
+ return String.format("%s/%s/_stream_load", loadUrlWithoutTable, table);
+ }
+
+
+ /**
+ * JSON import
+ *
+ * @param jsonData
+ * @throws Exception
+ */
+ public void loadJson(String jsonData, String table) throws Exception {
+ try (CloseableHttpClient client = httpClientBuilder.build()) {
+ HttpPut put = new HttpPut(getLoadURL(table));
+ put.removeHeaders(HttpHeaders.CONTENT_LENGTH);
+ put.removeHeaders(HttpHeaders.TRANSFER_ENCODING);
+ put.setHeader(HttpHeaders.EXPECT, "100-continue");
+ put.setHeader(HttpHeaders.AUTHORIZATION, basicAuthHeader(user, passwd));
+
+ // You can set stream load related properties in the Header, here we set label and column_separator.
+ put.setHeader("label", UUID.randomUUID().toString());
+ put.setHeader("column_separator", ",");
+ put.setHeader("format", "json");
+
+ // Set up the import file. Here you can also use StringEntity to transfer arbitrary data.
+ StringEntity entity = new StringEntity(jsonData);
+ put.setEntity(entity);
+ log.info(put.toString());
+ try (CloseableHttpResponse response = client.execute(put)) {
+ String loadResult = "";
+ if (response.getEntity() != null) {
+ loadResult = EntityUtils.toString(response.getEntity());
+ }
+
+ final int statusCode = response.getStatusLine().getStatusCode();
+ if (statusCode != 200) {
+ throw new IOException(String.format("Stream load failed. status: %s load result: %s", statusCode, loadResult));
+ }
+ log.info("Get load result: " + loadResult);
+ }
+ }
+ }
+
+ /**
+ * Construct authentication information, the authentication method used by doris here is Basic Auth
+ *
+ * @param username
+ * @param password
+ * @return
+ */
+ private String basicAuthHeader(String username, String password) {
+ final String tobeEncode = username + ":" + password;
+ byte[] encoded = Base64.encodeBase64(tobeEncode.getBytes(StandardCharsets.UTF_8));
+ return "Basic " + new String(encoded);
+ }
+
+
+ public static void main(String[] args) throws Exception {
+ DorisStreamLoader loader = new DorisStreamLoader("47.97.179.18", 7030, "db_1", "root", "rocketmq666");
+ //json load
+ String jsonData = "{\"id\":556393582,\"number\":\"123344\",\"price\":\"23.5\",\"skuname\":\"test\",\"skudesc\":\"zhangfeng_test,test\"}";
+ loader.loadJson(jsonData, "doris_test_sink");
+
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/RecordValidator.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/RecordValidator.java
new file mode 100644
index 00000000..8cf6988d
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/RecordValidator.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.sink;
+
+import io.openmessaging.connector.api.data.ConnectRecord;
+import io.openmessaging.connector.api.data.FieldType;
+import io.openmessaging.connector.api.data.Schema;
+import io.openmessaging.connector.api.errors.ConnectException;
+import org.apache.rocketmq.connect.doris.connector.DorisSinkConfig;
+
+@FunctionalInterface
+public interface RecordValidator {
+
+ RecordValidator NO_OP = (record) -> {
+ };
+
+ void validate(ConnectRecord record);
+
+ default RecordValidator and(RecordValidator other) {
+ if (other == null || other == NO_OP || other == this) {
+ return this;
+ }
+ if (this == NO_OP) {
+ return other;
+ }
+ RecordValidator thisValidator = this;
+ return (record) -> {
+ thisValidator.validate(record);
+ other.validate(record);
+ };
+ }
+
+ static RecordValidator create(DorisSinkConfig config) {
+ RecordValidator requiresKey = requiresKey(config);
+ RecordValidator requiresValue = requiresValue(config);
+
+ RecordValidator keyValidator = NO_OP;
+ RecordValidator valueValidator = NO_OP;
+ switch (config.pkMode) {
+ case RECORD_KEY:
+ keyValidator = keyValidator.and(requiresKey);
+ break;
+ case RECORD_VALUE:
+ case NONE:
+ valueValidator = valueValidator.and(requiresValue);
+ break;
+ default:
+ // no primary key is required
+ break;
+ }
+
+ if (config.isDeleteEnabled()) {
+ // When delete is enabled, we need a key
+ keyValidator = keyValidator.and(requiresKey);
+ } else {
+ // When delete is disabled, we need non-tombstone values
+ valueValidator = valueValidator.and(requiresValue);
+ }
+
+ // Compose the validator that may or may be NO_OP
+ return keyValidator.and(valueValidator);
+ }
+
+ static RecordValidator requiresValue(DorisSinkConfig config) {
+ return record -> {
+ Schema valueSchema = record.getSchema();
+ if (record.getData() != null
+ && valueSchema != null
+ && valueSchema.getFieldType() == FieldType.STRUCT) {
+ return;
+ }
+ throw new ConnectException(record.toString());
+ };
+ }
+
+ static RecordValidator requiresKey(DorisSinkConfig config) {
+ return record -> {
+ Schema keySchema = record.getKeySchema();
+ if (record.getKey() != null
+ && keySchema != null
+ && (keySchema.getFieldType() == FieldType.STRUCT || keySchema.getFieldType().isPrimitive())) {
+ return;
+ }
+ throw new ConnectException(record.toString());
+ };
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/Updater.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/Updater.java
new file mode 100644
index 00000000..7c3515bb
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/Updater.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.sink;
+
+import io.openmessaging.connector.api.data.ConnectRecord;
+import org.apache.rocketmq.connect.doris.connector.DorisSinkConfig;
+import org.apache.rocketmq.connect.doris.exception.TableAlterOrCreateException;
+import org.apache.rocketmq.connect.doris.schema.db.DbStructure;
+import org.apache.rocketmq.connect.doris.schema.table.TableId;
+import org.apache.rocketmq.connect.doris.util.TableUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.sql.SQLException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * jdbc db updater
+ */
+public class Updater {
+
+ private static final Logger log = LoggerFactory.getLogger(Updater.class);
+ private final DorisSinkConfig config;
+ private final DbStructure dbStructure;
+
+ public Updater(final DorisSinkConfig config) {
+ this.config = config;
+ this.dbStructure = null;
+ }
+
+ public void write(final Collection<ConnectRecord> records)
+ throws SQLException, TableAlterOrCreateException {
+ try {
+ final Map<TableId, BufferedRecords> bufferByTable = new HashMap<>();
+ for (ConnectRecord record : records) {
+ // destination table
+ final TableId tableId = TableUtil.destinationTable(record);
+ if (!config.filterWhiteTable(tableId)) {
+ continue;
+ }
+ BufferedRecords buffer = bufferByTable.get(tableId);
+ if (buffer == null) {
+ buffer = new BufferedRecords(config, tableId, dbStructure);
+ bufferByTable.put(tableId, buffer);
+ }
+ buffer.add(record);
+ }
+ for (Map.Entry<TableId, BufferedRecords> entry : bufferByTable.entrySet()) {
+ TableId tableId = entry.getKey();
+ BufferedRecords buffer = entry.getValue();
+ log.debug("Flushing records in JDBC Writer for table ID: {}", tableId);
+ buffer.flush();
+ }
+ } catch (SQLException | TableAlterOrCreateException e) {
+ log.error(e.toString());
+ }
+ }
+}
+
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/metadata/FieldsMetadata.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/metadata/FieldsMetadata.java
new file mode 100644
index 00000000..e82eba99
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/metadata/FieldsMetadata.java
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.sink.metadata;
+
+import io.openmessaging.KeyValue;
+import io.openmessaging.connector.api.data.Field;
+import io.openmessaging.connector.api.data.FieldType;
+import io.openmessaging.connector.api.data.Schema;
+import io.openmessaging.connector.api.errors.ConnectException;
+import org.apache.rocketmq.connect.doris.connector.DorisSinkConfig;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+/**
+ * fields metadata
+ */
+public class FieldsMetadata {
+
+ public final Set<String> keyFieldNames;
+ public final Set<String> nonKeyFieldNames;
+ public final Map<String, SinkRecordField> allFields;
+
+ // visible for testing
+ public FieldsMetadata(
+ Set<String> keyFieldNames,
+ Set<String> nonKeyFieldNames,
+ Map<String, SinkRecordField> allFields
+ ) {
+ boolean fieldCountsMatch = (keyFieldNames.size() + nonKeyFieldNames.size()) == allFields.size();
+ boolean allFieldsContained = allFields.keySet().containsAll(keyFieldNames)
+ && allFields.keySet().containsAll(nonKeyFieldNames);
+ if (!fieldCountsMatch || !allFieldsContained) {
+ throw new IllegalArgumentException(String.format(
+ "Validation fail -- keyFieldNames:%s nonKeyFieldNames:%s allFields:%s",
+ keyFieldNames, nonKeyFieldNames, allFields
+ ));
+ }
+ this.keyFieldNames = keyFieldNames;
+ this.nonKeyFieldNames = nonKeyFieldNames;
+ this.allFields = allFields;
+ }
+
+ public static FieldsMetadata extract(
+ final String tableName,
+ final DorisSinkConfig.PrimaryKeyMode pkMode,
+ final List<String> configuredPkFields,
+ final Set<String> fieldsWhitelist,
+ final SchemaPair schemaPair
+ ) {
+ return extract(
+ tableName,
+ pkMode,
+ configuredPkFields,
+ fieldsWhitelist,
+ schemaPair.keySchema,
+ schemaPair.schema,
+ schemaPair.extensions
+ );
+ }
+
+ /**
+ * extract metadata info
+ *
+ * @param tableName
+ * @param pkMode
+ * @param configuredPkFields
+ * @param fieldsWhitelist
+ * @param schema
+ * @param headers
+ * @return
+ */
+ public static FieldsMetadata extract(
+ final String tableName,
+ final DorisSinkConfig.PrimaryKeyMode pkMode,
+ final List<String> configuredPkFields,
+ final Set<String> fieldsWhitelist,
+ final Schema keySchema,
+ final Schema schema,
+ final KeyValue headers
+ ) {
+ if (schema != null && schema.getFieldType() != FieldType.STRUCT) {
+ throw new ConnectException("Value schema must be of type Struct");
+ }
+ final Map<String, SinkRecordField> allFields = new HashMap<>();
+ final Set<String> keyFieldNames = new LinkedHashSet<>();
+ switch (pkMode) {
+ case NONE:
+ break;
+ case RECORD_KEY:
+ extractRecordKeyPk(tableName, configuredPkFields, keySchema, allFields, keyFieldNames);
+ break;
+ case RECORD_VALUE:
+ extractRecordValuePk(tableName, configuredPkFields, schema, headers, allFields, keyFieldNames);
+ break;
+ default:
+ throw new ConnectException("Unknown primary key mode: " + pkMode);
+ }
+ final Set<String> nonKeyFieldNames = new LinkedHashSet<>();
+ if (schema != null) {
+ for (Field field : schema.getFields()) {
+ if (keyFieldNames.contains(field.getName())) {
+ continue;
+ }
+ if (!fieldsWhitelist.isEmpty() && !fieldsWhitelist.contains(field.getName())) {
+ continue;
+ }
+ nonKeyFieldNames.add(field.getName());
+ final Schema fieldSchema = field.getSchema();
+ allFields.put(field.getName(), new SinkRecordField(fieldSchema, field.getName(), false));
+ }
+ }
+
+ if (allFields.isEmpty()) {
+ throw new ConnectException(
+ "No fields found using key and value schemas for table: " + tableName
+ );
+ }
+
+ final Map<String, SinkRecordField> allFieldsOrdered = new LinkedHashMap<>();
+
+ if (schema != null) {
+ for (Field field : schema.getFields()) {
+ String fieldName = field.getName();
+ if (allFields.containsKey(fieldName)) {
+ allFieldsOrdered.put(fieldName, allFields.get(fieldName));
+ }
+ }
+ }
+
+ if (allFieldsOrdered.size() < allFields.size()) {
+ ArrayList<String> fieldKeys = new ArrayList<>(allFields.keySet());
+ Collections.sort(fieldKeys);
+ for (String fieldName : fieldKeys) {
+ if (!allFieldsOrdered.containsKey(fieldName)) {
+ allFieldsOrdered.put(fieldName, allFields.get(fieldName));
+ }
+ }
+ }
+
+ return new FieldsMetadata(keyFieldNames, nonKeyFieldNames, allFieldsOrdered);
+ }
+
+ private static void extractRecordKeyPk(
+ final String tableName,
+ final List<String> configuredPkFields,
+ final Schema keySchema,
+ final Map<String, SinkRecordField> allFields,
+ final Set<String> keyFieldNames
+ ) {
+ if (keySchema == null) {
+ throw new ConnectException(String.format(
+ "PK mode for table '%s' is %s, but record key schema is missing",
+ tableName,
+ DorisSinkConfig.PrimaryKeyMode.RECORD_KEY
+ ));
+ }
+ final FieldType keySchemaType = keySchema.getFieldType();
+ switch (keySchemaType) {
+ case STRUCT:
+ if (configuredPkFields.isEmpty()) {
+ keySchema.getFields().forEach(keyField -> {
+ keyFieldNames.add(keyField.getName());
+ });
+ } else {
+ for (String fieldName : configuredPkFields) {
+ final Field keyField = keySchema.getField(fieldName);
+ if (keyField == null) {
+ throw new ConnectException(String.format(
+ "PK mode for table '%s' is %s with configured PK fields %s, but record key "
+ + "schema does not contain field: %s",
+ tableName, DorisSinkConfig.PrimaryKeyMode.RECORD_KEY, configuredPkFields, fieldName
+ ));
+ }
+ }
+ keyFieldNames.addAll(configuredPkFields);
+ }
+ for (String fieldName : keyFieldNames) {
+ final Schema fieldSchema = keySchema.getField(fieldName).getSchema();
+ allFields.put(fieldName, new SinkRecordField(fieldSchema, fieldName, true));
+ }
+ break;
+ default:
+ // todo
+ if (keySchemaType.isPrimitive()) {
+ if (configuredPkFields.size() != 1) {
+ throw new ConnectException(String.format(
+ "Need exactly one PK column defined since the key schema for records is a "
+ + "primitive type, defined columns are: %s",
+ configuredPkFields
+ ));
+ }
+ final String fieldName = configuredPkFields.get(0);
+ keyFieldNames.add(fieldName);
+ allFields.put(fieldName, new SinkRecordField(keySchema, fieldName, true));
+ } else {
+ throw new ConnectException(
+ "Key schema must be primitive type or Struct, but is of type: " + keySchemaType
+ );
+ }
+ }
+ }
+
+ /**
+ * record value
+ *
+ * @param tableName
+ * @param configuredPkFields
+ * @param valueSchema
+ * @param headers
+ * @param allFields
+ * @param keyFieldNames
+ */
+ private static void extractRecordValuePk(
+ final String tableName,
+ final List<String> configuredPkFields,
+ final Schema valueSchema,
+ final KeyValue headers,
+ final Map<String, SinkRecordField> allFields,
+ final Set<String> keyFieldNames
+ ) {
+ if (valueSchema == null) {
+ throw new ConnectException(String.format(
+ "PK mode for table '%s' is %s, but record value schema is missing",
+ tableName,
+ DorisSinkConfig.PrimaryKeyMode.RECORD_VALUE)
+ );
+ }
+ List<String> pkFields = new ArrayList<>(configuredPkFields);
+ if (pkFields.isEmpty()) {
+ for (Field keyField : valueSchema.getFields()) {
+ keyFieldNames.add(keyField.getName());
+ }
+ } else {
+ for (Field keyField : valueSchema.getFields()) {
+ keyFieldNames.add(keyField.getName());
+ }
+ for (String fieldName : pkFields) {
+ if (!keyFieldNames.contains(fieldName)) {
+ throw new ConnectException(String.format(
+ "PK mode for table '%s' is %s with configured PK fields %s, but record value "
+ + "schema does not contain field: %s",
+ tableName,
+ DorisSinkConfig.PrimaryKeyMode.RECORD_VALUE,
+ pkFields,
+ fieldName
+ ));
+ }
+ }
+ keyFieldNames.addAll(pkFields);
+ }
+ for (String fieldName : keyFieldNames) {
+ final Schema fieldSchema = valueSchema.getField(fieldName).getSchema();
+ allFields.put(fieldName, new SinkRecordField(fieldSchema, fieldName, true));
+ }
+
+ }
+
+ public static boolean isPrimitive(FieldType type) {
+ switch (type) {
+ case INT8:
+ case INT32:
+ case INT64:
+ case FLOAT32:
+ case FLOAT64:
+ case BOOLEAN:
+ case STRING:
+ case BYTES:
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ @Override
+ public String toString() {
+ return "FieldsMetadata{"
+ + "keyFieldNames=" + keyFieldNames
+ + ", nonKeyFieldNames=" + nonKeyFieldNames
+ + ", allFields=" + allFields
+ + '}';
+ }
+
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/metadata/SchemaPair.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/metadata/SchemaPair.java
new file mode 100644
index 00000000..3b5a5f8a
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/metadata/SchemaPair.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.sink.metadata;
+
+import io.openmessaging.KeyValue;
+import io.openmessaging.connector.api.data.Schema;
+import java.util.Objects;
+
+/**
+ * schema pair
+ */
+public class SchemaPair {
+ public final Schema keySchema;
+ public final Schema schema;
+ public final KeyValue extensions;
+
+ public SchemaPair(Schema keySchema, Schema schema) {
+ this.keySchema = keySchema;
+ this.schema = schema;
+ this.extensions = null;
+ }
+
+ public SchemaPair(Schema keySchema, Schema schema, KeyValue extensions) {
+ this.keySchema = keySchema;
+ this.schema = schema;
+ this.extensions = extensions;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ SchemaPair that = (SchemaPair) o;
+ return Objects.equals(schema, that.schema);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(schema);
+ }
+
+ @Override
+ public String toString() {
+ return String.format("<SchemaPair: %s>", schema);
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/metadata/SinkRecordField.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/metadata/SinkRecordField.java
new file mode 100644
index 00000000..de9f082d
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/sink/metadata/SinkRecordField.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.sink.metadata;
+
+import io.openmessaging.connector.api.data.FieldType;
+import io.openmessaging.connector.api.data.Schema;
+
+public class SinkRecordField {
+
+ private final Schema schema;
+ private final String name;
+ private final boolean isPrimaryKey;
+
+ public SinkRecordField(Schema schema, String name, boolean isPrimaryKey) {
+ this.schema = schema;
+ this.name = name;
+ this.isPrimaryKey = isPrimaryKey;
+ }
+
+ public Schema schema() {
+ return schema;
+ }
+
+ public String schemaName() {
+ return schema.getName();
+ }
+
+ public FieldType schemaType() {
+ return schema.getFieldType();
+ }
+
+ public String name() {
+ return name;
+ }
+
+ public boolean isOptional() {
+ return !isPrimaryKey;
+ }
+
+ public Object defaultValue() {
+ return null;
+ }
+
+ public boolean isPrimaryKey() {
+ return isPrimaryKey;
+ }
+
+
+ public Object getDefaultValue(FieldType type) {
+ switch (type) {
+ case BOOLEAN:
+ case INT8:
+ return (byte) 0;
+ case INT32:
+ return 0;
+ case INT64:
+ return 0L;
+ case FLOAT32:
+ return 0.0f;
+ case FLOAT64:
+ return 0.0d;
+ default:
+ return null;
+ }
+
+ }
+
+ @Override
+ public String toString() {
+ return "SinkRecordField{"
+ + "schema=" + schema
+ + ", name='" + name + '\''
+ + ", isPrimaryKey=" + isPrimaryKey
+ + '}';
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/BytesUtil.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/BytesUtil.java
new file mode 100644
index 00000000..89294077
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/BytesUtil.java
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.util;
+
+public class BytesUtil {
+
+ private static final char[] HEX_CODE = "0123456789ABCDEF".toCharArray();
+ public static String toHex(byte[] data) {
+ StringBuilder r = new StringBuilder(data.length * 2);
+ for (byte b : data) {
+ r.append(HEX_CODE[(b >> 4) & 0xF]);
+ r.append(HEX_CODE[b & 0xF]);
+ }
+ return r.toString();
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/ConnectorGroupUtils.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/ConnectorGroupUtils.java
new file mode 100644
index 00000000..e3d07b63
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/ConnectorGroupUtils.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.util;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * connector group utils
+ */
+public class ConnectorGroupUtils {
+ public ConnectorGroupUtils() {
+ }
+
+ public static <T> List<List<T>> groupPartitions(List<T> elements, int numGroups) {
+ if (numGroups <= 0) {
+ throw new IllegalArgumentException("Number of groups must be positive." + numGroups);
+ } else {
+ List<List<T>> result = new ArrayList(numGroups);
+ int perGroup = elements.size() / numGroups;
+ int leftover = elements.size() - numGroups * perGroup;
+ int assigned = 0;
+
+ for (int group = 0; group < numGroups; ++group) {
+ int numThisGroup = group < leftover ? perGroup + 1 : perGroup;
+ List<T> groupList = new ArrayList(numThisGroup);
+
+ for (int i = 0; i < numThisGroup; ++i) {
+ groupList.add(elements.get(assigned));
+ ++assigned;
+ }
+
+ result.add(groupList);
+ }
+
+ return result;
+ }
+ }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/DateTimeUtils.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/DateTimeUtils.java
new file mode 100644
index 00000000..446aab8b
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/DateTimeUtils.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.util;
+
+import java.text.SimpleDateFormat;
+import java.util.Calendar;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.GregorianCalendar;
+import java.util.Date;
+
+public class DateTimeUtils {
+
+ private static final ThreadLocal<Map<TimeZone, Calendar>> TIMEZONE_CALENDARS =
+ ThreadLocal.withInitial(HashMap::new);
+
+ private static final ThreadLocal<Map<TimeZone, SimpleDateFormat>> TIMEZONE_DATE_FORMATS =
+ ThreadLocal.withInitial(HashMap::new);
+
+ private static final ThreadLocal<Map<TimeZone, SimpleDateFormat>> TIMEZONE_TIME_FORMATS =
+ ThreadLocal.withInitial(HashMap::new);
+
+ private static final ThreadLocal<Map<TimeZone, SimpleDateFormat>> TIMEZONE_TIMESTAMP_FORMATS =
+ ThreadLocal.withInitial(HashMap::new);
+
+ public static Calendar getTimeZoneCalendar(final TimeZone timeZone) {
+ return TIMEZONE_CALENDARS.get().computeIfAbsent(timeZone, GregorianCalendar::new);
+ }
+
+ public static String formatDate(Date date, TimeZone timeZone) {
+ return TIMEZONE_DATE_FORMATS.get().computeIfAbsent(timeZone, aTimeZone -> {
+ SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
+ sdf.setTimeZone(aTimeZone);
+ return sdf;
+ }).format(date);
+ }
+
+ public static String formatTime(Date date, TimeZone timeZone) {
+ return TIMEZONE_TIME_FORMATS.get().computeIfAbsent(timeZone, aTimeZone -> {
+ SimpleDateFormat sdf = new SimpleDateFormat("HH:mm:ss.SSS");
+ sdf.setTimeZone(aTimeZone);
+ return sdf;
+ }).format(date);
+ }
+
+ public static String formatTimestamp(Date date, TimeZone timeZone) {
+ return TIMEZONE_TIMESTAMP_FORMATS.get().computeIfAbsent(timeZone, aTimeZone -> {
+ SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
+ sdf.setTimeZone(aTimeZone);
+ return sdf;
+ }).format(date);
+ }
+
+ private DateTimeUtils() {
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/ExpressionBuilder.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/ExpressionBuilder.java
new file mode 100644
index 00000000..232087f5
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/ExpressionBuilder.java
@@ -0,0 +1,595 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.util;
+
+public class ExpressionBuilder {
+
+ /**
+ * A functional interface for anything that can be appended to an expression builder.
+ * This makes use of double-dispatch to allow implementations to customize the behavior,
+ * yet have callers not care about the differences in behavior.
+ */
+ @FunctionalInterface
+ public interface Expressable {
+
+ /**
+ * Append this object to the specified builder.
+ *
+ * @param builder the builder to use; may not be null
+ * @param useQuotes whether quotes should be used for this object
+ */
+ void appendTo(
+ ExpressionBuilder builder,
+ boolean useQuotes
+ );
+
+ /**
+ * Append this object to the specified builder.
+ *
+ * @param builder the builder to use; may not be null
+ * @param useQuotes whether quotes should be used for this object
+ */
+ default void appendTo(
+ ExpressionBuilder builder,
+ QuoteMethod useQuotes
+ ) {
+ switch (useQuotes) {
+ case ALWAYS:
+ appendTo(builder, true);
+ break;
+ case NEVER:
+ default:
+ // do nothing
+ break;
+ }
+ }
+ }
+
+ /**
+ * A functional interface for a transformation that an expression builder might use when
+ * appending one or more other objects.
+ *
+ * @param <T> the type of object to transform before appending.
+ */
+ @FunctionalInterface
+ public interface Transform<T> {
+ void apply(
+ ExpressionBuilder builder,
+ T input
+ );
+ }
+
+ /**
+ * A fluent API interface returned by the {@link ExpressionBuilder#appendList()} method that
+ * allows a caller to easily define a custom delimiter to be used between items in the list,
+ * an optional transformation that should be applied to each item in the list, and the
+ * items in the list. This is very handle when the number of items is not known a priori.
+ *
+ * @param <T> the type of object to be appended to the expression builder
+ */
+ public interface ListBuilder<T> {
+
+ /**
+ * Define the delimiter to appear between items in the list. If not specified, a comma
+ * is used as the default delimiter.
+ *
+ * @param delimiter the delimiter; may not be null
+ * @return this builder to enable methods to be chained; never null
+ */
+ ListBuilder<T> delimitedBy(String delimiter);
+
+ /**
+ * Define a {@link Transform} that should be applied to every item in the list as it is
+ * appended.
+ *
+ * @param transform the transform; may not be null
+ * @param <R> the type of item to be transformed
+ * @return this builder to enable methods to be chained; never null
+ */
+ <R> ListBuilder<R> transformedBy(Transform<R> transform);
+
+ /**
+ * Append to this list all of the items in the specified {@link Iterable}.
+ *
+ * @param objects the objects to be appended to the list
+ * @return this builder to enable methods to be chained; never null
+ */
+ ExpressionBuilder of(Iterable<? extends T> objects);
+
+ /**
+ * Append to this list all of the items in the specified {@link Iterable} objects.
+ *
+ * @param objects1 the first collection of objects to be added to the list
+ * @param objects2 a second collection of objects to be added to the list
+ * @return this builder to enable methods to be chained; never null
+ */
+ default ExpressionBuilder of(Iterable<? extends T> objects1, Iterable<? extends T> objects2) {
+ of(objects1);
+ return of(objects2);
+ }
+
+ /**
+ * Append to this list all of the items in the specified {@link Iterable} objects.
+ *
+ * @param objects1 the first collection of objects to be added to the list
+ * @param objects2 a second collection of objects to be added to the list
+ * @param objects3 a third collection of objects to be added to the list
+ * @return this builder to enable methods to be chained; never null
+ */
+ default ExpressionBuilder of(
+ Iterable<? extends T> objects1,
+ Iterable<? extends T> objects2,
+ Iterable<? extends T> objects3
+ ) {
+ of(objects1);
+ of(objects2);
+ return of(objects3);
+ }
+ }
+
+ /**
+ * Get a {@link Transform} that will surround the inputs with quotes.
+ *
+ * @return the transform; never null
+ */
+ public static Transform<String> quote() {
+ return (builder, input) -> builder.appendColumnName(input);
+ }
+
+ /**
+ * Get a {@link Transform} that will quote just the column names.
+ *
+ * @return the transform; never null
+ */
+// public static Transform<ColumnId> columnNames() {
+// return (builder, input) -> builder.appendColumnName(input.name());
+// }
+
+ /**
+ * Get a {@link Transform} that will quote just the column names and append the given string.
+ *
+ * @param appended the string to append after the quoted column names
+ * @return the transform; never null
+ */
+// public static Transform<ColumnId> columnNamesWith(final String appended) {
+// return (builder, input) -> {
+// builder.appendColumnName(input.name());
+// builder.append(appended);
+// };
+// }
+
+ /**
+ * Get a {@link Transform} that will append a placeholder rather than each of the column names.
+ *
+ * @param str the string to output instead the each column name
+ * @return the transform; never null
+ */
+// public static Transform<ColumnId> placeholderInsteadOfColumnNames(final String str) {
+// return (builder, input) -> builder.append(str);
+// }
+
+ /**
+ * Get a {@link Transform} that will append the prefix and then the quoted column name.
+ *
+ * @param prefix the string to output before the quoted column names
+ * @return the transform; never null
+ */
+// public static Transform<ColumnId> columnNamesWithPrefix(final String prefix) {
+// return (builder, input) -> {
+// builder.append(prefix);
+// builder.appendColumnName(input.name());
+// };
+// }
+
+ /**
+ * Create a new ExpressionBuilder using the default {@link IdentifierRules}.
+ *
+ * @return the expression builder
+ */
+ public static ExpressionBuilder create() {
+ return new ExpressionBuilder();
+ }
+
+ protected static final QuoteMethod DEFAULT_QUOTE_METHOD = QuoteMethod.ALWAYS;
+
+ private final IdentifierRules rules;
+ private final StringBuilder sb = new StringBuilder();
+ private QuoteMethod quoteSqlIdentifiers = DEFAULT_QUOTE_METHOD;
+
+ /**
+ * Create a new expression builder with the default {@link IdentifierRules}.
+ */
+ public ExpressionBuilder() {
+ this(null);
+ }
+
+ /**
+ * Create a new expression builder that uses the specified {@link IdentifierRules}.
+ *
+ * @param rules the rules; may be null if the default rules are to be used
+ */
+ public ExpressionBuilder(IdentifierRules rules) {
+ this.rules = rules != null ? rules : IdentifierRules.DEFAULT;
+ }
+
+ /**
+ * Set when this expression builder should quote identifiers, such as table and column names.
+ *
+ * @param method the quoting method; may be null if the default method
+ * ({@link QuoteMethod#ALWAYS always}) should be used
+ * @return this expression builder; never null
+ */
+ public ExpressionBuilder setQuoteIdentifiers(QuoteMethod method) {
+ this.quoteSqlIdentifiers = method != null ? method : DEFAULT_QUOTE_METHOD;
+ return this;
+ }
+
+ /**
+ * Return a new ExpressionBuilder that escapes quotes with the specified prefix.
+ * This builder remains unaffected.
+ *
+ * @param prefix the prefix
+ * @return the new ExpressionBuilder, or this builder if the prefix is null or empty
+ */
+ public ExpressionBuilder escapeQuotesWith(String prefix) {
+ if (prefix == null || prefix.isEmpty()) {
+ return this;
+ }
+ return new ExpressionBuilder(this.rules.escapeQuotesWith(prefix));
+ }
+
+ /**
+ * Append to this builder's expression the delimiter defined by this builder's
+ * {@link IdentifierRules}.
+ *
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendIdentifierDelimiter() {
+ sb.append(rules.identifierDelimiter());
+ return this;
+ }
+
+ /**
+ * Always append to this builder's expression the leading quote character(s) defined by this
+ * builder's {@link IdentifierRules}.
+ *
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendLeadingQuote() {
+ return appendLeadingQuote(QuoteMethod.ALWAYS);
+ }
+
+
+ protected ExpressionBuilder appendLeadingQuote(QuoteMethod method) {
+ switch (method) {
+ case ALWAYS:
+ sb.append(rules.leadingQuoteString());
+ break;
+ case NEVER:
+ default:
+ break;
+ }
+ return this;
+ }
+
+ /**
+ * Always append to this builder's expression the trailing quote character(s) defined by this
+ * builder's {@link IdentifierRules}.
+ *
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendTrailingQuote() {
+ return appendTrailingQuote(QuoteMethod.ALWAYS);
+ }
+
+ protected ExpressionBuilder appendTrailingQuote(QuoteMethod method) {
+ switch (method) {
+ case ALWAYS:
+ sb.append(rules.trailingQuoteString());
+ break;
+ case NEVER:
+ default:
+ break;
+ }
+ return this;
+ }
+
+ /**
+ * Append to this builder's expression the string quote character ({@code '}).
+ *
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendStringQuote() {
+ sb.append("'");
+ return this;
+ }
+
+ /**
+ * Append to this builder's expression a string surrounded by single quote characters ({@code '}).
+ * Use {@link #appendIdentifier(String, QuoteMethod)} for identifiers,
+ * {@link #appendColumnName(String, QuoteMethod)} for column names, or
+ * {@link #appendTableName(String, QuoteMethod)} for table names.
+ *
+ * @param name the object whose string representation is to be appended
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendStringQuoted(Object name) {
+ appendStringQuote();
+ sb.append(name);
+ appendStringQuote();
+ return this;
+ }
+
+ /**
+ * Append to this builder's expression the identifier.
+ *
+ * @param name the name to be appended
+ * @param quoted true if the name should be quoted, or false otherwise
+ * @return this builder to enable methods to be chained; never null
+ * @deprecated use {@link #appendIdentifier(String, QuoteMethod)} instead
+ */
+ @Deprecated
+ public ExpressionBuilder appendIdentifier(
+ String name,
+ boolean quoted
+ ) {
+ return appendIdentifier(name, quoted ? QuoteMethod.ALWAYS : QuoteMethod.NEVER);
+ }
+
+ /**
+ * Append to this builder's expression the identifier.
+ *
+ * @param name the name to be appended
+ * @param quoted true if the name should be quoted, or false otherwise
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendIdentifier(
+ String name,
+ QuoteMethod quoted
+ ) {
+ appendLeadingQuote(quoted);
+ sb.append(name);
+ appendTrailingQuote(quoted);
+ return this;
+ }
+
+ /**
+ * Append to this builder's expression the specified Column identifier, possibly surrounded by
+ * the leading and trailing quotes based upon {@link #setQuoteIdentifiers(QuoteMethod)}.
+ *
+ * @param name the name to be appended
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendTableName(String name) {
+ return appendTableName(name, quoteSqlIdentifiers);
+ }
+
+ /**
+ * Append to this builder's expression the specified Column identifier, possibly surrounded by
+ * the leading and trailing quotes based upon {@link #setQuoteIdentifiers(QuoteMethod)}.
+ *
+ * @param name the name to be appended
+ * @param quote the quote method to be used
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendTableName(String name, QuoteMethod quote) {
+ appendLeadingQuote(quote);
+ sb.append(name);
+ appendTrailingQuote(quote);
+ return this;
+ }
+
+ /**
+ * Append to this builder's expression the specified Column identifier, possibly surrounded by
+ * the leading and trailing quotes based upon {@link #setQuoteIdentifiers(QuoteMethod)}.
+ *
+ * @param name the name to be appended
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendColumnName(String name) {
+ return appendColumnName(name, quoteSqlIdentifiers);
+ }
+
+ /**
+ * Append to this builder's expression the specified Column identifier, possibly surrounded by
+ * the leading and trailing quotes based upon {@link #setQuoteIdentifiers(QuoteMethod)}.
+ *
+ * @param name the name to be appended
+ * @param quote whether to quote the column name; may not be null
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendColumnName(String name, QuoteMethod quote) {
+ appendLeadingQuote(quote);
+ sb.append(name);
+ appendTrailingQuote(quote);
+ return this;
+ }
+
+ /**
+ * Append to this builder's expression the specified identifier, surrounded by the leading and
+ * trailing quotes.
+ *
+ * @param name the name to be appended
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendIdentifierQuoted(String name) {
+ appendLeadingQuote();
+ sb.append(name);
+ appendTrailingQuote();
+ return this;
+ }
+
+ /**
+ * Append to this builder's expression the binary value as a hex string, prefixed and
+ * suffixed by a single quote character.
+ *
+ * @param value the value to be appended
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendBinaryLiteral(byte[] value) {
+ return append("x'").append(BytesUtil.toHex(value)).append("'");
+ }
+
+ /**
+ * Append to this builder's expression a new line.
+ *
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder appendNewLine() {
+ sb.append(System.lineSeparator());
+ return this;
+ }
+
+ /**
+ * Append to this builder's expression the specified object. If the object is {@link Expressable},
+ * then this builder delegates to the object's
+ * {@link Expressable#appendTo(ExpressionBuilder, boolean)} method. Otherwise, the string
+ * representation of the object is appended to the expression.
+ *
+ * @param obj the object to be appended
+ * @param useQuotes true if the object should be surrounded by quotes, or false otherwise
+ * @return this builder to enable methods to be chained; never null
+ * @deprecated use {@link #append(Object, QuoteMethod)} instead
+ */
+ @Deprecated
+ public ExpressionBuilder append(
+ Object obj,
+ boolean useQuotes
+ ) {
+ return append(obj, useQuotes ? QuoteMethod.ALWAYS : QuoteMethod.NEVER);
+ }
+
+ /**
+ * Append to this builder's expression the specified object. If the object is {@link Expressable},
+ * then this builder delegates to the object's
+ * {@link Expressable#appendTo(ExpressionBuilder, boolean)} method. Otherwise, the string
+ * representation of the object is appended to the expression.
+ *
+ * @param obj the object to be appended
+ * @param useQuotes true if the object should be surrounded by quotes, or false otherwise
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder append(
+ Object obj,
+ QuoteMethod useQuotes
+ ) {
+ if (obj instanceof Expressable) {
+ ((Expressable) obj).appendTo(this, useQuotes);
+ } else if (obj != null) {
+ sb.append(obj);
+ }
+ return this;
+ }
+
+ /**
+ * Append to this builder's expression the specified object surrounded by quotes. If the object
+ * is {@link Expressable}, then this builder delegates to the object's
+ * {@link Expressable#appendTo(ExpressionBuilder, boolean)} method. Otherwise, the string
+ * representation of the object is appended to the expression.
+ *
+ * @param obj the object to be appended
+ * @return this builder to enable methods to be chained; never null
+ */
+ public ExpressionBuilder append(Object obj) {
+ return append(obj, quoteSqlIdentifiers);
+ }
+
+ /**
+ * Append to this builder's expression the specified object surrounded by quotes. If the object
+ * is {@link Expressable}, then this builder delegates to the object's
+ * {@link Expressable#appendTo(ExpressionBuilder, boolean)} method. Otherwise, the string
+ * representation of the object is appended to the expression.
+ *
+ * @param obj the object to be appended
+ * @param transform the transform that should be used on the supplied object to obtain the
+ * representation that is appended to the expression; may be null
+ * @param <T> the type of object to transform before appending.
+ * @return this builder to enable methods to be chained; never null
+ */
+ public <T> ExpressionBuilder append(
+ T obj,
+ Transform<T> transform
+ ) {
+ if (transform != null) {
+ transform.apply(this, obj);
+ } else {
+ append(obj);
+ }
+ return this;
+ }
+
+ protected class BasicListBuilder<T> implements ListBuilder<T> {
+ private final String delimiter;
+ private final Transform<T> transform;
+ private boolean first = true;
+
+ BasicListBuilder() {
+ this(", ", null);
+ }
+
+ BasicListBuilder(String delimiter, Transform<T> transform) {
+ this.delimiter = delimiter;
+ this.transform = transform != null ? transform : ExpressionBuilder::append;
+ }
+
+ @Override
+ public ListBuilder<T> delimitedBy(String delimiter) {
+ return new BasicListBuilder<T>(delimiter, transform);
+ }
+
+ @Override
+ public <R> ListBuilder<R> transformedBy(Transform<R> transform) {
+ return new BasicListBuilder<>(delimiter, transform);
+ }
+
+ @Override
+ public ExpressionBuilder of(Iterable<? extends T> objects) {
+ for (T obj : objects) {
+ if (first) {
+ first = false;
+ } else {
+ append(delimiter);
+ }
+ append(obj, transform);
+ }
+ return ExpressionBuilder.this;
+ }
+ }
+
+ public ListBuilder<Object> appendList() {
+ return new BasicListBuilder<>();
+ }
+
+ public ExpressionBuilder appendMultiple(
+ String delimiter,
+ String expression,
+ int times
+ ) {
+ for (int i = 0; i < times; i++) {
+ if (i > 0) {
+ append(delimiter);
+ }
+ append(expression);
+ }
+ return this;
+ }
+
+ @Override
+ public String toString() {
+ return sb.toString();
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/IdentifierRules.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/IdentifierRules.java
new file mode 100644
index 00000000..830b4cd6
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/IdentifierRules.java
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.util;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * The rules for how identifiers are parsed and quoted.
+ */
+public class IdentifierRules {
+
+ public static final String UNSUPPORTED_QUOTE = " ";
+ private static final String DEFAULT_QUOTE = "\"";
+ private static final String DEFAULT_ID_DELIM = ".";
+
+ public static final IdentifierRules DEFAULT = new IdentifierRules(DEFAULT_ID_DELIM,
+ DEFAULT_QUOTE
+ );
+
+ private final String leadingQuoteString;
+ private final String trailingQuoteString;
+ private final String identifierDelimiter;
+
+ /**
+ * Create new identifier rules using the supplied quote string for both leading and trailing
+ * quotes, and the '{@link #DEFAULT_ID_DELIM}' character for identifier delimiters.
+ *
+ * @param quoteString the string used for leading and trailing quotes; may be null if {@link
+ * #DEFAULT_QUOTE} is to be used
+ */
+ public IdentifierRules(String quoteString) {
+ this(DEFAULT_ID_DELIM, quoteString, quoteString);
+ }
+
+ /**
+ * Create new identifier rules using the supplied parameters.
+ *
+ * @param delimiter the delimiter used within fully qualified names; may be null if {@link
+ * #DEFAULT_ID_DELIM} is to be used
+ * @param quoteString the string used for leading and trailing quotes; may be null if {@link
+ * #DEFAULT_QUOTE} is to be used
+ */
+ public IdentifierRules(
+ String delimiter,
+ String quoteString
+ ) {
+ this(delimiter, quoteString, quoteString);
+ }
+
+ /**
+ * Create new identifier rules using the supplied parameters.
+ *
+ * @param identifierDelimiter the delimiter used within fully qualified names; may be null if
+ * {@link #DEFAULT_ID_DELIM} is to be used
+ * @param leadingQuoteString the string used for leading quotes; may be null if {@link
+ * #DEFAULT_QUOTE} is to be used
+ * @param trailingQuoteString the string used for leading quotes; may be null if {@link
+ * #DEFAULT_QUOTE} is to be used
+ */
+ public IdentifierRules(
+ String identifierDelimiter,
+ String leadingQuoteString,
+ String trailingQuoteString
+ ) {
+ this.leadingQuoteString = leadingQuoteString != null ? leadingQuoteString : DEFAULT_QUOTE;
+ this.trailingQuoteString = trailingQuoteString != null ? trailingQuoteString : DEFAULT_QUOTE;
+ this.identifierDelimiter = identifierDelimiter != null ? identifierDelimiter : DEFAULT_ID_DELIM;
+ }
+
+ /**
+ * Get the delimiter that is used to delineate segments within fully-qualified identifiers.
+ *
+ * @return the identifier delimiter; never null
+ */
+ public String identifierDelimiter() {
+ return identifierDelimiter;
+ }
+
+ /**
+ * Get the string used as a leading quote.
+ *
+ * @return the leading quote string; never null
+ */
+ public String leadingQuoteString() {
+ return leadingQuoteString;
+ }
+
+ /**
+ * Get the string used as a trailing quote.
+ *
+ * @return the trailing quote string; never null
+ */
+ public String trailingQuoteString() {
+ return trailingQuoteString;
+ }
+
+ /**
+ * Get an expression builder that uses these identifier rules.
+ *
+ * @return the new expression builder; never null
+ */
+ public ExpressionBuilder expressionBuilder() {
+ return new ExpressionBuilder(this);
+ }
+
+ /**
+ * Parse the unqualified or fully qualified name into its segments.
+ *
+ * @param fqn the unqualified or fully-qualified name; may not be null
+ * @return the segments in the supplied name; never null, but possibly empty
+ */
+ public List<String> parseQualifiedIdentifier(String fqn) {
+ String orig = fqn;
+ String delim = identifierDelimiter();
+ String lead = leadingQuoteString();
+ String trail = trailingQuoteString();
+ List<String> parts = new ArrayList<>();
+ int index;
+ String segment;
+ do {
+ if (!lead.equals(UNSUPPORTED_QUOTE) && fqn.startsWith(lead)) {
+ int end = fqn.indexOf(trail, lead.length());
+ if (end < 0) {
+ throw new IllegalArgumentException(
+ "Failure parsing fully qualified identifier; missing trailing quote in " + orig);
+ }
+ segment = fqn.substring(lead.length(), end);
+ fqn = fqn.substring(end + trail.length());
+ if (fqn.startsWith(delim)) {
+ fqn = fqn.substring(delim.length());
+ if (fqn.isEmpty()) {
+ throw new IllegalArgumentException(
+ "Failure parsing fully qualified identifier; ends in delimiter " + orig);
+ }
+ }
+ } else {
+ index = fqn.indexOf(delim, 0);
+ if (index == -1) {
+ segment = fqn;
+ fqn = "";
+ } else {
+ segment = fqn.substring(0, index);
+ fqn = fqn.substring(index + delim.length());
+ if (fqn.isEmpty()) {
+ throw new IllegalArgumentException(
+ "Failure parsing fully qualified identifier; ends in delimiter " + orig);
+ }
+ }
+ }
+ parts.add(segment);
+ } while (fqn.length() > 0);
+ return parts;
+ }
+
+ /**
+ * Return a new IdentifierRules that escapes quotes with the specified prefix.
+ *
+ * @param prefix the prefix
+ * @return the new IdentifierRules, or this builder if the prefix is null or empty
+ */
+ public IdentifierRules escapeQuotesWith(String prefix) {
+ if (prefix == null || prefix.isEmpty()) {
+ return this;
+ }
+ return new IdentifierRules(
+ identifierDelimiter,
+ prefix + leadingQuoteString,
+ prefix + trailingQuoteString
+ );
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/JdbcDriverInfo.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/JdbcDriverInfo.java
new file mode 100644
index 00000000..b177ea34
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/JdbcDriverInfo.java
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.connect.doris.util;
+
+/**
+ * A summary of the version information about a JDBC driver and the database.
+ */
+public class JdbcDriverInfo {
+
+ private final int jdbcMajorVersion;
+ private final int jdbcMinorVersion;
+ private final String jdbcDriverName;
+ private final String productName;
+ private final String productVersion;
+
+ /**
+ * Create the driver information.
+ *
+ * @param jdbcMajorVersion the major version of the JDBC specification supported by the driver
+ * @param jdbcMinorVersion the minor version of the JDBC specification supported by the driver
+ * @param jdbcDriverName the name of the JDBC driver
+ * @param productName the name of the database product
+ * @param productVersion the version of the database product
+ */
+ public JdbcDriverInfo(
+ int jdbcMajorVersion,
+ int jdbcMinorVersion,
+ String jdbcDriverName,
+ String productName,
+ String productVersion
+ ) {
+ this.jdbcMajorVersion = jdbcMajorVersion;
+ this.jdbcMinorVersion = jdbcMinorVersion;
+ this.jdbcDriverName = jdbcDriverName;
+ this.productName = productName;
+ this.productVersion = productVersion;
+ }
+
+ /**
+ * Get the major version of the JDBC specification supported by the driver.
+ *
+ * @return the major version number
+ */
+ public int jdbcMajorVersion() {
+ return jdbcMajorVersion;
+ }
+
+ /**
+ * Get the minor version of the JDBC specification supported by the driver.
+ *
+ * @return the minor version number
+ */
+ public int jdbcMinorVersion() {
+ return jdbcMinorVersion;
+ }
+
+ /**
+ * Get the name of the database product.
+ *
+ * @return the name of the database product
+ */
+ public String productName() {
+ return productName;
+ }
+
+ /**
+ * Get the version of the database product.
+ *
+ * @return the version of the database product
+ */
+ public String productVersion() {
+ return productVersion;
+ }
+
+ /**
+ * Get the name of the JDBC driver.
+ *
+ * @return the name of the JDBC driver
+ */
+ public String jdbcDriverName() {
+ return jdbcDriverName;
+ }
+
+ /**
+ * Determine if the JDBC driver supports at least the specified major and minor version of the
+ * JDBC specifications. This can be used to determine whether or not to call JDBC methods.
+ *
+ * @param jdbcMajorVersion the required major version of the JDBC specification
+ * @param jdbcMinorVersion the required minor version of the JDBC specification
+ * @return true if the driver supports at least the specified version of the JDBC specification,
+ * or false if the driver supports an older version of the JDBC specification
+ */
+ public boolean jdbcVersionAtLeast(
+ int jdbcMajorVersion,
+ int jdbcMinorVersion
+ ) {
+ if (this.jdbcMajorVersion() > jdbcMajorVersion) {
+ return true;
+ }
+ if (jdbcMajorVersion == jdbcMajorVersion() && jdbcMinorVersion() >= jdbcMinorVersion) {
+ return true;
+ }
+ return false;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ if (productName() != null) {
+ sb.append(productName()).append(' ');
+ }
+ if (productVersion() != null) {
+ sb.append(productVersion()).append(' ');
+ }
+ if (jdbcDriverName() != null) {
+ sb.append(" using ").append(jdbcDriverName()).append(' ');
+ }
+ sb.append(jdbcMajorVersion()).append('.').append(jdbcMinorVersion());
+ return sb.toString();
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/NumericMapping.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/NumericMapping.java
new file mode 100644
index 00000000..97a2d850
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/NumericMapping.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.util;
+
+//import org.apache.rocketmq.connect.doris.connector.DorisSourceConfig;
+
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+
+public enum NumericMapping {
+ NONE,
+ PRECISION_ONLY,
+ BEST_FIT,
+ BEST_FIT_EAGER_DOUBLE;
+
+ private static final Map<String, NumericMapping> REVERSE = new HashMap<>(values().length);
+
+ static {
+ for (NumericMapping val : values()) {
+ REVERSE.put(val.name().toLowerCase(Locale.ROOT), val);
+ }
+ }
+
+ public static NumericMapping get(String prop) {
+ // not adding a check for null value because the recommender/validator should catch those.
+ return REVERSE.get(prop.toLowerCase(Locale.ROOT));
+ }
+
+// public static NumericMapping get(JdbcSourceConfig config) {
+// // We use 'null' as default to be able to check the old config if the new one is unset.
+// if (config.getNumericMapping() != null) {
+// return NumericMapping.valueOf(config.getNumericMapping());
+// }
+// if (config.getNumericPrecisionMapping()) {
+// return NumericMapping.PRECISION_ONLY;
+// }
+// return NumericMapping.NONE;
+// }
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/QuoteMethod.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/QuoteMethod.java
new file mode 100644
index 00000000..ace0b7d5
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/QuoteMethod.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.util;
+
+public enum QuoteMethod {
+ ALWAYS("always"),
+ NEVER("never");
+
+ public static QuoteMethod get(String name) {
+ for (QuoteMethod method : values()) {
+ if (method.toString().equalsIgnoreCase(name)) {
+ return method;
+ }
+ }
+ throw new IllegalArgumentException("No matching QuoteMethod found for '" + name + "'");
+ }
+
+ private final String name;
+
+ QuoteMethod(String name) {
+ this.name = name;
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+}
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/TableType.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/TableType.java
new file mode 100644
index 00000000..3a494464
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/TableType.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.connect.doris.util;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * table type
+ */
+public enum TableType {
+
+ TABLE("TABLE", "Table"),
+ VIEW("VIEW", "View");
+
+ private final String value;
+ private final String capitalCase;
+
+ TableType(String value, String capitalCase) {
+ this.value = value.toUpperCase();
+ this.capitalCase = capitalCase;
+ }
+
+ public String capitalized() {
+ return capitalCase;
+ }
+
+ public String jdbcName() {
+ return value;
+ }
+
+ @Override
+ public String toString() {
+ return value;
+ }
+
+ public static TableType get(String name) {
+ if (name != null) {
+ name = name.trim();
+ }
+ for (TableType method : values()) {
+ if (method.toString().equalsIgnoreCase(name)) {
+ return method;
+ }
+ }
+ throw new IllegalArgumentException("No matching QuoteMethod found for '" + name + "'");
+ }
+
+ public static EnumSet<TableType> parse(Collection<String> values) {
+ Set<TableType> types = values.stream().map(TableType::get).collect(Collectors.toSet());
+ return EnumSet.copyOf(types);
+ }
+
+ public static String asJdbcTableTypeNames(EnumSet<TableType> types, String delim) {
+ return types.stream()
+ .map(TableType::jdbcName)
+ .sorted()
+ .collect(Collectors.joining(delim));
+ }
+
+ public static String[] asJdbcTableTypeArray(EnumSet<TableType> types) {
+ return types.stream()
+ .map(TableType::jdbcName)
+ .sorted()
+ .collect(Collectors.toList())
+ .toArray(new String[types.size()]);
+ }
+
+}
\ No newline at end of file
diff --git a/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/TableUtil.java b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/TableUtil.java
new file mode 100644
index 00000000..87bf2b29
--- /dev/null
+++ b/connectors/rocketmq-connect-doris/src/main/java/org/apache/rocketmq/connect/doris/util/TableUtil.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.connect.doris.util;
+
+import io.openmessaging.connector.api.data.ConnectRecord;
+import io.openmessaging.connector.api.data.Struct;
+import org.apache.rocketmq.connect.doris.schema.table.TableId;
+
+public class TableUtil {
+
+ public static TableId parseToTableId(String fqn) {
+ return new TableId(null, null, fqn);
+ }
+ public static TableId destinationTable(ConnectRecord record) {
+ // todo table from header
+ Struct struct = (Struct) record.getData();
+ return new TableId(null, null, struct.getSchema().getName());
+ }
+}
diff --git a/connectors/rocketmq-connect-jdbc/README.md b/connectors/rocketmq-connect-jdbc/README.md
index 207da41b..f4871f4e 100644
--- a/connectors/rocketmq-connect-jdbc/README.md
+++ b/connectors/rocketmq-connect-jdbc/README.md
@@ -1,16 +1,16 @@
-# rocketmq-connect-jdbc
+# rocketmq-connect-doris
```
-注: 目前支持的数据库类型为 mysql ,openMLDB ,其它数据库的jdbc模式持续扩展中
+注: 目前支持将根据mysql ,openMLDB对应的source connect以流式的方式导入到rocketmq中的数据导入到doris
```
-## rocketmq-connect-jdbc 打包
+## rocketmq-connect-doirs 打包
```
mvn clean package -Dmaven.test.skip=true
```
-## rocketmq-connect-jdbc 启动
+## rocketmq-connect-doris 启动
* **jdbc-source-connector** 启动
@@ -29,24 +29,31 @@ POST http://${runtime-ip}:${runtime-port}/connectors/${rocketmq-jdbc-source-con
"key.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter",
"value.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter"
}
+
```
-* **jdbc-sink-connector** 启动
+* **doris-sink-connector** 启动
```
POST http://${runtime-ip}:${runtime-port}/connectors/${rocketmq-jdbc-sink-connector-name}
{
- "connector-class":"org.apache.rocketmq.connect.jdbc.connector.JdbcSinkConnector",
- "max-task":"2",
- "connect-topicname":"targetTopic",
- "connection.url":"jdbc:mysql://*****:3306/{sinkDbName}",
- "connection.user":"******",
- "connection.password":"******",
- "pk.fields":"id",
- "pk.mode":"record_value",
- "insert.mode":"UPSERT",
- "key.converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter",
- "value.converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter"
+ "connector.class":"org.apache.rocketmq.connect.doris.connector.DorisSinkConnector",
+ "max-task":"1",
+ "table.whitelist":"sink_test.doris_test_sink",
+ "connect.topicname":"doris_test_sink",
+ "connect.topicnames":"doris_test_sink",
+ "host":"xx.xx.xx.xx",
+ "port":"xxxx",
+ "user":"xx",
+ "passwd":"****",
+ "database":"database",
+ "insert.mode":"INSERT",
+ "db.timezone":"UTC",
+ "table.types":"TABLE",
+ "auto.create":"true",
+ "source-record-converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter",
+ "key.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter",
+ "value.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter"}'
}
```
@@ -62,22 +69,21 @@ http://${runtime-ip}:${runtime-port}/connectors/${rocketmq-jdbc-connector-name}/
* **jdbc-source-connector 参数说明**
-| KEY | TYPE | Must be filled | Description | Example
-|--------------------------|----|---------|------------------|-------------------------------------------------------------|
-| connection.url | String | YES | source端 jdbc连接 | jdbc:mysql://XXXXXXXXX:3306 |
-| connection.user | String | YES | source端 DB 用户名 | root |
-| connection.password | String | YES | source端 DB 密码 | root |
-| connection.attempts | String | YES | source端 DB连接重试次数 | 3 |
-| connection.backoff.ms | Long | YES | |
-| poll.interval.ms | Long | YES | 拉取间隔时间 | 3000ms |
-| batch.max.rows | Integer | NO | 每次拉取数量 | 300 |
-| mode | Integer | NO | 拉取模式 | bulk、timestamp、incrementing、timestamp+incrementing |
-| incrementing.column.name | Integer | NO | 增量字段,常用ID | id |
-| timestamp.column.name | String | YES | 时间增量字段 | modified_time |
-| table.whitelist | String | YES | 需要扫描的表 | db.table,db.table01 |
-| max.tasks | Integer | YES | 任务数量,最大不能大于表的数量 | 默认是1 |
-| value.converter | Integer | YES | value转换器 | org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter |
-| key.converter | Integer | YES | key转换器 | org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter |
+| KEY | TYPE | Must be filled | Description| Example
+|------------------------|----|---------|---------------|------------------|
+|connection.url | String | YES | source端 jdbc连接 | jdbc:mysql://XXXXXXXXX:3306|
+|connection.user | String | YES | source端 DB 用户名 | root |
+|connection.password | String | YES | source端 DB 密码 | root |
+|connection.attempts | String | YES | source端 DB连接重试次数 | 3 |
+|connection.backoff.ms | Long | YES | |
+|poll.interval.ms | Long | YES |拉取间隔时间 | 3000ms |
+|batch.max.rows | Integer | NO |每次拉取数量 | 300 |
+|mode | Integer | NO |拉取模式 | bulk、timestamp、incrementing、timestamp+incrementing |
+|incrementing.column.name | Integer | NO |增量字段,常用ID | id |
+|timestamp.column.name | String | YES |时间增量字段 | modified_time |
+|table.whitelist | String | YES |需要扫描的表 | db.table,db.table01 |
+|max-task | Integer | YES |任务数量,最大不能大于表的数量 | 2 |
+|source-record-converter | Integer | YES |data转换器 | org.apache.rocketmq.connect.doris.converter.JsonConverter |
```
注:1.source拉取的数据写入到以表名自动创建的topic中,如果需要写入特定的topic中则需要指定"connect-topicname" 参数
@@ -86,44 +92,15 @@ http://${runtime-ip}:${runtime-port}/connectors/${rocketmq-jdbc-connector-name}/
* **jdbc-sink-connector 参数说明**
-| KEY | TYPE | Must be filled | Description| Example
-|-------------------------|----|---------|---------------|------------------|
-| connection.url | String | YES | sink端 jdbc连接 | jdbc:mysql://XXXXXXXXX:3306|
-| connection.user | String | YES | sink端 DB 用户名 | root |
-| connection.password | String | YES | sink端 DB 密码 | root |
-| connection.attempts | String | NO | sink端 DB连接重试次数 | 3 |
-| connection.backoff.ms | Long | NO | |
-| connect-topicname | Long | YES |监听的topic | topic-name |
-| pk.fields | String | NO |写入侧主键配置,用于更新使用 | id |
-| pk.mode | String | NO |获取主键的模式 | none、record_value |
-| insert.mode | Integer | YES |写入模式 | UPDATE、UPSERT、INSERT |
-| max.tasks | Integer | NO |任务数量 | 2 |
-| value.converter | Integer | YES | value转换器 | org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter |
-| key.converter | Integer | YES | key转换器 | org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter |
-
-```
-注: openMLDB maven包的引入:
----------MacOS 系统下运行-------------
- <dependency>
- <groupId>com.4paradigm.openmldb</groupId>
- <artifactId>openmldb-native</artifactId>
- <version>0.5.0-macos</version>
- </dependency>
- <dependency>
- <groupId>com.4paradigm.openmldb</groupId>
- <artifactId>openmldb-jdbc</artifactId>
- <version>0.5.0</version>
- <exclusions>
- <exclusion>
- <groupId>com.4paradigm.openmldb</groupId>
- <artifactId>openmldb-native</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
----------Linux 系统下运行-------------
- <dependency>
- <groupId>com.4paradigm.openmldb</groupId>
- <artifactId>openmldb-jdbc</artifactId>
- <version>0.5.0</version>
- </dependency>
+| KEY | TYPE | Must be filled | Description| Example
+|------------------------|----|---------|---------------|------------------|
+|connection.url | String | YES | sink端 jdbc连接 | jdbc:mysql://XXXXXXXXX:3306|
+|connection.user | String | YES | sink端 DB 用户名 | root |
+|connection.password | String | YES | sink端 DB 密码 | root |
+|host | String | YES |doris host | 192.168.0.1 |
+|port | String | YES |doris http port | 8030 |
+|user | String | YES |监听的topic | root |
+|passwd | String | YES |监听的topic | passwd |
+|max-task | Integer | NO |任务数量 | 2 |
+|source-record-converter | Integer | YES |data转换器 | org.apache.rocketmq.connect.doris.converter.JsonConverter |
```