You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Alibaba-HZY (via GitHub)" <gi...@apache.org> on 2023/05/26 04:40:58 UTC
[GitHub] [incubator-paimon] Alibaba-HZY opened a new pull request, #1239: [flink] Supports KafkaSyncTableAction
Alibaba-HZY opened a new pull request, #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239
### Purpose
Supports Kafka with canal CDC format.(https://github.com/apache/incubator-paimon/issues/911)
### Tests
KafkaCanalSyncDatabaseActionITCase
KafkaCanalSyncTableActionITCase
### API and Format
no
### Documentation
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#discussion_r1206407465
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private String databaseName;
+ private String tableName;
+ private final Map<String, DataType> fields;
+ private final List<String> primaryKeys;
+ private static final int RETRY = Integer.MAX_VALUE;
Review Comment:
Yes, if we just wait a long time, I think there may be some timeout exceptions in Flink submit..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] Alibaba-HZY commented on pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "Alibaba-HZY (via GitHub)" <gi...@apache.org>.
Alibaba-HZY commented on PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#issuecomment-1567699130
> I created #1254 for sink API public.
Wait for this pr to merge and modify the kafkaSynctableAction.class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#discussion_r1206406088
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private String databaseName;
+ private String tableName;
+ private final Map<String, DataType> fields;
+ private final List<String> primaryKeys;
+ private static final int RETRY = Integer.MAX_VALUE;
+
+ public KafkaSchema(Configuration kafkaConfig, String topic) throws Exception {
+
+ fields = new LinkedHashMap<>();
+ primaryKeys = new ArrayList<>();
+ KafkaConsumer<String, String> consumer = getKafkaConsumer(kafkaConfig);
+
+ consumer.subscribe(Collections.singletonList(topic));
+
+ int retry = 0;
+ boolean success = false;
+ while (!success) {
+ ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
+ for (ConsumerRecord<String, String> record : records) {
Review Comment:
> it may be wrong
Throw exception? There is no catch?
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private String databaseName;
+ private String tableName;
+ private final Map<String, DataType> fields;
+ private final List<String> primaryKeys;
+ private static final int RETRY = Integer.MAX_VALUE;
+
+ public KafkaSchema(Configuration kafkaConfig, String topic) throws Exception {
+
+ fields = new LinkedHashMap<>();
+ primaryKeys = new ArrayList<>();
+ KafkaConsumer<String, String> consumer = getKafkaConsumer(kafkaConfig);
+
+ consumer.subscribe(Collections.singletonList(topic));
+
+ int retry = 0;
+ boolean success = false;
+ while (!success) {
+ ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
+ for (ConsumerRecord<String, String> record : records) {
Review Comment:
> it may be wrong
Throw exception? There is no catch? Any test cover?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#discussion_r1206269103
##########
docs/content/how-to/cdc-ingestion.md:
##########
@@ -157,3 +157,120 @@ Example
--table-conf changelog-producer=input \
--table-conf sink.parallelism=4
```
+
+## Kafka
+
+### Prepare Kafka Bundled Jar
+
+```
+flink-sql-connector-kafka-*.jar
+```
+### Supported Formats
+Flink provides several Kafka CDC formats :canal-json、debezium-json,ogg-json,maxwell-json.
+If a message in a Kafka topic is a change event captured from another database using the Change Data Capture (CDC) tool, then you can use the Paimon Kafka CDC. Write the INSERT, UPDATE, DELETE messages parsed into the paimon table.
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Formats</th>
+ <th class="text-left">Supported</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/canal/">Canal CDC</a></td>
+ <td>True</td>
+ </tr>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/debezium/">Debezium CDC</a></td>
+ <td>False</td>
+ </tr>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/maxwell/ >}}">Maxwell CDC</a></td>
+ <td>False</td>
+ </tr>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/ogg/">OGG CDC</a></td>
+ <td>False</td>
+ </tr>
+ </tbody>
+</table>
+
+### Synchronizing Tables
+
+By using [KafkaSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from Kafka's one topic into one Paimon table.
+
+To use this feature through `flink run`, run the following shell command.
+
+```bash
+<FLINK_HOME>/bin/flink run \
+ -c org.apache.paimon.flink.action.FlinkActions \
+ /path/to/paimon-flink-**-{{< version >}}.jar \
+ kafka-sync-table
+ --warehouse <warehouse-path> \
+ --database <database-name> \
+ --table <table-name> \
+ [--partition-keys <partition-keys>] \
+ [--primary-keys <primary-keys>] \
+ [--computed-column <'column-name=expr-name(args[, ...])'> [--computed-column ...]] \
+ [--kafka-conf <kafka-source-conf> [--kafka-conf <kafka-source-conf> ...]] \
+ [--catalog-conf <paimon-catalog-conf> [--catalog-conf <paimon-catalog-conf> ...]] \
+ [--table-conf <paimon-table-sink-conf> [--table-conf <paimon-table-sink-conf> ...]]
+```
+
+* `--warehouse` is the path to Paimon warehouse.
Review Comment:
Use table, see above.
##########
docs/content/how-to/cdc-ingestion.md:
##########
@@ -157,3 +157,120 @@ Example
--table-conf changelog-producer=input \
--table-conf sink.parallelism=4
```
+
+## Kafka
+
+### Prepare Kafka Bundled Jar
+
+```
+flink-sql-connector-kafka-*.jar
+```
+### Supported Formats
+Flink provides several Kafka CDC formats :canal-json、debezium-json,ogg-json,maxwell-json.
+If a message in a Kafka topic is a change event captured from another database using the Change Data Capture (CDC) tool, then you can use the Paimon Kafka CDC. Write the INSERT, UPDATE, DELETE messages parsed into the paimon table.
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Formats</th>
+ <th class="text-left">Supported</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/canal/">Canal CDC</a></td>
+ <td>True</td>
+ </tr>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/debezium/">Debezium CDC</a></td>
+ <td>False</td>
+ </tr>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/maxwell/ >}}">Maxwell CDC</a></td>
+ <td>False</td>
+ </tr>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/ogg/">OGG CDC</a></td>
+ <td>False</td>
+ </tr>
+ </tbody>
+</table>
+
+### Synchronizing Tables
+
+By using [KafkaSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from Kafka's one topic into one Paimon table.
+
+To use this feature through `flink run`, run the following shell command.
+
+```bash
+<FLINK_HOME>/bin/flink run \
+ -c org.apache.paimon.flink.action.FlinkActions \
+ /path/to/paimon-flink-**-{{< version >}}.jar \
+ kafka-sync-table
+ --warehouse <warehouse-path> \
+ --database <database-name> \
+ --table <table-name> \
+ [--partition-keys <partition-keys>] \
+ [--primary-keys <primary-keys>] \
+ [--computed-column <'column-name=expr-name(args[, ...])'> [--computed-column ...]] \
+ [--kafka-conf <kafka-source-conf> [--kafka-conf <kafka-source-conf> ...]] \
+ [--catalog-conf <paimon-catalog-conf> [--catalog-conf <paimon-catalog-conf> ...]] \
+ [--table-conf <paimon-table-sink-conf> [--table-conf <paimon-table-sink-conf> ...]]
+```
+
+* `--warehouse` is the path to Paimon warehouse.
+* `--database` is the database name in Paimon catalog.
+* `--table` is the Paimon table name.
+* `--partition-keys` are the partition keys for Paimon table. If there are multiple partition keys, connect them with comma, for example `dt,hh,mm`.
+* `--primary-keys` are the primary keys for Paimon table. If there are multiple primary keys, connect them with comma, for example `buyer_id,seller_id`.
+* `--kafka-conf` is the configuration for Flink Kafka sources. Each configuration should be specified in the format `key=value`. `properties.bootstrap.servers`, `topic`, `properties.group.id`, and `value.format` are required configurations, others are optional. See its [document](https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/kafka/#%e8%bf%9e%e6%8e%a5%e5%99%a8%e5%8f%82%e6%95%b0) for a complete list of configurations.
+* `--catalog-conf` is the configuration for Paimon catalog. Each configuration should be specified in the format `key=value`. See [here]({{< ref "maintenance/configurations" >}}) for a complete list of catalog configurations.
+* `--table-conf` is the configuration for Paimon table sink. Each configuration should be specified in the format `key=value`. See [here]({{< ref "maintenance/configurations" >}}) for a complete list of table configurations.
+
+If the Paimon table you specify does not exist, this action will automatically create the table. Its schema will be derived from all specified Kafka topic's tables,it gets the earliest non-DDL data parsing schema from topic. If the Paimon table already exists, its schema will be compared against the schema of all specified Kafka topic's tables.
+
+This action supports a limited number of schema changes. Currently, the framework can not drop columns, so the behaviors of `DROP` will be ignored, `RENAME` will add a new column. Currently supported schema changes includes:
+
+* Adding columns.
+
+* Altering column types. More specifically,
+
+ * altering from a string type (char, varchar, text) to another string type with longer length,
+ * altering from a binary type (binary, varbinary, blob) to another binary type with longer length,
+ * altering from an integer type (tinyint, smallint, int, bigint) to another integer type with wider range,
+ * altering from a floating-point type (float, double) to another floating-point type with wider range,
+
+ are supported.
+
+Example
+
+```bash
+<FLINK_HOME>/bin/flink run \
+ -c org.apache.paimon.flink.action.FlinkActions \
+ /path/to/paimon-flink-**-{{< version >}}.jar \
+ kafka-sync-table \
+ --warehouse hdfs:///path/to/warehouse \
+ --database test_db \
+ --table test_table \
+ --partition-keys pt \
+ --primary-keys pt,uid \
+ --computed-columns '_year=year(age)' \
+ --kafka-conf properties.bootstrap.servers=127.0.0.1:9020 \
+ --kafka-conf topic=order \
+ --kafka-conf properties.group.id=123456 \
+ --kafka-conf value.format=canal-json \
+ --catalog-conf metastore=hive \
+ --catalog-conf uri=thrift://hive-metastore:9083 \
+ --table-conf bucket=4 \
+ --table-conf changelog-producer=input \
+ --table-conf sink.parallelism=4
+```
+
+## Computed Functions
+* `--computed-column` are the definitions of computed columns. The argument field is from Kafka topic's table field name. Supported expressions are:
Review Comment:
Use table and remove above computed functions of mysql cdc.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private String databaseName;
+ private String tableName;
+ private final Map<String, DataType> fields;
+ private final List<String> primaryKeys;
+ private static final int RETRY = Integer.MAX_VALUE;
+
+ public KafkaSchema(Configuration kafkaConfig, String topic) throws Exception {
+
+ fields = new LinkedHashMap<>();
+ primaryKeys = new ArrayList<>();
+ KafkaConsumer<String, String> consumer = getKafkaConsumer(kafkaConfig);
+
+ consumer.subscribe(Collections.singletonList(topic));
+
+ int retry = 0;
+ boolean success = false;
+ while (!success) {
+ ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
+ for (ConsumerRecord<String, String> record : records) {
Review Comment:
It seems that this loop can just be one parsing?
```
if (records.size > 0) {
onsumerRecord<String, String> record = records.get(0);
.......
}
```
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private String databaseName;
+ private String tableName;
+ private final Map<String, DataType> fields;
+ private final List<String> primaryKeys;
+ private static final int RETRY = Integer.MAX_VALUE;
+
+ public KafkaSchema(Configuration kafkaConfig, String topic) throws Exception {
+
+ fields = new LinkedHashMap<>();
+ primaryKeys = new ArrayList<>();
+ KafkaConsumer<String, String> consumer = getKafkaConsumer(kafkaConfig);
+
+ consumer.subscribe(Collections.singletonList(topic));
+
+ int retry = 0;
+ boolean success = false;
+ while (!success) {
+ ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
+ for (ConsumerRecord<String, String> record : records) {
+ try {
+ String format = kafkaConfig.get(KafkaConnectorOptions.VALUE_FORMAT);
+ if ("canal-json".equals(format)) {
+ parseCanalJson(record.value());
+ } else {
+ throw new UnsupportedOperationException(
+ "This format: " + format + " is not support.");
+ }
+ success = true;
+ break;
+ } catch (JsonProcessingException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ if (!success && retry == RETRY) {
+ throw new Exception("Could not get metadata from server,topic :" + topic);
+ }
+ Thread.sleep(1000);
+ retry++;
+ }
+ }
+
+ public String tableName() {
+ return tableName;
+ }
+
+ public String databaseName() {
+ return databaseName;
+ }
+
+ public Map<String, DataType> fields() {
+ return fields;
+ }
+
+ public List<String> primaryKeys() {
+ return primaryKeys;
+ }
+
+ static KafkaConsumer<String, String> getKafkaConsumer(Configuration kafkaConfig) {
Review Comment:
private? I think lots of methods can be private.
##########
docs/content/how-to/cdc-ingestion.md:
##########
@@ -157,3 +157,120 @@ Example
--table-conf changelog-producer=input \
--table-conf sink.parallelism=4
```
+
+## Kafka
+
+### Prepare Kafka Bundled Jar
+
+```
+flink-sql-connector-kafka-*.jar
+```
+### Supported Formats
+Flink provides several Kafka CDC formats :canal-json、debezium-json,ogg-json,maxwell-json.
+If a message in a Kafka topic is a change event captured from another database using the Change Data Capture (CDC) tool, then you can use the Paimon Kafka CDC. Write the INSERT, UPDATE, DELETE messages parsed into the paimon table.
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left">Formats</th>
+ <th class="text-left">Supported</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/canal/">Canal CDC</a></td>
+ <td>True</td>
+ </tr>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/debezium/">Debezium CDC</a></td>
+ <td>False</td>
+ </tr>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/maxwell/ >}}">Maxwell CDC</a></td>
+ <td>False</td>
+ </tr>
+ <tr>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/formats/ogg/">OGG CDC</a></td>
+ <td>False</td>
+ </tr>
+ </tbody>
+</table>
+
+### Synchronizing Tables
+
+By using [KafkaSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from Kafka's one topic into one Paimon table.
+
+To use this feature through `flink run`, run the following shell command.
+
+```bash
+<FLINK_HOME>/bin/flink run \
+ -c org.apache.paimon.flink.action.FlinkActions \
+ /path/to/paimon-flink-**-{{< version >}}.jar \
Review Comment:
See latest usage for action.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private String databaseName;
+ private String tableName;
+ private final Map<String, DataType> fields;
+ private final List<String> primaryKeys;
+ private static final int RETRY = Integer.MAX_VALUE;
Review Comment:
this is static field, move this top.
`MAX_RETRY`?
How about ten instead of max_value?
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private String databaseName;
+ private String tableName;
+ private final Map<String, DataType> fields;
+ private final List<String> primaryKeys;
+ private static final int RETRY = Integer.MAX_VALUE;
+
+ public KafkaSchema(Configuration kafkaConfig, String topic) throws Exception {
+
+ fields = new LinkedHashMap<>();
+ primaryKeys = new ArrayList<>();
+ KafkaConsumer<String, String> consumer = getKafkaConsumer(kafkaConfig);
+
+ consumer.subscribe(Collections.singletonList(topic));
+
+ int retry = 0;
+ boolean success = false;
+ while (!success) {
+ ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
+ for (ConsumerRecord<String, String> record : records) {
+ try {
+ String format = kafkaConfig.get(KafkaConnectorOptions.VALUE_FORMAT);
+ if ("canal-json".equals(format)) {
+ parseCanalJson(record.value());
+ } else {
+ throw new UnsupportedOperationException(
+ "This format: " + format + " is not support.");
+ }
+ success = true;
+ break;
+ } catch (JsonProcessingException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ if (!success && retry == RETRY) {
+ throw new Exception("Could not get metadata from server,topic :" + topic);
+ }
+ Thread.sleep(1000);
+ retry++;
+ }
+ }
+
+ public String tableName() {
+ return tableName;
+ }
+
+ public String databaseName() {
+ return databaseName;
+ }
+
+ public Map<String, DataType> fields() {
+ return fields;
+ }
+
+ public List<String> primaryKeys() {
+ return primaryKeys;
+ }
+
+ static KafkaConsumer<String, String> getKafkaConsumer(Configuration kafkaConfig) {
Review Comment:
`getKafkaEarliestConsumer`?
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
Review Comment:
Should this be `static`?
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncTableAction.java:
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.flink.FlinkConnectorOptions;
+import org.apache.paimon.flink.action.Action;
+import org.apache.paimon.flink.action.cdc.ComputedColumn;
+import org.apache.paimon.flink.action.cdc.kafka.canal.CanalJsonEventParser;
+import org.apache.paimon.flink.sink.cdc.EventParser;
+import org.apache.paimon.flink.sink.cdc.FlinkCdcSyncTableSinkBuilder;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.schema.Schema;
+import org.apache.paimon.table.FileStoreTable;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.utils.MultipleParameterTool;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.kafka.source.KafkaSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static org.apache.paimon.utils.Preconditions.checkArgument;
+
+/**
+ * An {@link Action} which synchronize one kafka topic into one Paimon table.
+ *
+ * <p>This topic must be from canal-json format.
+ *
+ * <p>You should specify Kafka source topic in {@code kafkaConfig}. See <a
+ * href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/kafka/">document
+ * of flink-connectors</a> for detailed keys and values.
+ *
+ * <p>If the specified Paimon table does not exist, this action will automatically create the table.
+ * Its schema will be derived from all specified Kafka topic. If the Paimon table already exists,
+ * its schema will be compared against the schema of all specified Kafka topic.
+ *
+ * <p>This action supports a limited number of schema changes. Unsupported schema changes will be
+ * ignored. Currently supported schema changes includes:
+ *
+ * <ul>
+ * <li>Adding columns.
+ * <li>Altering column types. More specifically,
+ * <ul>
+ * <li>altering from a string type (char, varchar, text) to another string type with longer
+ * length,
+ * <li>altering from a binary type (binary, varbinary, blob) to another binary type with
+ * longer length,
+ * <li>altering from an integer type (tinyint, smallint, int, bigint) to another integer
+ * type with wider range,
+ * <li>altering from a floating-point type (float, double) to another floating-point type
+ * with wider range,
+ * </ul>
+ * are supported. Other type changes will cause exceptions.
+ * </ul>
+ */
+public class KafkaSyncTableAction implements Action {
+
+ private final Configuration kafkaConfig;
+ private final String warehouse;
+ private final String database;
+ private final String table;
+ private final List<String> partitionKeys;
+ private final List<String> primaryKeys;
+
+ private final List<String> computedColumnArgs;
+
+ Map<String, String> catalogConfig;
+ private final Map<String, String> paimonConfig;
+
+ KafkaSyncTableAction(
+ Map<String, String> kafkaConfig,
+ String warehouse,
+ String database,
+ String table,
+ List<String> partitionKeys,
+ List<String> primaryKeys,
+ Map<String, String> catalogConfig,
+ Map<String, String> paimonConfig) {
+ this(
+ kafkaConfig,
+ warehouse,
+ database,
+ table,
+ partitionKeys,
+ primaryKeys,
+ Collections.emptyList(),
+ catalogConfig,
+ paimonConfig);
+ }
+
+ public KafkaSyncTableAction(
+ Map<String, String> kafkaConfig,
+ String warehouse,
+ String database,
+ String table,
+ List<String> partitionKeys,
+ List<String> primaryKeys,
+ List<String> computedColumnArgs,
+ Map<String, String> catalogConfig,
+ Map<String, String> paimonConfig) {
+ this.kafkaConfig = Configuration.fromMap(kafkaConfig);
+ this.warehouse = warehouse;
+ this.database = database;
+ this.table = table;
+ this.partitionKeys = partitionKeys;
+ this.primaryKeys = primaryKeys;
+ this.computedColumnArgs = computedColumnArgs;
+ this.catalogConfig = catalogConfig;
+ this.paimonConfig = paimonConfig;
+ }
+
+ public void build(StreamExecutionEnvironment env) throws Exception {
+ checkArgument(
+ kafkaConfig.contains(KafkaConnectorOptions.VALUE_FORMAT),
+ KafkaConnectorOptions.VALUE_FORMAT.key() + " cannot be null ");
+ KafkaSource<String> source = KafkaActionUtils.buildKafkaSource(kafkaConfig);
+ String topic = kafkaConfig.get(KafkaConnectorOptions.TOPIC).get(0);
+ KafkaSchema kafkaSchema = new KafkaSchema(kafkaConfig, topic);
+
+ Catalog catalog =
Review Comment:
Wait for https://github.com/apache/incubator-paimon/pull/1241 to use ActionBase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi merged pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#issuecomment-1566716922
I created https://github.com/apache/incubator-paimon/pull/1254 for sink API public.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#discussion_r1206503810
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSyncTableAction.java:
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.flink.FlinkConnectorOptions;
+import org.apache.paimon.flink.action.Action;
+import org.apache.paimon.flink.action.cdc.ComputedColumn;
+import org.apache.paimon.flink.action.cdc.kafka.canal.CanalJsonEventParser;
+import org.apache.paimon.flink.sink.cdc.EventParser;
+import org.apache.paimon.flink.sink.cdc.FlinkCdcSyncTableSinkBuilder;
+import org.apache.paimon.options.CatalogOptions;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.schema.Schema;
+import org.apache.paimon.table.FileStoreTable;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.utils.MultipleParameterTool;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.kafka.source.KafkaSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+import static org.apache.paimon.utils.Preconditions.checkArgument;
+
+/**
+ * An {@link Action} which synchronize one kafka topic into one Paimon table.
+ *
+ * <p>This topic must be from canal-json format.
+ *
+ * <p>You should specify Kafka source topic in {@code kafkaConfig}. See <a
+ * href="https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/connectors/table/kafka/">document
+ * of flink-connectors</a> for detailed keys and values.
+ *
+ * <p>If the specified Paimon table does not exist, this action will automatically create the table.
+ * Its schema will be derived from all specified Kafka topic. If the Paimon table already exists,
+ * its schema will be compared against the schema of all specified Kafka topic.
+ *
+ * <p>This action supports a limited number of schema changes. Unsupported schema changes will be
+ * ignored. Currently supported schema changes includes:
+ *
+ * <ul>
+ * <li>Adding columns.
+ * <li>Altering column types. More specifically,
+ * <ul>
+ * <li>altering from a string type (char, varchar, text) to another string type with longer
+ * length,
+ * <li>altering from a binary type (binary, varbinary, blob) to another binary type with
+ * longer length,
+ * <li>altering from an integer type (tinyint, smallint, int, bigint) to another integer
+ * type with wider range,
+ * <li>altering from a floating-point type (float, double) to another floating-point type
+ * with wider range,
+ * </ul>
+ * are supported. Other type changes will cause exceptions.
+ * </ul>
+ */
+public class KafkaSyncTableAction implements Action {
+
+ private final Configuration kafkaConfig;
+ private final String warehouse;
+ private final String database;
+ private final String table;
+ private final List<String> partitionKeys;
+ private final List<String> primaryKeys;
+
+ private final List<String> computedColumnArgs;
+
+ Map<String, String> catalogConfig;
+ private final Map<String, String> paimonConfig;
+
+ KafkaSyncTableAction(
+ Map<String, String> kafkaConfig,
+ String warehouse,
+ String database,
+ String table,
+ List<String> partitionKeys,
+ List<String> primaryKeys,
+ Map<String, String> catalogConfig,
+ Map<String, String> paimonConfig) {
+ this(
+ kafkaConfig,
+ warehouse,
+ database,
+ table,
+ partitionKeys,
+ primaryKeys,
+ Collections.emptyList(),
+ catalogConfig,
+ paimonConfig);
+ }
+
+ public KafkaSyncTableAction(
+ Map<String, String> kafkaConfig,
+ String warehouse,
+ String database,
+ String table,
+ List<String> partitionKeys,
+ List<String> primaryKeys,
+ List<String> computedColumnArgs,
+ Map<String, String> catalogConfig,
+ Map<String, String> paimonConfig) {
+ this.kafkaConfig = Configuration.fromMap(kafkaConfig);
+ this.warehouse = warehouse;
+ this.database = database;
+ this.table = table;
+ this.partitionKeys = partitionKeys;
+ this.primaryKeys = primaryKeys;
+ this.computedColumnArgs = computedColumnArgs;
+ this.catalogConfig = catalogConfig;
+ this.paimonConfig = paimonConfig;
+ }
+
+ public void build(StreamExecutionEnvironment env) throws Exception {
+ checkArgument(
+ kafkaConfig.contains(KafkaConnectorOptions.VALUE_FORMAT),
+ KafkaConnectorOptions.VALUE_FORMAT.key() + " cannot be null ");
+ KafkaSource<String> source = KafkaActionUtils.buildKafkaSource(kafkaConfig);
+ String topic = kafkaConfig.get(KafkaConnectorOptions.TOPIC).get(0);
+ KafkaSchema kafkaSchema = new KafkaSchema(kafkaConfig, topic);
+
+ Catalog catalog =
Review Comment:
Merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] Alibaba-HZY commented on a diff in pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "Alibaba-HZY (via GitHub)" <gi...@apache.org>.
Alibaba-HZY commented on code in PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#discussion_r1206327654
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private String databaseName;
+ private String tableName;
+ private final Map<String, DataType> fields;
+ private final List<String> primaryKeys;
+ private static final int RETRY = Integer.MAX_VALUE;
Review Comment:
If it use a small value, this is equivalent to assuming that kafka topic already has data when the user starts the cdc job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] Alibaba-HZY commented on a diff in pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "Alibaba-HZY (via GitHub)" <gi...@apache.org>.
Alibaba-HZY commented on code in PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#discussion_r1206320120
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private String databaseName;
+ private String tableName;
+ private final Map<String, DataType> fields;
+ private final List<String> primaryKeys;
+ private static final int RETRY = Integer.MAX_VALUE;
+
+ public KafkaSchema(Configuration kafkaConfig, String topic) throws Exception {
+
+ fields = new LinkedHashMap<>();
+ primaryKeys = new ArrayList<>();
+ KafkaConsumer<String, String> consumer = getKafkaConsumer(kafkaConfig);
+
+ consumer.subscribe(Collections.singletonList(topic));
+
+ int retry = 0;
+ boolean success = false;
+ while (!success) {
+ ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
+ for (ConsumerRecord<String, String> record : records) {
Review Comment:
it needs multiple cycles,now it may be wrong when first record is ddl event, so parseCanalJson need return bool
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] Alibaba-HZY commented on a diff in pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "Alibaba-HZY (via GitHub)" <gi...@apache.org>.
Alibaba-HZY commented on code in PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#discussion_r1206408770
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final ObjectMapper objectMapper = new ObjectMapper();
+
+ private String databaseName;
+ private String tableName;
+ private final Map<String, DataType> fields;
+ private final List<String> primaryKeys;
+ private static final int RETRY = Integer.MAX_VALUE;
+
+ public KafkaSchema(Configuration kafkaConfig, String topic) throws Exception {
+
+ fields = new LinkedHashMap<>();
+ primaryKeys = new ArrayList<>();
+ KafkaConsumer<String, String> consumer = getKafkaConsumer(kafkaConfig);
+
+ consumer.subscribe(Collections.singletonList(topic));
+
+ int retry = 0;
+ boolean success = false;
+ while (!success) {
+ ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
+ for (ConsumerRecord<String, String> record : records) {
Review Comment:
i will add test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#issuecomment-1567759090
> > I created #1254 for sink API public.
>
> Wait for this pr to merge and modify the kafkaSynctableAction.class?
You can rebase now~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] Alibaba-HZY closed pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "Alibaba-HZY (via GitHub)" <gi...@apache.org>.
Alibaba-HZY closed pull request #1239: [flink] Supports KafkaSyncTableAction
URL: https://github.com/apache/incubator-paimon/pull/1239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] Alibaba-HZY commented on pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "Alibaba-HZY (via GitHub)" <gi...@apache.org>.
Alibaba-HZY commented on PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#issuecomment-1567697963
> I created #1254 for sink API public.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] Alibaba-HZY commented on pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "Alibaba-HZY (via GitHub)" <gi...@apache.org>.
Alibaba-HZY commented on PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#issuecomment-1567774725
> > > I created #1254 for sink API public.
> >
> >
> > Wait for this pr to merge and modify the kafkaSynctableAction.class?
>
> You can rebase now~
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1239: [flink] Supports KafkaSyncTableAction
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1239:
URL: https://github.com/apache/incubator-paimon/pull/1239#discussion_r1208811407
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
Review Comment:
Add some ut cases for `KafkaSchema` and `KafkaActionUtils`?
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.action.cdc.kafka;
+
+import org.apache.paimon.flink.action.cdc.mysql.MySqlTypeUtils;
+import org.apache.paimon.types.DataType;
+
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.serialization.StringDeserializer;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/** Utility class to load canal kafka schema. */
+public class KafkaSchema {
+
+ private static final int MAX_RETRY = 10;
Review Comment:
Maybe 10 is too small, we can change it to 100.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org