You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ku...@apache.org on 2026/03/12 07:47:30 UTC
(flink-cdc) branch FLINK-38729-2 updated (2d09d68c0 -> 3240e50fa)
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a change to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
omit 2d09d68c0 Address comment.
omit 41172a30f Remove flink-cdc-pipeline-e2e-tests-2.x module.
omit 714021db1 Rename module to avoid using minor version.
omit c4247b266 Update java doc.
omit 1bfa51294 Add flink-cdc-flink1.20-compat module.
omit e5e8b48be Fix compile.
omit 16c483c5a Add flink-cdc-flink2.2-compat module.
omit a3adc9d3f Add flink-cdc-flink2.2-compat module.
omit b0764446c Fix CI failure.
omit 112bc35ac Fix CI failure.
omit 1d48d77d0 Address comments.
omit b03ff0ce4 Add support for Flink 2.x.
add c003b9352 [FLINK-39152][pipeline-connector][Fluss]Fix FlussConfig Utils class is not in the correct directory and should have a private instance (#4306)
add dc5e92341 [FLINK-38160][starrocks] Add support for BINARY and VARBINARY types in StarRocks connector (#4303)
add 88b120324 [DOC][MINOR] Update starrocks.md (#4304)
add 0ae9abda1 [FLINK-37292] [cdc] Remove the flink-kubernetes dependencies during the packaging of flink-cdc-dist (#4024)
add 37783004a [minor][docs][style] Update documentation link and improve code style (#3961)
add 570faa7a0 [FLINK-38247][MySQL] Handle BIGINT UNSIGNED overflow correctly in PreparedStatement. (#4117)
add 91ae6776e [FLINK-39055][Iceberg] Support default column values in Iceberg sink connector (#4277)
add 363095e07 [FLINK-38244][hotfix] Fix the full snapshot phase, the field case is adjusted based on isTableIdCaseInsensitive (#4284)
add 3580bf414 [FLINK-38844][pipeline-connector][postgres]Add metadata column support (#4202)
add 368a8079b [FLINK-39231][test][sqlserver] Fix unstable sqlserver connector test (#4309)
add db595e405 [hotfix] Fix spotless check failure (#4313)
new 7c28071ea Add support for Flink 2.x.
new 5b763bd0b Address comments.
new 557cd42c5 Fix CI failure.
new bbdba2707 Fix CI failure.
new 0ddbfb4fa Add flink-cdc-flink2.2-compat module.
new 60acff8c0 Add flink-cdc-flink2.2-compat module.
new 0faf78079 Fix compile.
new cbeecde33 Add flink-cdc-flink1.20-compat module.
new 10e3747d8 Update java doc.
new f102d04dc Rename module to avoid using minor version.
new 45948829e Remove flink-cdc-pipeline-e2e-tests-2.x module.
new 3240e50fa Address comment.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (2d09d68c0)
\
N -- N -- N refs/heads/FLINK-38729-2 (3240e50fa)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 12 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../connectors/pipeline-connectors/postgres.md | 76 +++++-
.../connectors/pipeline-connectors/starrocks.md | 17 +-
.../connectors/pipeline-connectors/postgres.md | 76 +++++-
.../connectors/pipeline-connectors/starrocks.md | 17 +-
flink-cdc-composer/pom.xml | 1 +
.../sink/ElasticsearchDataSinkFactory.java | 2 +-
.../fluss/factory/FlussDataSinkFactory.java | 4 +-
.../fluss/{sink => utils}/FlussConfigUtils.java | 4 +-
.../iceberg/sink/IcebergMetadataApplier.java | 56 +++-
.../iceberg/sink/utils/IcebergTypeUtils.java | 67 +++++
.../iceberg/sink/IcebergMetadataApplierTest.java | 290 +++++++++++++++++++++
.../iceberg/sink/utils/IcebergTypeUtilsTest.java | 139 ++++++++++
.../connectors/mysql/source/MySqlDataSource.java | 9 +-
.../mysql/source/MySqlEventDeserializer.java | 5 +-
.../source/reader/MySqlPipelineRecordEmitter.java | 18 +-
.../mysql/source/MySqlDataSourceFactoryTest.java | 3 +-
.../mysql/source/MySqlPipelineITCase.java | 8 +-
.../source/MySqlTableIdCaseInsensitveITCase.java | 91 +++++--
.../src/test/resources/ddl/inventory.sql | 22 +-
.../source/DatabaseNameMetadataColumn.java} | 17 +-
.../postgres}/source/OpTsMetadataColumn.java | 2 +-
.../postgres/source/PostgresDataSource.java | 18 ++
.../postgres/source/SchemaNameMetadataColumn.java} | 17 +-
.../postgres/source/TableNameMetadataColumn.java} | 16 +-
.../factory/PostgresDataSourceFactoryTest.java | 32 +++
.../postgres/source/PostgresFullTypesITCase.java | 106 ++++++++
.../starrocks/sink/StarRocksDataSinkOptions.java | 4 +-
.../connectors/starrocks/sink/StarRocksUtils.java | 26 ++
.../starrocks/sink/CdcDataTypeTransformerTest.java | 37 +++
.../sink/EventRecordSerializationSchemaTest.java | 74 ++++++
.../sink/StarRocksMetadataApplierITCase.java | 13 +-
.../starrocks/sink/StarRocksUtilsTest.java | 74 +++++-
.../mysql/source/utils/StatementUtils.java | 36 ++-
.../connectors/mysql/source/MySqlSourceITCase.java | 94 +++++++
.../mysql/source/utils/StatementUtilsTest.java | 113 ++++++++
.../sqlserver/table/SqlServerConnectorITCase.java | 26 +-
36 files changed, 1492 insertions(+), 118 deletions(-)
rename flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/main/java/org/apache/flink/cdc/connectors/fluss/{sink => utils}/FlussConfigUtils.java (97%)
create mode 100644 flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/utils/IcebergTypeUtilsTest.java
copy flink-cdc-connect/flink-cdc-pipeline-connectors/{flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/source/OpTsMetadataColumn.java => flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres/source/DatabaseNameMetadataColumn.java} (73%)
copy flink-cdc-connect/flink-cdc-pipeline-connectors/{flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql => flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres}/source/OpTsMetadataColumn.java (96%)
copy flink-cdc-connect/flink-cdc-pipeline-connectors/{flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/source/OpTsMetadataColumn.java => flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres/source/SchemaNameMetadataColumn.java} (73%)
copy flink-cdc-connect/flink-cdc-pipeline-connectors/{flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/OpTsMetadataColumn.java => flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres/source/TableNameMetadataColumn.java} (74%)
create mode 100644 flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/utils/StatementUtilsTest.java
(flink-cdc) 04/12: Fix CI failure.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit bbdba2707d9593a8527289331f4d821b95ddbf57
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Mon Mar 9 18:02:07 2026 +0800
Fix CI failure.
---
.../sink/DataSinkWriterOperatorFactory.java | 47 +++++++++++++++++++---
1 file changed, 41 insertions(+), 6 deletions(-)
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java
index a286ac2e7..ab64f9c2b 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java
@@ -17,6 +17,7 @@
package org.apache.flink.cdc.runtime.operators.sink;
+import org.apache.flink.api.common.operators.MailboxExecutor;
import org.apache.flink.api.connector.sink2.Sink;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.event.Event;
@@ -28,6 +29,8 @@ import org.apache.flink.streaming.api.operators.OneInputStreamOperatorFactory;
import org.apache.flink.streaming.api.operators.StreamOperator;
import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import java.lang.reflect.Method;
+
/** Operator factory for {@link DataSinkWriterOperator}. */
@Internal
public class DataSinkWriterOperatorFactory<CommT>
@@ -50,10 +53,10 @@ public class DataSinkWriterOperatorFactory<CommT>
@Override
public <T extends StreamOperator<CommittableMessage<CommT>>> T createStreamOperator(
StreamOperatorParameters<CommittableMessage<CommT>> parameters) {
+ MailboxExecutor mailboxExecutor = getMailboxExecutor(parameters);
if (isBounded) {
BatchDataSinkWriterOperator<CommT> writerOperator =
- new BatchDataSinkWriterOperator<>(
- sink, processingTimeService, parameters.getMailboxExecutor());
+ new BatchDataSinkWriterOperator<>(sink, processingTimeService, mailboxExecutor);
writerOperator.setup(
parameters.getContainingTask(),
parameters.getStreamConfig(),
@@ -62,10 +65,7 @@ public class DataSinkWriterOperatorFactory<CommT>
}
DataSinkWriterOperator<CommT> writerOperator =
new DataSinkWriterOperator<>(
- sink,
- processingTimeService,
- parameters.getMailboxExecutor(),
- schemaOperatorID);
+ sink, processingTimeService, mailboxExecutor, schemaOperatorID);
writerOperator.setup(
parameters.getContainingTask(),
parameters.getStreamConfig(),
@@ -73,6 +73,41 @@ public class DataSinkWriterOperatorFactory<CommT>
return (T) writerOperator;
}
+ /**
+ * Obtains the {@link MailboxExecutor} in a way compatible with both Flink 1.19 and 1.20+.
+ *
+ * <p>Flink 1.20+ added {@code StreamOperatorParameters.getMailboxExecutor()}. In Flink 1.19 and
+ * earlier, the executor must be obtained via {@code
+ * StreamTask.getMailboxExecutorFactory().createExecutor(chainIndex)}.
+ */
+ private MailboxExecutor getMailboxExecutor(
+ StreamOperatorParameters<CommittableMessage<CommT>> parameters) {
+ // Try Flink 1.20+ / 2.x API: StreamOperatorParameters.getMailboxExecutor()
+ try {
+ Method m = parameters.getClass().getMethod("getMailboxExecutor");
+ return (MailboxExecutor) m.invoke(parameters);
+ } catch (NoSuchMethodException ignored) {
+ // Fall through to Flink 1.19 path
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to invoke getMailboxExecutor on StreamOperatorParameters", e);
+ }
+
+ // Flink 1.19 and earlier: obtain from containingTask.getMailboxExecutorFactory()
+ try {
+ Object containingTask = parameters.getContainingTask();
+ Method getFactory = containingTask.getClass().getMethod("getMailboxExecutorFactory");
+ Object factory = getFactory.invoke(containingTask);
+ int chainIndex = parameters.getStreamConfig().getChainIndex();
+ Method createExecutor = factory.getClass().getMethod("createExecutor", int.class);
+ return (MailboxExecutor) createExecutor.invoke(factory, chainIndex);
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to obtain MailboxExecutor from StreamTask for Flink 1.19 compatibility",
+ e);
+ }
+ }
+
@Override
public Class<? extends StreamOperator> getStreamOperatorClass(ClassLoader classLoader) {
if (isBounded) {
(flink-cdc) 12/12: Address comment.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 3240e50fa53e4d9be66a4366f6f480cd3d9c5227
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Thu Mar 12 10:41:13 2026 +0800
Address comment.
---
.../flink-cdc-pipeline-e2e-tests/pom.xml | 39 ----------------------
.../flink/cdc/pipeline/tests/FlussE2eITCase.java | 2 ++
.../cdc/pipeline/tests/MaxComputeE2eITCase.java | 2 ++
.../cdc/pipeline/tests/MySqlToDorisE2eITCase.java | 2 ++
.../cdc/pipeline/tests/MySqlToHudiE2eITCase.java | 2 ++
.../pipeline/tests/MySqlToIcebergE2eITCase.java | 2 ++
.../cdc/pipeline/tests/MySqlToPaimonE2eITCase.java | 2 ++
.../flink/cdc/pipeline/tests/MysqlE2eITCase.java | 2 ++
.../tests/MysqlE2eWithYarnApplicationITCase.java | 2 ++
.../cdc/pipeline/tests/MysqlToKafkaE2eITCase.java | 2 ++
.../cdc/pipeline/tests/OceanBaseE2eITCase.java | 2 ++
.../flink/cdc/pipeline/tests/OracleE2eITCase.java | 2 ++
.../cdc/pipeline/tests/PostgresE2eITCase.java | 2 ++
.../flink/cdc/pipeline/tests/RouteE2eITCase.java | 2 ++
.../cdc/pipeline/tests/SchemaEvolveE2eITCase.java | 2 ++
.../tests/SchemaEvolvingTransformE2eITCase.java | 2 ++
.../cdc/pipeline/tests/TransformE2eITCase.java | 2 ++
.../flink/cdc/pipeline/tests/UdfE2eITCase.java | 2 ++
.../migration/MySqlToPaimonMigrationITCase.java | 2 ++
.../tests/migration/YamlJobMigrationITCase.java | 2 ++
.../tests/specs/FlinkPipelineSpecsITCase.java | 2 ++
21 files changed, 40 insertions(+), 39 deletions(-)
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/pom.xml b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/pom.xml
index 897d1acc3..8c263f8de 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/pom.xml
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/pom.xml
@@ -805,43 +805,4 @@ limitations under the License.
</plugin>
</plugins>
</build>
-
- <profiles>
- <profile>
- <id>flink2</id>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <executions>
- <execution>
- <id>default-test</id>
- <phase>none</phase>
- </execution>
- <execution>
- <id>integration-tests</id>
- <phase>none</phase>
- </execution>
- <execution>
- <id>end-to-end-tests</id>
- <configuration>
- <includes>
- <include>**/ValuesE2eITCase.java</include>
- </includes>
- <excludes>
- <exclude>**/MysqlE2eWithYarnApplicationITCase.java</exclude>
- </excludes>
- </configuration>
- </execution>
- <execution>
- <id>run-last-test</id>
- <phase>none</phase>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- </profile>
- </profiles>
</project>
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/FlussE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/FlussE2eITCase.java
index 8f12e20c1..05b34dab4 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/FlussE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/FlussE2eITCase.java
@@ -26,6 +26,7 @@ import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
import org.assertj.core.api.Assertions;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import org.slf4j.Logger;
@@ -48,6 +49,7 @@ import java.util.List;
import java.util.stream.Collectors;
/** An End-to-end test case for Fluss pipeline connector. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
@Testcontainers
public class FlussE2eITCase extends PipelineTestEnvironment {
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java
index 59d966bc2..f8ae049f5 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java
@@ -34,6 +34,7 @@ import com.aliyun.odps.task.SQLTask;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.GenericContainer;
@@ -56,6 +57,7 @@ import java.util.concurrent.TimeUnit;
import static org.assertj.core.api.Assertions.assertThat;
/** End-to-end tests for maxcompute cdc pipeline job. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class MaxComputeE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(MaxComputeE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToDorisE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToDorisE2eITCase.java
index 434ccbcf6..351e3ccea 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToDorisE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToDorisE2eITCase.java
@@ -27,6 +27,7 @@ import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy;
@@ -49,6 +50,7 @@ import java.util.stream.Collectors;
import java.util.stream.Stream;
/** End-to-end tests for mysql cdc to Doris pipeline job. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class MySqlToDorisE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(MySqlToDorisE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java
index 76936b698..d1838041d 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java
@@ -35,6 +35,7 @@ import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.BindMode;
@@ -64,6 +65,7 @@ import java.util.stream.Collectors;
import java.util.stream.Stream;
/** End-to-end tests for mysql cdc to Iceberg pipeline job. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
public class MySqlToHudiE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(MySqlToHudiE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToIcebergE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToIcebergE2eITCase.java
index c63c54ad6..6614d448a 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToIcebergE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToIcebergE2eITCase.java
@@ -38,6 +38,7 @@ import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.api.io.TempDir;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -70,6 +71,7 @@ import java.util.stream.Collectors;
import java.util.stream.Stream;
/** End-to-end tests for mysql cdc to Iceberg pipeline job. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
public class MySqlToIcebergE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(MySqlToIcebergE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToPaimonE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToPaimonE2eITCase.java
index d263dfd53..f0e7cb082 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToPaimonE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToPaimonE2eITCase.java
@@ -26,6 +26,7 @@ import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.Container;
@@ -48,6 +49,7 @@ import java.util.stream.Collectors;
import java.util.stream.Stream;
/** End-to-end tests for mysql cdc to Paimon pipeline job. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class MySqlToPaimonE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(MySqlToPaimonE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
index b85d77ebc..372b9f7dd 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
@@ -24,6 +24,7 @@ import org.apache.flink.cdc.pipeline.tests.utils.PipelineTestEnvironment;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -36,6 +37,7 @@ import java.time.Duration;
import java.util.function.Function;
/** End-to-end tests for mysql cdc pipeline job. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class MysqlE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(MysqlE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eWithYarnApplicationITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eWithYarnApplicationITCase.java
index cab98c6af..7de50469b 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eWithYarnApplicationITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eWithYarnApplicationITCase.java
@@ -27,6 +27,7 @@ import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.Network;
@@ -50,6 +51,7 @@ import java.util.stream.Stream;
import static org.assertj.core.api.Assertions.assertThat;
/** End-to-end tests for mysql cdc pipeline job. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
public class MysqlE2eWithYarnApplicationITCase extends PipelineTestOnYarnEnvironment {
private static final Logger LOG =
LoggerFactory.getLogger(MysqlE2eWithYarnApplicationITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlToKafkaE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlToKafkaE2eITCase.java
index a8093f0f7..5090d56f4 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlToKafkaE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlToKafkaE2eITCase.java
@@ -38,6 +38,7 @@ import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.KafkaContainer;
@@ -71,6 +72,7 @@ import static org.apache.flink.util.DockerImageVersions.KAFKA;
import static org.assertj.core.api.Assertions.assertThat;
/** End-to-end tests for mysql cdc to Kafka pipeline job. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class MysqlToKafkaE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(MysqlToKafkaE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/OceanBaseE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/OceanBaseE2eITCase.java
index 2bebedb3b..7bbf61378 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/OceanBaseE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/OceanBaseE2eITCase.java
@@ -29,6 +29,7 @@ import org.apache.commons.text.StringEscapeUtils;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.output.Slf4jLogConsumer;
@@ -51,6 +52,7 @@ import java.util.stream.Stream;
import static org.assertj.core.api.Assertions.assertThat;
/** OceanBase flink cdc pipeline connector sink integrate test. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class OceanBaseE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(OceanBaseE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/OracleE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/OracleE2eITCase.java
index 0be4734e7..f474815e4 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/OracleE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/OracleE2eITCase.java
@@ -27,6 +27,7 @@ import org.assertj.core.api.Assertions;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.DockerClientFactory;
@@ -61,6 +62,7 @@ import static org.apache.flink.cdc.connectors.oracle.source.OracleSourceTestBase
import static org.apache.flink.cdc.connectors.oracle.source.OracleSourceTestBase.TOP_USER;
/** End-to-end tests for Oracle cdc pipeline job. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
public class OracleE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(OracleE2eITCase.class);
protected static final Pattern COMMENT_PATTERN = Pattern.compile("^(.*)--.*$");
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/PostgresE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/PostgresE2eITCase.java
index e4b33b3ff..69a65682f 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/PostgresE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/PostgresE2eITCase.java
@@ -24,6 +24,7 @@ import org.apache.flink.cdc.pipeline.tests.utils.PipelineTestEnvironment;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.PostgreSQLContainer;
@@ -42,6 +43,7 @@ import static org.apache.flink.cdc.connectors.postgres.PostgresTestBase.getJdbcC
import static org.apache.flink.cdc.connectors.postgres.PostgresTestBase.getSlotName;
/** End-to-end tests for postgres cdc pipeline job. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
public class PostgresE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(PostgresE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java
index 2514711c0..c9169388d 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java
@@ -23,6 +23,7 @@ import org.apache.flink.cdc.pipeline.tests.utils.PipelineTestEnvironment;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import org.slf4j.Logger;
@@ -37,6 +38,7 @@ import java.util.function.Function;
import java.util.stream.IntStream;
/** E2e tests for routing features. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class RouteE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(RouteE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
index a6ade384c..7595dd6d1 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
@@ -24,6 +24,7 @@ import org.assertj.core.api.Assertions;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -36,6 +37,7 @@ import java.util.Collections;
import java.util.List;
/** E2e tests for Schema Evolution cases. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class SchemaEvolveE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(SchemaEvolveE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
index 130dba0c4..5410eb903 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
@@ -24,6 +24,7 @@ import org.assertj.core.api.Assertions;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -36,6 +37,7 @@ import java.util.Collections;
import java.util.List;
/** E2e tests for Schema Evolution cases. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class SchemaEvolvingTransformE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG =
LoggerFactory.getLogger(SchemaEvolvingTransformE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
index 351bd76fd..925bc6059 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
@@ -27,6 +27,7 @@ import org.apache.flink.cdc.runtime.operators.transform.PreTransformOperator;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import org.slf4j.Logger;
@@ -51,6 +52,7 @@ import java.util.stream.Stream;
import static org.assertj.core.api.Assertions.assertThat;
/** E2e tests for the {@link PreTransformOperator} and {@link PostTransformOperator}. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class TransformE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(TransformE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java
index ca8e3d764..c55a67c76 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java
@@ -23,6 +23,7 @@ import org.apache.flink.cdc.pipeline.tests.utils.PipelineTestEnvironment;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;
@@ -39,6 +40,7 @@ import java.util.function.Function;
import java.util.stream.Stream;
/** E2e tests for User-defined functions. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class UdfE2eITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(TransformE2eITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/migration/MySqlToPaimonMigrationITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/migration/MySqlToPaimonMigrationITCase.java
index bcedcf7b0..bcfac098f 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/migration/MySqlToPaimonMigrationITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/migration/MySqlToPaimonMigrationITCase.java
@@ -28,6 +28,7 @@ import org.assertj.core.api.Assertions;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EnumSource;
import org.slf4j.Logger;
@@ -52,6 +53,7 @@ import java.util.stream.Collectors;
/**
* E2e cases for stopping & restarting jobs of `MySQL source to Paimon sink` from previous state.
*/
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class MySqlToPaimonMigrationITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(MySqlToPaimonMigrationITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/migration/YamlJobMigrationITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/migration/YamlJobMigrationITCase.java
index bdf970ea6..ef2112935 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/migration/YamlJobMigrationITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/migration/YamlJobMigrationITCase.java
@@ -29,6 +29,7 @@ import org.assertj.core.api.Assertions;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EnumSource;
import org.slf4j.Logger;
@@ -43,6 +44,7 @@ import java.time.Duration;
import java.util.function.Function;
/** E2e cases for stopping & restarting jobs from previous state. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
class YamlJobMigrationITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(YamlJobMigrationITCase.class);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/specs/FlinkPipelineSpecsITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/specs/FlinkPipelineSpecsITCase.java
index b7ac0507c..0d5515cb3 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/specs/FlinkPipelineSpecsITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/specs/FlinkPipelineSpecsITCase.java
@@ -23,6 +23,7 @@ import org.apache.flink.cdc.pipeline.tests.utils.PipelineTestEnvironment;
import org.assertj.core.api.Assertions;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;
@@ -50,6 +51,7 @@ import java.util.stream.Collectors;
import java.util.stream.Stream;
/** YAML-spec based pipeline test cases. */
+@EnabledIfSystemProperty(named = "specifiedFlinkVersion", matches = "^1.*")
public class FlinkPipelineSpecsITCase extends PipelineTestEnvironment {
private static final Logger LOG = LoggerFactory.getLogger(FlinkPipelineSpecsITCase.class);
(flink-cdc) 10/12: Rename module to avoid using minor version.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit f102d04dca3309e7f2adb52e34a1c032dcf9ae1e
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Wed Mar 11 12:52:17 2026 +0800
Rename module to avoid using minor version.
---
flink-cdc-cli/pom.xml | 2 +-
flink-cdc-common/pom.xml | 2 +-
flink-cdc-composer/pom.xml | 4 ++--
.../flink-cdc-pipeline-connector-values/pom.xml | 2 +-
flink-cdc-dist/pom.xml | 4 ++--
{flink-cdc-flink1.20-compat => flink-cdc-flink1-compat}/pom.xml | 2 +-
.../flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java | 0
.../flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java | 0
.../main/java/org/apache/flink/runtime/event/WatermarkEvent.java | 0
{flink-cdc-flink2.2-compat => flink-cdc-flink2-compat}/pom.xml | 2 +-
.../flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java | 0
.../src/main/java/org/apache/flink/api/connector/sink2/Sink.java | 0
.../apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java | 0
.../flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java | 0
.../src/main/java/org/apache/flink/core/execution/RestoreMode.java | 0
.../org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java | 0
.../flink/streaming/api/connector/sink2/WithPostCommitTopology.java | 0
.../flink/streaming/api/connector/sink2/WithPreCommitTopology.java | 0
.../flink/streaming/api/connector/sink2/WithPreWriteTopology.java | 0
.../org/apache/flink/streaming/api/functions/sink/SinkFunction.java | 0
.../streaming/api/functions/source/RichParallelSourceFunction.java | 0
.../apache/flink/streaming/api/functions/source/SourceFunction.java | 0
.../flink/streaming/api/operators/YieldingOperatorFactory.java | 0
.../main/java/org/apache/flink/table/api/ValidationException.java | 0
flink-cdc-runtime/pom.xml | 6 +++---
pom.xml | 4 ++--
26 files changed, 14 insertions(+), 14 deletions(-)
diff --git a/flink-cdc-cli/pom.xml b/flink-cdc-cli/pom.xml
index 6230f22a1..fe335fc28 100644
--- a/flink-cdc-cli/pom.xml
+++ b/flink-cdc-cli/pom.xml
@@ -91,7 +91,7 @@ limitations under the License.
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <artifactId>flink-cdc-flink2-compat</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
diff --git a/flink-cdc-common/pom.xml b/flink-cdc-common/pom.xml
index 4101ad51b..80237d26d 100644
--- a/flink-cdc-common/pom.xml
+++ b/flink-cdc-common/pom.xml
@@ -65,7 +65,7 @@ limitations under the License.
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <artifactId>flink-cdc-flink2-compat</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
diff --git a/flink-cdc-composer/pom.xml b/flink-cdc-composer/pom.xml
index bf3eff42a..4161491ba 100644
--- a/flink-cdc-composer/pom.xml
+++ b/flink-cdc-composer/pom.xml
@@ -117,7 +117,7 @@ limitations under the License.
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <artifactId>flink-cdc-flink2-compat</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
@@ -131,7 +131,7 @@ limitations under the License.
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-flink1.20-compat</artifactId>
+ <artifactId>flink-cdc-flink1-compat</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/pom.xml b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/pom.xml
index 96052ee64..d28b7fa01 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/pom.xml
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/pom.xml
@@ -122,7 +122,7 @@ limitations under the License.
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <artifactId>flink-cdc-flink2-compat</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
diff --git a/flink-cdc-dist/pom.xml b/flink-cdc-dist/pom.xml
index 1a6b5500d..8c918fddc 100644
--- a/flink-cdc-dist/pom.xml
+++ b/flink-cdc-dist/pom.xml
@@ -60,7 +60,7 @@ limitations under the License.
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <artifactId>flink-cdc-flink2-compat</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
@@ -78,7 +78,7 @@ limitations under the License.
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-flink1.20-compat</artifactId>
+ <artifactId>flink-cdc-flink1-compat</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
diff --git a/flink-cdc-flink1.20-compat/pom.xml b/flink-cdc-flink1-compat/pom.xml
similarity index 96%
rename from flink-cdc-flink1.20-compat/pom.xml
rename to flink-cdc-flink1-compat/pom.xml
index 45a99cccd..8b21cdc2e 100644
--- a/flink-cdc-flink1.20-compat/pom.xml
+++ b/flink-cdc-flink1-compat/pom.xml
@@ -25,7 +25,7 @@ limitations under the License.
</parent>
<modelVersion>4.0.0</modelVersion>
- <artifactId>flink-cdc-flink1.20-compat</artifactId>
+ <artifactId>flink-cdc-flink1-compat</artifactId>
<properties>
<flink.version>${flink.1.x.version}</flink.version>
diff --git a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java b/flink-cdc-flink1-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
similarity index 100%
rename from flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
rename to flink-cdc-flink1-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
diff --git a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java b/flink-cdc-flink1-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
similarity index 100%
rename from flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
rename to flink-cdc-flink1-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
diff --git a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/runtime/event/WatermarkEvent.java b/flink-cdc-flink1-compat/src/main/java/org/apache/flink/runtime/event/WatermarkEvent.java
similarity index 100%
rename from flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/runtime/event/WatermarkEvent.java
rename to flink-cdc-flink1-compat/src/main/java/org/apache/flink/runtime/event/WatermarkEvent.java
diff --git a/flink-cdc-flink2.2-compat/pom.xml b/flink-cdc-flink2-compat/pom.xml
similarity index 96%
rename from flink-cdc-flink2.2-compat/pom.xml
rename to flink-cdc-flink2-compat/pom.xml
index 5e755b2ee..4170b6f90 100644
--- a/flink-cdc-flink2.2-compat/pom.xml
+++ b/flink-cdc-flink2-compat/pom.xml
@@ -25,7 +25,7 @@ limitations under the License.
</parent>
<modelVersion>4.0.0</modelVersion>
- <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <artifactId>flink-cdc-flink2-compat</artifactId>
<properties>
<flink.version>${flink.2.x.version}</flink.version>
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/Sink.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/api/connector/sink2/Sink.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/Sink.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/api/connector/sink2/Sink.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/core/execution/RestoreMode.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/core/execution/RestoreMode.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/core/execution/RestoreMode.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/core/execution/RestoreMode.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPostCommitTopology.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPostCommitTopology.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPostCommitTopology.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPostCommitTopology.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreCommitTopology.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreCommitTopology.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreCommitTopology.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreCommitTopology.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreWriteTopology.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreWriteTopology.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreWriteTopology.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreWriteTopology.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/RichParallelSourceFunction.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/RichParallelSourceFunction.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/RichParallelSourceFunction.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/RichParallelSourceFunction.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/operators/YieldingOperatorFactory.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/operators/YieldingOperatorFactory.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/operators/YieldingOperatorFactory.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/streaming/api/operators/YieldingOperatorFactory.java
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/table/api/ValidationException.java b/flink-cdc-flink2-compat/src/main/java/org/apache/flink/table/api/ValidationException.java
similarity index 100%
rename from flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/table/api/ValidationException.java
rename to flink-cdc-flink2-compat/src/main/java/org/apache/flink/table/api/ValidationException.java
diff --git a/flink-cdc-runtime/pom.xml b/flink-cdc-runtime/pom.xml
index 13c1eff0f..fc474013c 100644
--- a/flink-cdc-runtime/pom.xml
+++ b/flink-cdc-runtime/pom.xml
@@ -124,7 +124,7 @@ limitations under the License.
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <artifactId>flink-cdc-flink2-compat</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
@@ -138,7 +138,7 @@ limitations under the License.
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-flink1.20-compat</artifactId>
+ <artifactId>flink-cdc-flink1-compat</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
@@ -154,7 +154,7 @@ limitations under the License.
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-flink1.20-compat</artifactId>
+ <artifactId>flink-cdc-flink1-compat</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
<exclusions>
diff --git a/pom.xml b/pom.xml
index a9f49f51b..53220c9ef 100644
--- a/pom.xml
+++ b/pom.xml
@@ -33,8 +33,8 @@ limitations under the License.
<packaging>pom</packaging>
<modules>
- <module>flink-cdc-flink1.20-compat</module>
- <module>flink-cdc-flink2.2-compat</module>
+ <module>flink-cdc-flink1-compat</module>
+ <module>flink-cdc-flink2-compat</module>
<module>flink-cdc-cli</module>
<module>flink-cdc-common</module>
<module>flink-cdc-composer</module>
(flink-cdc) 08/12: Add flink-cdc-flink1.20-compat module.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit cbeecde33bd26a1a5dc06d7a5a71eee65832298d
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Tue Mar 10 11:27:58 2026 +0800
Add flink-cdc-flink1.20-compat module.
---
.../java/org/apache/flink/cdc/cli/CliFrontend.java | 9 +-
.../flink/cdc/cli/utils/ConfigurationUtils.java | 14 +-
.../org/apache/flink/cdc/cli/CliFrontendTest.java | 3 +-
flink-cdc-composer/pom.xml | 14 +
flink-cdc-dist/pom.xml | 18 ++
flink-cdc-flink1.20-compat/pom.xml | 95 +++++++
.../typeutils/TypeSerializerSnapshotAdapter.java | 21 ++
.../operators/AbstractStreamOperatorAdapter.java | 23 ++
.../apache/flink/runtime/event/WatermarkEvent.java | 21 ++
.../typeutils/TypeSerializerSnapshotAdapter.java | 31 +++
.../operators/AbstractStreamOperatorAdapter.java | 38 +++
.../api/operators/YieldingOperatorFactory.java | 22 ++
flink-cdc-runtime/pom.xml | 36 +++
.../schema/distributed/SchemaOperator.java | 4 +-
.../schema/regular/BatchSchemaOperator.java | 7 +-
.../operators/schema/regular/SchemaOperator.java | 7 +-
.../sink/BatchDataSinkFunctionOperator.java | 3 -
.../sink/BatchDataSinkWriterOperator.java | 172 ++++--------
.../operators/sink/DataSinkFunctionOperator.java | 3 -
.../operators/sink/DataSinkWriterOperator.java | 292 +++++----------------
.../sink/DataSinkWriterOperatorFactory.java | 51 +---
.../operators/transform/PostTransformOperator.java | 4 +-
.../operators/transform/PreTransformOperator.java | 7 +-
.../BatchRegularPrePartitionOperator.java | 8 +-
.../DistributedPrePartitionOperator.java | 8 +-
.../partitioning/RegularPrePartitionOperator.java | 7 +-
.../cdc/runtime/serializer/EnumSerializer.java | 32 +--
.../NestedSerializersSnapshotDelegate.java | 36 +--
.../serializer/NullableSerializerWrapper.java | 21 +-
.../serializer/data/ArrayDataSerializer.java | 25 +-
.../serializer/data/DecimalDataSerializer.java | 22 +-
.../data/LocalZonedTimestampDataSerializer.java | 23 +-
.../runtime/serializer/data/MapDataSerializer.java | 23 +-
.../serializer/data/TimestampDataSerializer.java | 22 +-
.../data/ZonedTimestampDataSerializer.java | 20 +-
.../cdc/runtime/utils/FlinkCompatibilityUtils.java | 153 -----------
.../schema/distributed/SchemaEvolveTest.java | 7 +-
.../operators/sink/DataSinkOperatorAdapter.java | 14 +-
.../cdc/runtime/serializer/EnumSerializerTest.java | 7 +-
.../cdc/runtime/serializer/SerializerTestBase.java | 3 +-
.../DistributedEventOperatorTestHarness.java | 101 +++----
.../operators/RegularEventOperatorTestHarness.java | 112 ++++----
pom.xml | 1 +
43 files changed, 642 insertions(+), 898 deletions(-)
diff --git a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java
index 5f327eca2..0f170367a 100644
--- a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java
+++ b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java
@@ -27,7 +27,6 @@ import org.apache.flink.cdc.common.utils.StringUtils;
import org.apache.flink.cdc.composer.PipelineExecution;
import org.apache.flink.configuration.DeploymentOptions;
import org.apache.flink.core.fs.Path;
-import org.apache.flink.runtime.jobgraph.SavepointConfigOptions;
import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
import org.apache.flink.shaded.guava31.com.google.common.base.Joiner;
@@ -173,7 +172,13 @@ public class CliFrontend {
commandLine.getOptionValue(SAVEPOINT_CLAIM_MODE),
ConfigurationUtils.getClaimModeClass());
} else {
- restoreMode = SavepointConfigOptions.RESTORE_MODE.defaultValue();
+ try {
+ restoreMode =
+ ConfigurationUtils.getClaimModeClass().getField("DEFAULT").get(null);
+ } catch (NoSuchFieldException | IllegalAccessException e) {
+ throw new RuntimeException(
+ "Failed to get DEFAULT value from RestoreMode class.", e);
+ }
}
// allowNonRestoredState is always false because all operators are predefined.
diff --git a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java
index 7280c0679..eafd4da09 100644
--- a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java
+++ b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java
@@ -67,12 +67,16 @@ public class ConfigurationUtils {
public static Class<?> getClaimModeClass() {
try {
- return Class.forName("org.apache.flink.core.execution.RestoreMode");
- } catch (ClassNotFoundException ignored) {
+ return Class.forName("org.apache.flink.core.execution.RecoveryClaimMode");
+ } catch (ClassNotFoundException classNotFoundException) {
try {
- return Class.forName("org.apache.flink.runtime.jobgraph.RestoreMode");
- } catch (ClassNotFoundException e) {
- throw new RuntimeException(e);
+ return Class.forName("org.apache.flink.core.execution.RestoreMode");
+ } catch (ClassNotFoundException ignored) {
+ try {
+ return Class.forName("org.apache.flink.runtime.jobgraph.RestoreMode");
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
}
}
}
diff --git a/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java b/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java
index b9f42cefc..70aeebbfa 100644
--- a/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java
+++ b/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java
@@ -107,7 +107,8 @@ class CliFrontendTest {
"-n");
assertThat(executor.getFlinkConfig().get(SAVEPOINT_PATH))
.isEqualTo(flinkHome() + "/savepoints/savepoint-1");
- assertThat(executor.getFlinkConfig().get(RESTORE_MODE)).isEqualTo(RestoreMode.NO_CLAIM);
+ assertThat(executor.getFlinkConfig().get(RESTORE_MODE).toString())
+ .isEqualTo(RestoreMode.NO_CLAIM.toString());
assertThat(executor.getFlinkConfig().get(SAVEPOINT_IGNORE_UNCLAIMED_STATE)).isTrue();
}
diff --git a/flink-cdc-composer/pom.xml b/flink-cdc-composer/pom.xml
index 7d0244e6a..bf3eff42a 100644
--- a/flink-cdc-composer/pom.xml
+++ b/flink-cdc-composer/pom.xml
@@ -123,6 +123,20 @@ limitations under the License.
</dependency>
</dependencies>
</profile>
+ <profile>
+ <id>flink1</id>
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-flink1.20-compat</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+ </profile>
</profiles>
</project>
\ No newline at end of file
diff --git a/flink-cdc-dist/pom.xml b/flink-cdc-dist/pom.xml
index 5e9890bfe..1a6b5500d 100644
--- a/flink-cdc-dist/pom.xml
+++ b/flink-cdc-dist/pom.xml
@@ -70,6 +70,24 @@ limitations under the License.
</dependency>
</dependencies>
</profile>
+ <profile>
+ <id>flink1</id>
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-flink1.20-compat</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-shaded-guava</artifactId>
+ <version>${flink.1.x.shaded.guava.version}</version>
+ </dependency>
+ </dependencies>
+ </profile>
</profiles>
<build>
diff --git a/flink-cdc-flink1.20-compat/pom.xml b/flink-cdc-flink1.20-compat/pom.xml
new file mode 100644
index 000000000..b65c53497
--- /dev/null
+++ b/flink-cdc-flink1.20-compat/pom.xml
@@ -0,0 +1,95 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>flink-cdc-parent</artifactId>
+ <groupId>org.apache.flink</groupId>
+ <version>${revision}</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>flink-cdc-flink1.20-compat</artifactId>
+
+ <properties>
+ <flink.version>${flink.1.x.version}</flink.version>
+ <shaded.guava.version>${flink.1.x.shaded.guava.version}</shaded.guava.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-common</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-base</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-api-java-bridge</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-core</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <version>${slf4j.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-shaded-guava</artifactId>
+ <version>${shaded.guava.version}</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>test-jar</id>
+ <goals>
+ <goal>test-jar</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+</project>
\ No newline at end of file
diff --git a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
new file mode 100644
index 000000000..a74e14c97
--- /dev/null
+++ b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
@@ -0,0 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.typeutils;
+
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public interface TypeSerializerSnapshotAdapter<T> extends TypeSerializerSnapshot<T> {}
diff --git a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
new file mode 100644
index 000000000..b4fe16f5e
--- /dev/null
+++ b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.runtime.operators;
+
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public class AbstractStreamOperatorAdapter<OUT> extends AbstractStreamOperator<OUT> {}
diff --git a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/runtime/event/WatermarkEvent.java b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/runtime/event/WatermarkEvent.java
new file mode 100644
index 000000000..afa423894
--- /dev/null
+++ b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/runtime/event/WatermarkEvent.java
@@ -0,0 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.event;
+
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public class WatermarkEvent {}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
new file mode 100644
index 000000000..a969801ee
--- /dev/null
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.common.typeutils;
+
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public interface TypeSerializerSnapshotAdapter<T> extends TypeSerializerSnapshot<T> {
+
+ TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
+ TypeSerializer<T> typeSerializer);
+
+ @Override
+ default TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
+ TypeSerializerSnapshot<T> typeSerializerSnapshot) {
+ return resolveSchemaCompatibility(typeSerializerSnapshot.restoreSerializer());
+ }
+}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
new file mode 100644
index 000000000..f8cdb6f25
--- /dev/null
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.runtime.operators;
+
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public class AbstractStreamOperatorAdapter<OUT> extends AbstractStreamOperator<OUT> {
+ protected ChainingStrategy chainingStrategy;
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/operators/YieldingOperatorFactory.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/operators/YieldingOperatorFactory.java
new file mode 100644
index 000000000..34aaac426
--- /dev/null
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/operators/YieldingOperatorFactory.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public interface YieldingOperatorFactory<OUT>
+ extends org.apache.flink.streaming.api.operators.legacy.YieldingOperatorFactory<OUT> {}
diff --git a/flink-cdc-runtime/pom.xml b/flink-cdc-runtime/pom.xml
index 2a2072f0b..13c1eff0f 100644
--- a/flink-cdc-runtime/pom.xml
+++ b/flink-cdc-runtime/pom.xml
@@ -130,5 +130,41 @@ limitations under the License.
</dependency>
</dependencies>
</profile>
+ <profile>
+ <id>flink1</id>
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-flink1.20-compat</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+ </profile>
+ <profile>
+ <id>flink2-exclude-flink1</id>
+ <activation>
+ <property>
+ <name>flink.version</name>
+ <value>${flink.2.x.version}</value>
+ </property>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-flink1.20-compat</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ </profile>
</profiles>
</project>
\ No newline at end of file
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaOperator.java
index 45f91afaf..9ec2b2fc9 100755
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaOperator.java
@@ -29,6 +29,7 @@ import org.apache.flink.cdc.common.route.RouteRule;
import org.apache.flink.cdc.common.route.TableIdRouter;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.schema.common.CoordinationResponseUtils;
import org.apache.flink.cdc.runtime.operators.schema.common.SchemaDerivator;
import org.apache.flink.cdc.runtime.operators.schema.common.metrics.SchemaOperatorMetrics;
@@ -39,7 +40,6 @@ import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.runtime.operators.coordination.CoordinationRequest;
import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
import org.apache.flink.streaming.api.graph.StreamConfig;
-import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
@@ -61,7 +61,7 @@ import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
/** This operator merges upstream inferred schema into a centralized Schema Registry. */
-public class SchemaOperator extends AbstractStreamOperator<Event>
+public class SchemaOperator extends AbstractStreamOperatorAdapter<Event>
implements OneInputStreamOperator<PartitioningEvent, Event>, Serializable {
private static final Logger LOG = LoggerFactory.getLogger(SchemaOperator.class);
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/BatchSchemaOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/BatchSchemaOperator.java
index da089e6c5..8ed7d0916 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/BatchSchemaOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/BatchSchemaOperator.java
@@ -28,11 +28,10 @@ import org.apache.flink.cdc.common.route.RouteRule;
import org.apache.flink.cdc.common.route.TableIdRouter;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.sink.MetadataApplier;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.schema.common.SchemaDerivator;
import org.apache.flink.cdc.runtime.operators.schema.common.SchemaManager;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.streaming.api.graph.StreamConfig;
-import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
@@ -50,7 +49,7 @@ import java.util.stream.Collectors;
/** The operator will apply create table event and router mapper in batch mode. */
@Internal
-public class BatchSchemaOperator extends AbstractStreamOperator<Event>
+public class BatchSchemaOperator extends AbstractStreamOperatorAdapter<Event>
implements OneInputStreamOperator<Event, Event>, Serializable {
private static final long serialVersionUID = 1L;
@@ -71,7 +70,7 @@ public class BatchSchemaOperator extends AbstractStreamOperator<Event>
public BatchSchemaOperator(
List<RouteRule> routingRules, MetadataApplier metadataApplier, String timezone) {
- FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
+ this.chainingStrategy = ChainingStrategy.ALWAYS;
this.timezone = timezone;
this.routingRules = routingRules;
this.metadataApplier = metadataApplier;
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/SchemaOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/SchemaOperator.java
index f35d62ce6..7dd9a210a 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/SchemaOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/SchemaOperator.java
@@ -29,17 +29,16 @@ import org.apache.flink.cdc.common.route.RouteRule;
import org.apache.flink.cdc.common.route.TableIdRouter;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.schema.common.CoordinationResponseUtils;
import org.apache.flink.cdc.runtime.operators.schema.common.SchemaDerivator;
import org.apache.flink.cdc.runtime.operators.schema.common.metrics.SchemaOperatorMetrics;
import org.apache.flink.cdc.runtime.operators.schema.regular.event.SchemaChangeRequest;
import org.apache.flink.cdc.runtime.operators.schema.regular.event.SchemaChangeResponse;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.runtime.operators.coordination.CoordinationRequest;
import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
import org.apache.flink.streaming.api.graph.StreamConfig;
-import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
@@ -66,7 +65,7 @@ import static org.apache.flink.cdc.common.pipeline.PipelineOptions.DEFAULT_SCHEM
* SchemaChangeEvent}s and block the stream for tables before their schema changes finish.
*/
@Internal
-public class SchemaOperator extends AbstractStreamOperator<Event>
+public class SchemaOperator extends AbstractStreamOperatorAdapter<Event>
implements OneInputStreamOperator<Event, Event>, Serializable {
private static final long serialVersionUID = 1L;
@@ -110,7 +109,7 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
Duration rpcTimeOut,
SchemaChangeBehavior schemaChangeBehavior,
String timezone) {
- FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
+ this.chainingStrategy = ChainingStrategy.ALWAYS;
this.rpcTimeout = rpcTimeOut;
this.schemaChangeBehavior = schemaChangeBehavior;
this.timezone = timezone;
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkFunctionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkFunctionOperator.java
index 73a57f326..5b3925aad 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkFunctionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkFunctionOperator.java
@@ -20,9 +20,7 @@ package org.apache.flink.cdc.runtime.operators.sink;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.event.Event;
import org.apache.flink.cdc.runtime.operators.sink.exception.SinkWrapperException;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.streaming.api.functions.sink.SinkFunction;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.StreamSink;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
@@ -38,7 +36,6 @@ public class BatchDataSinkFunctionOperator extends StreamSink<Event> {
public BatchDataSinkFunctionOperator(SinkFunction<Event> userFunction) {
super(userFunction);
- FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
}
@Override
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java
index 5f4bd9acf..6ea1ceaee 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java
@@ -23,6 +23,7 @@ import org.apache.flink.api.connector.sink2.SinkWriter;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.event.Event;
import org.apache.flink.cdc.common.event.FlushEvent;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.sink.exception.SinkWrapperException;
import org.apache.flink.runtime.state.StateInitializationContext;
import org.apache.flink.runtime.state.StateSnapshotContext;
@@ -30,8 +31,10 @@ import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
import org.apache.flink.streaming.api.graph.StreamConfig;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
import org.apache.flink.streaming.api.watermark.Watermark;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
@@ -53,7 +56,7 @@ import java.lang.reflect.Method;
*/
@Internal
public class BatchDataSinkWriterOperator<CommT>
- extends AbstractStreamOperator<CommittableMessage<CommT>>
+ extends AbstractStreamOperatorAdapter<CommittableMessage<CommT>>
implements OneInputStreamOperator<Event, CommittableMessage<CommT>>, BoundedOneInput {
private final Sink<Event> sink;
@@ -78,6 +81,7 @@ public class BatchDataSinkWriterOperator<CommT>
this.sink = sink;
this.processingTimeService = processingTimeService;
this.mailboxExecutor = mailboxExecutor;
+ this.chainingStrategy = ChainingStrategy.ALWAYS;
}
@Override
@@ -87,43 +91,39 @@ public class BatchDataSinkWriterOperator<CommT>
Output<StreamRecord<CommittableMessage<CommT>>> output) {
super.setup(containingTask, config, output);
flinkWriterOperator = createFlinkWriterOperator();
- invokeFlinkWriterOperatorMethod(
- "setup",
- new Class<?>[] {StreamTask.class, StreamConfig.class, Output.class},
- containingTask,
- config,
- output);
+ invokeSetup(flinkWriterOperator, containingTask, config, output);
}
@Override
public void open() throws Exception {
- invokeFlinkWriterOperatorMethod("open", new Class<?>[0]);
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator().open();
copySinkWriter = getFieldValue("sinkWriter");
}
@Override
public void initializeState(StateInitializationContext context) throws Exception {
- invokeFlinkWriterOperatorMethod(
- "initializeState", new Class<?>[] {StateInitializationContext.class}, context);
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .initializeState(context);
}
@Override
public void snapshotState(StateSnapshotContext context) throws Exception {
- invokeFlinkWriterOperatorMethod(
- "snapshotState", new Class<?>[] {StateSnapshotContext.class}, context);
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .snapshotState(context);
}
@Override
public void processWatermark(Watermark mark) throws Exception {
super.processWatermark(mark);
- invokeFlinkWriterOperatorMethod("processWatermark", new Class<?>[] {Watermark.class}, mark);
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .processWatermark(mark);
}
@Override
public void processWatermarkStatus(WatermarkStatus watermarkStatus) throws Exception {
super.processWatermarkStatus(watermarkStatus);
- invokeFlinkWriterOperatorMethod(
- "processWatermarkStatus", new Class<?>[] {WatermarkStatus.class}, watermarkStatus);
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .processWatermarkStatus(watermarkStatus);
}
@Override
@@ -140,18 +140,19 @@ public class BatchDataSinkWriterOperator<CommT>
@Override
public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
- invokeFlinkWriterOperatorMethod(
- "prepareSnapshotPreBarrier", new Class<?>[] {long.class}, checkpointId);
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .prepareSnapshotPreBarrier(checkpointId);
}
@Override
public void close() throws Exception {
- invokeFlinkWriterOperatorMethod("close", new Class<?>[0]);
+ this.<OneInputStreamOperator<Event, CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .close();
}
@Override
public void endInput() throws Exception {
- invokeFlinkWriterOperatorMethod("endInput", new Class<?>[0]);
+ this.<BoundedOneInput>getFlinkWriterOperator().endInput();
}
// ----------------------------- Helper functions -------------------------------
@@ -162,6 +163,22 @@ public class BatchDataSinkWriterOperator<CommT>
// -------------------------- Reflection helper functions --------------------------
+ private void invokeSetup(
+ Object operator,
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<?> output) {
+ try {
+ Method setupMethod =
+ AbstractStreamOperator.class.getDeclaredMethod(
+ "setup", StreamTask.class, StreamConfig.class, Output.class);
+ setupMethod.setAccessible(true);
+ setupMethod.invoke(operator, containingTask, config, output);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to invoke setup on flinkWriterOperator", e);
+ }
+ }
+
private Object createFlinkWriterOperator() {
try {
Class<?> flinkWriterClass =
@@ -169,70 +186,29 @@ public class BatchDataSinkWriterOperator<CommT>
.getUserCodeClassLoader()
.loadClass(
"org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator");
-
- // Try to find a constructor whose first three parameters are compatible with
- // (Sink, ProcessingTimeService, MailboxExecutor). This makes the code resilient
- // against Flink 2.x adding extra parameters to the constructor.
- Constructor<?> target = null;
- for (Constructor<?> c : flinkWriterClass.getDeclaredConstructors()) {
- Class<?>[] p = c.getParameterTypes();
- if (p.length >= 3
- && Sink.class.isAssignableFrom(p[0])
- && ProcessingTimeService.class.isAssignableFrom(p[1])
- && MailboxExecutor.class.isAssignableFrom(p[2])) {
- target = c;
- break;
- }
- }
-
- if (target == null) {
- // Fallback: use the first declared constructor and best-effort argument mapping
- // below. This covers Flink 2.2 where the constructor signature may have changed.
- Constructor<?>[] all = flinkWriterClass.getDeclaredConstructors();
- if (all.length == 0) {
- throw new RuntimeException(
- "No constructors found on SinkWriterOperator in Flink runtime");
- }
- target = all[0];
- }
-
- target.setAccessible(true);
- Class<?>[] paramTypes = target.getParameterTypes();
- Object[] args = new Object[paramTypes.length];
- for (int i = 0; i < paramTypes.length; i++) {
- Class<?> t = paramTypes[i];
- if (Sink.class.isAssignableFrom(t)) {
- args[i] = sink;
- } else if (ProcessingTimeService.class.isAssignableFrom(t)) {
- args[i] = processingTimeService;
- } else if (MailboxExecutor.class.isAssignableFrom(t)) {
- args[i] = mailboxExecutor;
- } else if (t.isPrimitive()) {
- if (t == boolean.class) {
- args[i] = false;
- } else if (t == char.class) {
- args[i] = '\0';
- } else if (t == byte.class) {
- args[i] = (byte) 0;
- } else if (t == short.class) {
- args[i] = (short) 0;
- } else if (t == int.class) {
- args[i] = 0;
- } else if (t == long.class) {
- args[i] = 0L;
- } else if (t == float.class) {
- args[i] = 0.0f;
- } else if (t == double.class) {
- args[i] = 0.0d;
- }
- } else {
- // Best effort: pass null for any extra, unknown parameters.
- args[i] = null;
- }
+ Constructor<?> constructor =
+ flinkWriterClass.getDeclaredConstructor(
+ Sink.class, ProcessingTimeService.class, MailboxExecutor.class);
+ constructor.setAccessible(true);
+ return constructor.newInstance(sink, processingTimeService, mailboxExecutor);
+ } catch (Exception ignore) {
+ try {
+ Class<?> flinkWriterClass =
+ getRuntimeContext()
+ .getUserCodeClassLoader()
+ .loadClass(
+ "org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator");
+ Constructor<?> constructor =
+ flinkWriterClass.getDeclaredConstructor(
+ StreamOperatorParameters.class,
+ Sink.class,
+ ProcessingTimeService.class,
+ MailboxExecutor.class);
+ constructor.setAccessible(true);
+ return constructor.newInstance(null, sink, processingTimeService, mailboxExecutor);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to create SinkWriterOperator in Flink", e);
}
- return target.newInstance(args);
- } catch (Exception e) {
- throw new RuntimeException("Failed to create SinkWriterOperator in Flink", e);
}
}
@@ -262,36 +238,4 @@ public class BatchDataSinkWriterOperator<CommT>
private <T> T getFlinkWriterOperator() {
return (T) flinkWriterOperator;
}
-
- private void invokeFlinkWriterOperatorMethod(
- String methodName, Class<?>[] parameterTypes, Object... args) {
- try {
- Method m = findMethod(flinkWriterOperator.getClass(), methodName, parameterTypes);
- if (m == null) {
- // Method does not exist in this Flink version (for example, open() signature
- // changed); ignore for compatibility.
- return;
- }
- m.setAccessible(true);
- m.invoke(flinkWriterOperator, args);
- } catch (Exception e) {
- throw new RuntimeException(
- "Failed to invoke method "
- + methodName
- + " on wrapped flink writer operator "
- + flinkWriterOperator.getClass().getName(),
- e);
- }
- }
-
- private static Method findMethod(Class<?> clazz, String methodName, Class<?>[] parameterTypes) {
- while (clazz != null) {
- try {
- return clazz.getDeclaredMethod(methodName, parameterTypes);
- } catch (NoSuchMethodException e) {
- clazz = clazz.getSuperclass();
- }
- }
- return null;
- }
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkFunctionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkFunctionOperator.java
index 505f131e1..4286b3548 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkFunctionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkFunctionOperator.java
@@ -26,12 +26,10 @@ import org.apache.flink.cdc.common.event.SchemaChangeEventType;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.runtime.operators.sink.exception.SinkWrapperException;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.state.StateInitializationContext;
import org.apache.flink.streaming.api.functions.sink.SinkFunction;
import org.apache.flink.streaming.api.graph.StreamConfig;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.Output;
import org.apache.flink.streaming.api.operators.StreamSink;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
@@ -63,7 +61,6 @@ public class DataSinkFunctionOperator extends StreamSink<Event> {
super(userFunction);
this.schemaOperatorID = schemaOperatorID;
processedTableIds = new HashSet<>();
- FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
}
@Override
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
index 79261bc43..0c45b57c2 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
@@ -28,6 +28,7 @@ import org.apache.flink.cdc.common.event.FlushEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEventType;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.sink.exception.SinkWrapperException;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.state.StateInitializationContext;
@@ -36,17 +37,16 @@ import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
import org.apache.flink.streaming.api.graph.StreamConfig;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
import org.apache.flink.streaming.api.watermark.Watermark;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
import org.apache.flink.streaming.runtime.tasks.StreamTask;
import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
import java.lang.reflect.Constructor;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
@@ -65,11 +65,10 @@ import java.util.Set;
* @param <CommT> the type of the committable (to send to downstream operators)
*/
@Internal
-public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>
+public class DataSinkWriterOperator<CommT>
+ extends AbstractStreamOperatorAdapter<CommittableMessage<CommT>>
implements OneInputStreamOperator<Event, CommittableMessage<CommT>>, BoundedOneInput {
- private static final Logger LOG = LoggerFactory.getLogger(DataSinkWriterOperator.class);
-
private SchemaEvolutionClient schemaEvolutionClient;
private final OperatorID schemaOperatorID;
@@ -102,6 +101,7 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
this.mailboxExecutor = mailboxExecutor;
this.schemaOperatorID = schemaOperatorID;
this.processedTableIds = new HashSet<>();
+ this.chainingStrategy = ChainingStrategy.ALWAYS;
}
@Override
@@ -111,14 +111,7 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
Output<StreamRecord<CommittableMessage<CommT>>> output) {
super.setup(containingTask, config, output);
flinkWriterOperator = createFlinkWriterOperator();
- // Call protected AbstractStreamOperator.setup(...) on the wrapped operator via reflection
- // to avoid JVM verifier issues with protected access across packages.
- invokeFlinkWriterOperatorMethod(
- "setup",
- new Class<?>[] {StreamTask.class, StreamConfig.class, Output.class},
- containingTask,
- config,
- output);
+ invokeSetup(flinkWriterOperator, containingTask, config, output);
schemaEvolutionClient =
new SchemaEvolutionClient(
containingTask.getEnvironment().getOperatorCoordinatorEventGateway(),
@@ -127,109 +120,36 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
@Override
public void open() throws Exception {
- invokeFlinkWriterOperatorMethod("open", new Class<?>[0]);
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator().open();
copySinkWriter = getFieldValue("sinkWriter");
- ensureEmitDownstream();
- }
-
- /**
- * Flink 2.2 {@code SinkWriterOperator} sets {@code emitDownstream = sink instanceof
- * SupportsCommitter}. Sinks like Paimon implement the older {@code TwoPhaseCommittingSink}
- * rather than Flink 2.x {@code SupportsCommitter}, so committables are silently discarded. This
- * method forces the flag to {@code true} and fills in the committable serializer so that the
- * wrapped operator properly emits committables to the downstream committer.
- *
- * <p>This is only applied when the sink actually supports committing (implements either {@code
- * TwoPhaseCommittingSink} or {@code SupportsCommitter}). For sinks that do not support
- * committing, forcing {@code emitDownstream = true} would cause a {@link ClassCastException}
- * when the wrapped operator tries to cast the writer to {@code CommittingSinkWriter}.
- */
- private void ensureEmitDownstream() {
- if (!sinkSupportsCommitting()) {
- return;
- }
- try {
- Field emitField = findField(flinkWriterOperator.getClass(), "emitDownstream");
- if (emitField == null) {
- return;
- }
- emitField.setAccessible(true);
- if (!emitField.getBoolean(flinkWriterOperator)) {
- emitField.setBoolean(flinkWriterOperator, true);
-
- Field serField = findField(flinkWriterOperator.getClass(), "committableSerializer");
- if (serField != null) {
- serField.setAccessible(true);
- if (serField.get(flinkWriterOperator) == null) {
- try {
- Method getSerializer =
- sink.getClass().getMethod("getCommittableSerializer");
- getSerializer.setAccessible(true);
- serField.set(flinkWriterOperator, getSerializer.invoke(sink));
- } catch (NoSuchMethodException ignored) {
- }
- }
- }
- }
- } catch (Exception e) {
- LOG.warn("Could not force emitDownstream on wrapped SinkWriterOperator", e);
- }
- }
-
- private boolean sinkSupportsCommitting() {
- try {
- Class<?> tpc =
- Class.forName("org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink");
- if (tpc.isInstance(sink)) {
- return true;
- }
- } catch (ClassNotFoundException ignored) {
- }
- try {
- Class<?> sc = Class.forName("org.apache.flink.api.connector.sink2.SupportsCommitter");
- if (sc.isInstance(sink)) {
- return true;
- }
- } catch (ClassNotFoundException ignored) {
- }
- return false;
- }
-
- private static Field findField(Class<?> clazz, String name) {
- while (clazz != null) {
- try {
- return clazz.getDeclaredField(name);
- } catch (NoSuchFieldException e) {
- clazz = clazz.getSuperclass();
- }
- }
- return null;
}
@Override
public void initializeState(StateInitializationContext context) throws Exception {
- schemaEvolutionClient.registerSubtask(getSubtaskIndexCompat());
- invokeFlinkWriterOperatorMethod(
- "initializeState", new Class<?>[] {StateInitializationContext.class}, context);
+ schemaEvolutionClient.registerSubtask(
+ getRuntimeContext().getTaskInfo().getIndexOfThisSubtask());
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .initializeState(context);
}
@Override
public void snapshotState(StateSnapshotContext context) throws Exception {
- invokeFlinkWriterOperatorMethod(
- "snapshotState", new Class<?>[] {StateSnapshotContext.class}, context);
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .snapshotState(context);
}
@Override
public void processWatermark(Watermark mark) throws Exception {
super.processWatermark(mark);
- invokeFlinkWriterOperatorMethod("processWatermark", new Class<?>[] {Watermark.class}, mark);
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .processWatermark(mark);
}
@Override
public void processWatermarkStatus(WatermarkStatus watermarkStatus) throws Exception {
super.processWatermarkStatus(watermarkStatus);
- invokeFlinkWriterOperatorMethod(
- "processWatermarkStatus", new Class<?>[] {WatermarkStatus.class}, watermarkStatus);
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .processWatermarkStatus(watermarkStatus);
}
@Override
@@ -272,18 +192,19 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
@Override
public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
- invokeFlinkWriterOperatorMethod(
- "prepareSnapshotPreBarrier", new Class<?>[] {long.class}, checkpointId);
+ this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .prepareSnapshotPreBarrier(checkpointId);
}
@Override
public void close() throws Exception {
- invokeFlinkWriterOperatorMethod("close", new Class<?>[0]);
+ this.<OneInputStreamOperator<Event, CommittableMessage<CommT>>>getFlinkWriterOperator()
+ .close();
}
@Override
public void endInput() throws Exception {
- invokeFlinkWriterOperatorMethod("endInput", new Class<?>[0]);
+ this.<BoundedOneInput>getFlinkWriterOperator().endInput();
}
// ----------------------------- Helper functions -------------------------------
@@ -306,7 +227,8 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
});
}
schemaEvolutionClient.notifyFlushSuccess(
- getSubtaskIndexCompat(), event.getSourceSubTaskId());
+ getRuntimeContext().getTaskInfo().getIndexOfThisSubtask(),
+ event.getSourceSubTaskId());
}
private void emitLatestSchema(TableId tableId) throws Exception {
@@ -326,6 +248,22 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
// -------------------------- Reflection helper functions --------------------------
+ private void invokeSetup(
+ Object operator,
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<?> output) {
+ try {
+ Method setupMethod =
+ AbstractStreamOperator.class.getDeclaredMethod(
+ "setup", StreamTask.class, StreamConfig.class, Output.class);
+ setupMethod.setAccessible(true);
+ setupMethod.invoke(operator, containingTask, config, output);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to invoke setup on flinkWriterOperator", e);
+ }
+ }
+
private Object createFlinkWriterOperator() {
try {
Class<?> flinkWriterClass =
@@ -333,103 +271,29 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
.getUserCodeClassLoader()
.loadClass(
"org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator");
-
- // Try to find a constructor whose first three parameters are compatible with
- // (Sink, ProcessingTimeService, MailboxExecutor). This makes the code resilient
- // against Flink 2.x adding extra parameters to the constructor.
- Constructor<?> target = null;
- for (Constructor<?> c : flinkWriterClass.getDeclaredConstructors()) {
- Class<?>[] p = c.getParameterTypes();
- if (p.length >= 3
- && Sink.class.isAssignableFrom(p[0])
- && ProcessingTimeService.class.isAssignableFrom(p[1])
- && MailboxExecutor.class.isAssignableFrom(p[2])) {
- target = c;
- break;
- }
- }
-
- if (target == null) {
- // Fallback: use the first declared constructor and best-effort argument mapping
- // below. This covers Flink 2.2 where the constructor signature may have changed.
- Constructor<?>[] all = flinkWriterClass.getDeclaredConstructors();
- if (all.length == 0) {
- throw new RuntimeException(
- "No constructors found on SinkWriterOperator in Flink runtime");
- }
- target = all[0];
- }
-
- target.setAccessible(true);
- Class<?>[] paramTypes = target.getParameterTypes();
- Object[] args = new Object[paramTypes.length];
- for (int i = 0; i < paramTypes.length; i++) {
- Class<?> t = paramTypes[i];
- if (Sink.class.isAssignableFrom(t)) {
- args[i] = sink;
- } else if (ProcessingTimeService.class.isAssignableFrom(t)) {
- args[i] = processingTimeService;
- } else if (MailboxExecutor.class.isAssignableFrom(t)) {
- args[i] = mailboxExecutor;
- } else if (t.isPrimitive()) {
- if (t == boolean.class) {
- args[i] = false;
- } else if (t == char.class) {
- args[i] = '\0';
- } else if (t == byte.class) {
- args[i] = (byte) 0;
- } else if (t == short.class) {
- args[i] = (short) 0;
- } else if (t == int.class) {
- args[i] = 0;
- } else if (t == long.class) {
- args[i] = 0L;
- } else if (t == float.class) {
- args[i] = 0.0f;
- } else if (t == double.class) {
- args[i] = 0.0d;
- }
- } else {
- // Best effort: pass null for any extra, unknown parameters.
- args[i] = null;
- }
- }
- return target.newInstance(args);
- } catch (Exception e) {
- throw new RuntimeException("Failed to create SinkWriterOperator in Flink", e);
- }
- }
-
- /**
- * Obtains the subtask index in a way that is compatible with both Flink 1.x and 2.x.
- *
- * <p>Flink 2.x removed {@code StreamingRuntimeContext.getIndexOfThisSubtask()}. The correct
- * replacement is {@code getRuntimeContext().getTaskInfo().getIndexOfThisSubtask()}.
- *
- * <p><b>IMPORTANT:</b> Returning a wrong (fixed) index here causes ALL sink subtasks to report
- * the same ID to SchemaCoordinator. The coordinator then waits forever for the missing subtask
- * IDs, causing a 3-minute timeout in SchemaOperator.
- */
- private int getSubtaskIndexCompat() {
- try {
- Object ctx = getRuntimeContext();
- // Flink 2.x: getTaskInfo().getIndexOfThisSubtask()
+ Constructor<?> constructor =
+ flinkWriterClass.getDeclaredConstructor(
+ Sink.class, ProcessingTimeService.class, MailboxExecutor.class);
+ constructor.setAccessible(true);
+ return constructor.newInstance(sink, processingTimeService, mailboxExecutor);
+ } catch (Exception ignore) {
try {
- Method getTaskInfo = ctx.getClass().getMethod("getTaskInfo");
- getTaskInfo.setAccessible(true);
- Object taskInfo = getTaskInfo.invoke(ctx);
- Method getIndex = taskInfo.getClass().getMethod("getIndexOfThisSubtask");
- getIndex.setAccessible(true);
- return (Integer) getIndex.invoke(taskInfo);
- } catch (NoSuchMethodException ignored) {
- // fall through to Flink 1.x path
+ Class<?> flinkWriterClass =
+ getRuntimeContext()
+ .getUserCodeClassLoader()
+ .loadClass(
+ "org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator");
+ Constructor<?> constructor =
+ flinkWriterClass.getDeclaredConstructor(
+ StreamOperatorParameters.class,
+ Sink.class,
+ ProcessingTimeService.class,
+ MailboxExecutor.class);
+ constructor.setAccessible(true);
+ return constructor.newInstance(null, sink, processingTimeService, mailboxExecutor);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to create SinkWriterOperator in Flink", e);
}
- // Flink 1.x: direct getIndexOfThisSubtask()
- Method m = ctx.getClass().getMethod("getIndexOfThisSubtask");
- m.setAccessible(true);
- return (Integer) m.invoke(ctx);
- } catch (Exception e) {
- throw new RuntimeException("Failed to obtain subtask index from RuntimeContext", e);
}
}
@@ -459,36 +323,4 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
private <T> T getFlinkWriterOperator() {
return (T) flinkWriterOperator;
}
-
- private void invokeFlinkWriterOperatorMethod(
- String methodName, Class<?>[] parameterTypes, Object... args) {
- try {
- Method m = findMethod(flinkWriterOperator.getClass(), methodName, parameterTypes);
- if (m == null) {
- // Method does not exist in this Flink version (for example, open() signature
- // changed); ignore for compatibility.
- return;
- }
- m.setAccessible(true);
- m.invoke(flinkWriterOperator, args);
- } catch (Exception e) {
- throw new RuntimeException(
- "Failed to invoke method "
- + methodName
- + " on wrapped flink writer operator "
- + flinkWriterOperator.getClass().getName(),
- e);
- }
- }
-
- private static Method findMethod(Class<?> clazz, String methodName, Class<?>[] parameterTypes) {
- while (clazz != null) {
- try {
- return clazz.getDeclaredMethod(methodName, parameterTypes);
- } catch (NoSuchMethodException e) {
- clazz = clazz.getSuperclass();
- }
- }
- return null;
- }
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java
index ab64f9c2b..278518f97 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java
@@ -17,25 +17,23 @@
package org.apache.flink.cdc.runtime.operators.sink;
-import org.apache.flink.api.common.operators.MailboxExecutor;
import org.apache.flink.api.connector.sink2.Sink;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.event.Event;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
import org.apache.flink.streaming.api.operators.AbstractStreamOperatorFactory;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperatorFactory;
import org.apache.flink.streaming.api.operators.StreamOperator;
import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
-
-import java.lang.reflect.Method;
+import org.apache.flink.streaming.api.operators.YieldingOperatorFactory;
/** Operator factory for {@link DataSinkWriterOperator}. */
@Internal
public class DataSinkWriterOperatorFactory<CommT>
extends AbstractStreamOperatorFactory<CommittableMessage<CommT>>
- implements OneInputStreamOperatorFactory<Event, CommittableMessage<CommT>> {
+ implements OneInputStreamOperatorFactory<Event, CommittableMessage<CommT>>,
+ YieldingOperatorFactory<CommittableMessage<CommT>> {
private final Sink<Event> sink;
private final boolean isBounded;
@@ -43,7 +41,6 @@ public class DataSinkWriterOperatorFactory<CommT>
public DataSinkWriterOperatorFactory(
Sink<Event> sink, boolean isBounded, OperatorID schemaOperatorID) {
- setChainingStrategy(ChainingStrategy.ALWAYS);
this.sink = sink;
this.isBounded = isBounded;
this.schemaOperatorID = schemaOperatorID;
@@ -53,10 +50,11 @@ public class DataSinkWriterOperatorFactory<CommT>
@Override
public <T extends StreamOperator<CommittableMessage<CommT>>> T createStreamOperator(
StreamOperatorParameters<CommittableMessage<CommT>> parameters) {
- MailboxExecutor mailboxExecutor = getMailboxExecutor(parameters);
+
if (isBounded) {
BatchDataSinkWriterOperator<CommT> writerOperator =
- new BatchDataSinkWriterOperator<>(sink, processingTimeService, mailboxExecutor);
+ new BatchDataSinkWriterOperator<>(
+ sink, processingTimeService, getMailboxExecutor());
writerOperator.setup(
parameters.getContainingTask(),
parameters.getStreamConfig(),
@@ -65,7 +63,7 @@ public class DataSinkWriterOperatorFactory<CommT>
}
DataSinkWriterOperator<CommT> writerOperator =
new DataSinkWriterOperator<>(
- sink, processingTimeService, mailboxExecutor, schemaOperatorID);
+ sink, processingTimeService, getMailboxExecutor(), schemaOperatorID);
writerOperator.setup(
parameters.getContainingTask(),
parameters.getStreamConfig(),
@@ -73,41 +71,6 @@ public class DataSinkWriterOperatorFactory<CommT>
return (T) writerOperator;
}
- /**
- * Obtains the {@link MailboxExecutor} in a way compatible with both Flink 1.19 and 1.20+.
- *
- * <p>Flink 1.20+ added {@code StreamOperatorParameters.getMailboxExecutor()}. In Flink 1.19 and
- * earlier, the executor must be obtained via {@code
- * StreamTask.getMailboxExecutorFactory().createExecutor(chainIndex)}.
- */
- private MailboxExecutor getMailboxExecutor(
- StreamOperatorParameters<CommittableMessage<CommT>> parameters) {
- // Try Flink 1.20+ / 2.x API: StreamOperatorParameters.getMailboxExecutor()
- try {
- Method m = parameters.getClass().getMethod("getMailboxExecutor");
- return (MailboxExecutor) m.invoke(parameters);
- } catch (NoSuchMethodException ignored) {
- // Fall through to Flink 1.19 path
- } catch (Exception e) {
- throw new RuntimeException(
- "Failed to invoke getMailboxExecutor on StreamOperatorParameters", e);
- }
-
- // Flink 1.19 and earlier: obtain from containingTask.getMailboxExecutorFactory()
- try {
- Object containingTask = parameters.getContainingTask();
- Method getFactory = containingTask.getClass().getMethod("getMailboxExecutorFactory");
- Object factory = getFactory.invoke(containingTask);
- int chainIndex = parameters.getStreamConfig().getChainIndex();
- Method createExecutor = factory.getClass().getMethod("createExecutor", int.class);
- return (MailboxExecutor) createExecutor.invoke(factory, chainIndex);
- } catch (Exception e) {
- throw new RuntimeException(
- "Failed to obtain MailboxExecutor from StreamTask for Flink 1.19 compatibility",
- e);
- }
- }
-
@Override
public Class<? extends StreamOperator> getStreamOperatorClass(ClassLoader classLoader) {
if (isBounded) {
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperator.java
index ae0b48d57..36e348e09 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperator.java
@@ -34,12 +34,12 @@ import org.apache.flink.cdc.common.schema.Selectors;
import org.apache.flink.cdc.common.udf.UserDefinedFunctionContext;
import org.apache.flink.cdc.common.utils.SchemaMergingUtils;
import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.transform.converter.PostTransformConverters;
import org.apache.flink.cdc.runtime.operators.transform.exceptions.TransformException;
import org.apache.flink.cdc.runtime.parser.TransformParser;
import org.apache.flink.cdc.runtime.typeutils.BinaryInternalObjectConverter;
import org.apache.flink.cdc.runtime.typeutils.BinaryRecordDataGenerator;
-import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
@@ -64,7 +64,7 @@ import static org.apache.flink.cdc.common.utils.Preconditions.checkNotNull;
* A data process function that performs column filtering, calculated column evaluation & final
* projection.
*/
-public class PostTransformOperator extends AbstractStreamOperator<Event>
+public class PostTransformOperator extends AbstractStreamOperatorAdapter<Event>
implements OneInputStreamOperator<Event, Event>, Serializable {
private static final long serialVersionUID = 1L;
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java
index fc3509196..03909e3a7 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java
@@ -33,13 +33,12 @@ import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.schema.Selectors;
import org.apache.flink.cdc.common.utils.Preconditions;
import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.transform.exceptions.TransformException;
import org.apache.flink.cdc.runtime.parser.TransformParser;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.state.StateInitializationContext;
import org.apache.flink.runtime.state.StateSnapshotContext;
import org.apache.flink.streaming.api.graph.StreamConfig;
-import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
@@ -61,7 +60,7 @@ import java.util.stream.Collectors;
/**
* A data process function that filters out columns which aren't (directly & indirectly) referenced.
*/
-public class PreTransformOperator extends AbstractStreamOperator<Event>
+public class PreTransformOperator extends AbstractStreamOperatorAdapter<Event>
implements OneInputStreamOperator<Event, Event>, Serializable {
private static final long serialVersionUID = 1L;
@@ -86,7 +85,7 @@ public class PreTransformOperator extends AbstractStreamOperator<Event>
this.preTransformChangeInfoMap = new HashMap<>();
this.preTransformProcessorMap = new HashMap<>();
this.schemaMetadataTransformers = new ArrayList<>();
- FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
+ this.chainingStrategy = ChainingStrategy.ALWAYS;
this.transformRules = transformRules;
this.udfFunctions = udfFunctions;
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/BatchRegularPrePartitionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/BatchRegularPrePartitionOperator.java
index e273bd694..1a2a69798 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/BatchRegularPrePartitionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/BatchRegularPrePartitionOperator.java
@@ -25,11 +25,10 @@ import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.function.HashFunction;
import org.apache.flink.cdc.common.function.HashFunctionProvider;
import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator;
import org.apache.flink.cdc.runtime.serializer.event.EventSerializer;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.state.StateSnapshotContext;
-import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
@@ -45,7 +44,8 @@ import java.util.Optional;
* regular topology in batch mode.
*/
@Internal
-public class BatchRegularPrePartitionOperator extends AbstractStreamOperator<PartitioningEvent>
+public class BatchRegularPrePartitionOperator
+ extends AbstractStreamOperatorAdapter<PartitioningEvent>
implements OneInputStreamOperator<Event, PartitioningEvent>, Serializable {
private static final long serialVersionUID = 1L;
@@ -59,7 +59,7 @@ public class BatchRegularPrePartitionOperator extends AbstractStreamOperator<Par
public BatchRegularPrePartitionOperator(
int downstreamParallelism, HashFunctionProvider<DataChangeEvent> hashFunctionProvider) {
- FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
+ this.chainingStrategy = ChainingStrategy.ALWAYS;
this.downstreamParallelism = downstreamParallelism;
this.hashFunctionProvider = hashFunctionProvider;
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/DistributedPrePartitionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/DistributedPrePartitionOperator.java
index 8b8fcf322..4750c685a 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/DistributedPrePartitionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/DistributedPrePartitionOperator.java
@@ -26,11 +26,10 @@ import org.apache.flink.cdc.common.function.HashFunction;
import org.apache.flink.cdc.common.function.HashFunctionProvider;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator;
import org.apache.flink.cdc.runtime.serializer.event.EventSerializer;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.state.StateSnapshotContext;
-import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
@@ -41,7 +40,8 @@ import java.util.Map;
/** Operator for processing events from upstream before flowing to {@link SchemaOperator}. */
@Internal
-public class DistributedPrePartitionOperator extends AbstractStreamOperator<PartitioningEvent>
+public class DistributedPrePartitionOperator
+ extends AbstractStreamOperatorAdapter<PartitioningEvent>
implements OneInputStreamOperator<Event, PartitioningEvent>, Serializable {
private static final long serialVersionUID = 1L;
@@ -56,7 +56,7 @@ public class DistributedPrePartitionOperator extends AbstractStreamOperator<Part
public DistributedPrePartitionOperator(
int downstreamParallelism, HashFunctionProvider<DataChangeEvent> hashFunctionProvider) {
- FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
+ this.chainingStrategy = ChainingStrategy.ALWAYS;
this.downstreamParallelism = downstreamParallelism;
this.hashFunctionProvider = hashFunctionProvider;
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/RegularPrePartitionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/RegularPrePartitionOperator.java
index d65e3d755..a5d81bfb1 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/RegularPrePartitionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/RegularPrePartitionOperator.java
@@ -26,14 +26,13 @@ import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.function.HashFunction;
import org.apache.flink.cdc.common.function.HashFunctionProvider;
import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator;
import org.apache.flink.cdc.runtime.operators.sink.SchemaEvolutionClient;
import org.apache.flink.cdc.runtime.serializer.event.EventSerializer;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.runtime.state.StateSnapshotContext;
-import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
@@ -51,7 +50,7 @@ import java.util.Optional;
* regular topology.
*/
@Internal
-public class RegularPrePartitionOperator extends AbstractStreamOperator<PartitioningEvent>
+public class RegularPrePartitionOperator extends AbstractStreamOperatorAdapter<PartitioningEvent>
implements OneInputStreamOperator<Event, PartitioningEvent>, Serializable {
private static final long serialVersionUID = 1L;
@@ -68,7 +67,7 @@ public class RegularPrePartitionOperator extends AbstractStreamOperator<Partitio
OperatorID schemaOperatorId,
int downstreamParallelism,
HashFunctionProvider<DataChangeEvent> hashFunctionProvider) {
- FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
+ this.chainingStrategy = ChainingStrategy.ALWAYS;
this.schemaOperatorId = schemaOperatorId;
this.downstreamParallelism = downstreamParallelism;
this.hashFunctionProvider = hashFunctionProvider;
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/EnumSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/EnumSerializer.java
index c78281089..9987e06aa 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/EnumSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/EnumSerializer.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.runtime.serializer;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotAdapter;
import org.apache.flink.cdc.common.annotation.VisibleForTesting;
import org.apache.flink.core.memory.DataInputView;
import org.apache.flink.core.memory.DataOutputView;
@@ -176,7 +177,7 @@ public final class EnumSerializer<T extends Enum<T>> extends TypeSerializer<T> {
/** {@link TypeSerializerSnapshot} for {@link EnumSerializer}. */
public static final class EnumSerializerSnapshot<T extends Enum<T>>
- implements TypeSerializerSnapshot<T> {
+ implements TypeSerializerSnapshotAdapter<T> {
private static final int CURRENT_VERSION = 3;
private T[] previousEnums;
@@ -238,8 +239,7 @@ public final class EnumSerializer<T extends Enum<T>> extends TypeSerializer<T> {
return new EnumSerializer<>(enumClass, previousEnums);
}
- // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
- // compatibility
+ @Override
public TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
TypeSerializer<T> newSerializer) {
if (!(newSerializer instanceof EnumSerializer)) {
@@ -270,32 +270,6 @@ public final class EnumSerializer<T extends Enum<T>> extends TypeSerializer<T> {
return TypeSerializerSchemaCompatibility.compatibleWithReconfiguredSerializer(
reconfiguredSerializer);
}
-
- // Flink 2.x new abstract method - no @Override so this also compiles against Flink 1.x
- public TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
- TypeSerializerSnapshot<T> oldSerializerSnapshot) {
- if (!(oldSerializerSnapshot instanceof EnumSerializerSnapshot)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- @SuppressWarnings("unchecked")
- EnumSerializerSnapshot<T> oldSnapshot =
- (EnumSerializerSnapshot<T>) oldSerializerSnapshot;
- if (!enumClass.equals(oldSnapshot.enumClass)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- if (Arrays.equals(oldSnapshot.previousEnums, previousEnums)) {
- return TypeSerializerSchemaCompatibility.compatibleAsIs();
- }
- Set<T> reconfiguredEnumSet =
- new LinkedHashSet<>(Arrays.asList(oldSnapshot.previousEnums));
- reconfiguredEnumSet.addAll(Arrays.asList(previousEnums));
- @SuppressWarnings("unchecked")
- T[] reconfiguredEnums =
- reconfiguredEnumSet.toArray(
- (T[]) Array.newInstance(enumClass, reconfiguredEnumSet.size()));
- return TypeSerializerSchemaCompatibility.compatibleWithReconfiguredSerializer(
- new EnumSerializer<>(enumClass, reconfiguredEnums));
- }
}
// --------------------------------------------------------------------------------------------
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NestedSerializersSnapshotDelegate.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NestedSerializersSnapshotDelegate.java
index 8eedaeef1..c9a053bbf 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NestedSerializersSnapshotDelegate.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NestedSerializersSnapshotDelegate.java
@@ -184,14 +184,7 @@ public class NestedSerializersSnapshotDelegate {
// Utilities
// ------------------------------------------------------------------------
- /**
- * Utility method to conjure up a new scope for the generic parameters.
- *
- * <p>In Flink 1.x, TypeSerializerSnapshot.resolveSchemaCompatibility(TypeSerializer) is the
- * abstract method. In Flink 2.x, it was replaced by
- * resolveSchemaCompatibility(TypeSerializerSnapshot). This method uses reflection to call the
- * right variant at runtime.
- */
+ /** Utility method to conjure up a new scope for the generic parameters. */
@SuppressWarnings("unchecked")
private static <E> TypeSerializerSchemaCompatibility<E> resolveCompatibility(
TypeSerializer<?> serializer, TypeSerializerSnapshot<?> snapshot) {
@@ -199,32 +192,7 @@ public class NestedSerializersSnapshotDelegate {
TypeSerializer<E> typedSerializer = (TypeSerializer<E>) serializer;
TypeSerializerSnapshot<E> typedSnapshot = (TypeSerializerSnapshot<E>) snapshot;
- // Try Flink 1.x approach: oldSnapshot.resolveSchemaCompatibility(newSerializer)
- try {
- java.lang.reflect.Method m =
- typedSnapshot
- .getClass()
- .getMethod("resolveSchemaCompatibility", TypeSerializer.class);
- return (TypeSerializerSchemaCompatibility<E>) m.invoke(typedSnapshot, typedSerializer);
- } catch (NoSuchMethodException ignored) {
- // fall through to Flink 2.x approach
- } catch (Exception e) {
- throw new RuntimeException(
- "Failed to resolve schema compatibility (Flink 1.x path)", e);
- }
-
- // Flink 2.x approach: newSnapshot.resolveSchemaCompatibility(oldSnapshot)
- try {
- TypeSerializerSnapshot<E> newSnapshot = typedSerializer.snapshotConfiguration();
- java.lang.reflect.Method m =
- newSnapshot
- .getClass()
- .getMethod("resolveSchemaCompatibility", TypeSerializerSnapshot.class);
- return (TypeSerializerSchemaCompatibility<E>) m.invoke(newSnapshot, typedSnapshot);
- } catch (Exception e) {
- throw new RuntimeException(
- "Failed to resolve schema compatibility (Flink 2.x path)", e);
- }
+ return typedSnapshot.resolveSchemaCompatibility(typedSerializer.snapshotConfiguration());
}
private static TypeSerializer<?>[] snapshotsToRestoreSerializers(
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NullableSerializerWrapper.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NullableSerializerWrapper.java
index 33eb92742..853754b30 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NullableSerializerWrapper.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NullableSerializerWrapper.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.runtime.serializer;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotAdapter;
import org.apache.flink.cdc.common.types.utils.runtime.DataInputViewStream;
import org.apache.flink.cdc.common.types.utils.runtime.DataOutputViewStream;
import org.apache.flink.cdc.common.utils.InstantiationUtil;
@@ -130,7 +131,7 @@ public class NullableSerializerWrapper<T> extends TypeSerializer<T> {
/** Serializer configuration snapshot for compatibility and format evolution. */
@SuppressWarnings("WeakerAccess")
public static final class NullableSerializerWrapperSnapshot<T>
- implements TypeSerializerSnapshot<T> {
+ implements TypeSerializerSnapshotAdapter<T> {
private static final int CURRENT_VERSION = 1;
@@ -172,8 +173,7 @@ public class NullableSerializerWrapper<T> extends TypeSerializer<T> {
return new NullableSerializerWrapper<>(innerSerializer);
}
- // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
- // compatibility
+ @Override
public TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
TypeSerializer<T> newSerializer) {
if (!(newSerializer instanceof NullableSerializerWrapper)) {
@@ -188,20 +188,5 @@ public class NullableSerializerWrapper<T> extends TypeSerializer<T> {
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
-
- // Flink 2.x new abstract method - no @Override so this also compiles against Flink 1.x
- @SuppressWarnings("unchecked")
- public TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
- TypeSerializerSnapshot<T> oldSerializerSnapshot) {
- if (!(oldSerializerSnapshot instanceof NullableSerializerWrapperSnapshot)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- NullableSerializerWrapperSnapshot<T> snapshot =
- (NullableSerializerWrapperSnapshot<T>) oldSerializerSnapshot;
- if (!innerSerializer.equals(snapshot.innerSerializer)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- return TypeSerializerSchemaCompatibility.compatibleAsIs();
- }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ArrayDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ArrayDataSerializer.java
index ecbc03d7e..d9f6987b6 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ArrayDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ArrayDataSerializer.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.runtime.serializer.data;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotAdapter;
import org.apache.flink.api.java.typeutils.runtime.DataInputViewStream;
import org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream;
import org.apache.flink.cdc.common.annotation.VisibleForTesting;
@@ -52,12 +53,12 @@ public class ArrayDataSerializer extends TypeSerializer<ArrayData> {
private transient BinaryArrayWriter reuseWriter;
public ArrayDataSerializer(DataType eleType) {
- this(eleType, InternalSerializers.create(eleType));
+ this(eleType, new NullableSerializerWrapper<>(InternalSerializers.create(eleType)));
}
private ArrayDataSerializer(DataType eleType, TypeSerializer<Object> eleSer) {
this.eleType = eleType;
- this.eleSer = new NullableSerializerWrapper<>(eleSer);
+ this.eleSer = eleSer;
this.elementGetter = ArrayData.createElementGetter(eleType);
}
@@ -218,7 +219,7 @@ public class ArrayDataSerializer extends TypeSerializer<ArrayData> {
/** {@link TypeSerializerSnapshot} for {@link ArrayDataSerializer}. */
public static final class ArrayDataSerializerSnapshot
- implements TypeSerializerSnapshot<ArrayData> {
+ implements TypeSerializerSnapshotAdapter<ArrayData> {
private static final int CURRENT_VERSION = 3;
private DataType previousType;
@@ -265,8 +266,7 @@ public class ArrayDataSerializer extends TypeSerializer<ArrayData> {
return new ArrayDataSerializer(previousType, previousEleSer);
}
- // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
- // compatibility
+ @Override
public TypeSerializerSchemaCompatibility<ArrayData> resolveSchemaCompatibility(
TypeSerializer<ArrayData> newSerializer) {
if (!(newSerializer instanceof ArrayDataSerializer)) {
@@ -281,20 +281,5 @@ public class ArrayDataSerializer extends TypeSerializer<ArrayData> {
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
-
- // Flink 2.x new abstract method - no @Override so this also compiles against Flink 1.x
- public TypeSerializerSchemaCompatibility<ArrayData> resolveSchemaCompatibility(
- TypeSerializerSnapshot<ArrayData> oldSerializerSnapshot) {
- if (!(oldSerializerSnapshot instanceof ArrayDataSerializerSnapshot)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- ArrayDataSerializerSnapshot snapshot =
- (ArrayDataSerializerSnapshot) oldSerializerSnapshot;
- if (!previousType.equals(snapshot.previousType)
- || !previousEleSer.equals(snapshot.previousEleSer)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- return TypeSerializerSchemaCompatibility.compatibleAsIs();
- }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/DecimalDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/DecimalDataSerializer.java
index 97d2e3561..7b799240c 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/DecimalDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/DecimalDataSerializer.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.runtime.serializer.data;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotAdapter;
import org.apache.flink.cdc.common.data.DecimalData;
import org.apache.flink.core.memory.DataInputView;
import org.apache.flink.core.memory.DataOutputView;
@@ -138,7 +139,7 @@ public final class DecimalDataSerializer extends TypeSerializer<DecimalData> {
/** {@link TypeSerializerSnapshot} for {@link DecimalDataSerializer}. */
public static final class DecimalSerializerSnapshot
- implements TypeSerializerSnapshot<DecimalData> {
+ implements TypeSerializerSnapshotAdapter<DecimalData> {
private static final int CURRENT_VERSION = 3;
@@ -178,8 +179,7 @@ public final class DecimalDataSerializer extends TypeSerializer<DecimalData> {
return new DecimalDataSerializer(previousPrecision, previousScale);
}
- // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
- // compatibility
+ @Override
public TypeSerializerSchemaCompatibility<DecimalData> resolveSchemaCompatibility(
TypeSerializer<DecimalData> newSerializer) {
if (!(newSerializer instanceof DecimalDataSerializer)) {
@@ -194,21 +194,5 @@ public final class DecimalDataSerializer extends TypeSerializer<DecimalData> {
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
-
- // Flink 2.x changed the abstract method to
- // resolveSchemaCompatibility(TypeSerializerSnapshot).
- // No @Override so this also compiles against Flink 1.x where the method does not exist.
- public TypeSerializerSchemaCompatibility<DecimalData> resolveSchemaCompatibility(
- TypeSerializerSnapshot<DecimalData> oldSerializerSnapshot) {
- if (!(oldSerializerSnapshot instanceof DecimalSerializerSnapshot)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- DecimalSerializerSnapshot snapshot = (DecimalSerializerSnapshot) oldSerializerSnapshot;
- if (previousPrecision != snapshot.previousPrecision
- || previousScale != snapshot.previousScale) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- return TypeSerializerSchemaCompatibility.compatibleAsIs();
- }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/LocalZonedTimestampDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/LocalZonedTimestampDataSerializer.java
index 987f20823..2743d157d 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/LocalZonedTimestampDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/LocalZonedTimestampDataSerializer.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.runtime.serializer.data;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotAdapter;
import org.apache.flink.cdc.common.data.LocalZonedTimestampData;
import org.apache.flink.cdc.common.data.TimestampData;
import org.apache.flink.core.memory.DataInputView;
@@ -142,7 +143,7 @@ public class LocalZonedTimestampDataSerializer extends TypeSerializer<LocalZoned
/** {@link TypeSerializerSnapshot} for {@link TimestampDataSerializer}. */
public static final class LocalZonedTimestampDataSerializerSnapshot
- implements TypeSerializerSnapshot<LocalZonedTimestampData> {
+ implements TypeSerializerSnapshotAdapter<LocalZonedTimestampData> {
private static final int CURRENT_VERSION = 1;
@@ -177,8 +178,7 @@ public class LocalZonedTimestampDataSerializer extends TypeSerializer<LocalZoned
return new LocalZonedTimestampDataSerializer(previousPrecision);
}
- // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
- // compatibility
+ @Override
public TypeSerializerSchemaCompatibility<LocalZonedTimestampData>
resolveSchemaCompatibility(TypeSerializer<LocalZonedTimestampData> newSerializer) {
if (!(newSerializer instanceof LocalZonedTimestampDataSerializer)) {
@@ -193,22 +193,5 @@ public class LocalZonedTimestampDataSerializer extends TypeSerializer<LocalZoned
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
-
- // Flink 2.x changed the abstract method to
- // resolveSchemaCompatibility(TypeSerializerSnapshot).
- // No @Override so this also compiles against Flink 1.x where the method does not exist.
- public TypeSerializerSchemaCompatibility<LocalZonedTimestampData>
- resolveSchemaCompatibility(
- TypeSerializerSnapshot<LocalZonedTimestampData> oldSerializerSnapshot) {
- if (!(oldSerializerSnapshot instanceof LocalZonedTimestampDataSerializerSnapshot)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- LocalZonedTimestampDataSerializerSnapshot snapshot =
- (LocalZonedTimestampDataSerializerSnapshot) oldSerializerSnapshot;
- if (previousPrecision != snapshot.previousPrecision) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- return TypeSerializerSchemaCompatibility.compatibleAsIs();
- }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/MapDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/MapDataSerializer.java
index 8b2e19cde..0c5050962 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/MapDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/MapDataSerializer.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.runtime.serializer.data;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotAdapter;
import org.apache.flink.api.java.typeutils.runtime.DataInputViewStream;
import org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream;
import org.apache.flink.cdc.common.annotation.Internal;
@@ -248,7 +249,8 @@ public class MapDataSerializer extends TypeSerializer<MapData> {
}
/** {@link TypeSerializerSnapshot} for {@link MapDataSerializer}. */
- public static final class MapDataSerializerSnapshot implements TypeSerializerSnapshot<MapData> {
+ public static final class MapDataSerializerSnapshot
+ implements TypeSerializerSnapshotAdapter<MapData> {
private static final int CURRENT_VERSION = 0;
private DataType keyType;
@@ -306,8 +308,7 @@ public class MapDataSerializer extends TypeSerializer<MapData> {
return new MapDataSerializer(keyType, valueType, keySerializer, valueSerializer);
}
- // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
- // compatibility
+ @Override
public TypeSerializerSchemaCompatibility<MapData> resolveSchemaCompatibility(
TypeSerializer<MapData> newSerializer) {
if (!(newSerializer instanceof MapDataSerializer)) {
@@ -324,21 +325,5 @@ public class MapDataSerializer extends TypeSerializer<MapData> {
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
-
- // Flink 2.x new abstract method - no @Override so this also compiles against Flink 1.x
- public TypeSerializerSchemaCompatibility<MapData> resolveSchemaCompatibility(
- TypeSerializerSnapshot<MapData> oldSerializerSnapshot) {
- if (!(oldSerializerSnapshot instanceof MapDataSerializerSnapshot)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- MapDataSerializerSnapshot snapshot = (MapDataSerializerSnapshot) oldSerializerSnapshot;
- if (!keyType.equals(snapshot.keyType)
- || !valueType.equals(snapshot.valueType)
- || !keySerializer.equals(snapshot.keySerializer)
- || !valueSerializer.equals(snapshot.valueSerializer)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- return TypeSerializerSchemaCompatibility.compatibleAsIs();
- }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/TimestampDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/TimestampDataSerializer.java
index 10dfc0606..25d63f3a7 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/TimestampDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/TimestampDataSerializer.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.runtime.serializer.data;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotAdapter;
import org.apache.flink.cdc.common.data.TimestampData;
import org.apache.flink.core.memory.DataInputView;
import org.apache.flink.core.memory.DataOutputView;
@@ -136,7 +137,7 @@ public class TimestampDataSerializer extends TypeSerializer<TimestampData> {
/** {@link TypeSerializerSnapshot} for {@link TimestampDataSerializer}. */
public static final class TimestampDataSerializerSnapshot
- implements TypeSerializerSnapshot<TimestampData> {
+ implements TypeSerializerSnapshotAdapter<TimestampData> {
private static final int CURRENT_VERSION = 1;
@@ -171,8 +172,7 @@ public class TimestampDataSerializer extends TypeSerializer<TimestampData> {
return new TimestampDataSerializer(previousPrecision);
}
- // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
- // compatibility
+ @Override
public TypeSerializerSchemaCompatibility<TimestampData> resolveSchemaCompatibility(
TypeSerializer<TimestampData> newSerializer) {
if (!(newSerializer instanceof TimestampDataSerializer)) {
@@ -187,21 +187,5 @@ public class TimestampDataSerializer extends TypeSerializer<TimestampData> {
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
-
- // Flink 2.x changed the abstract method to
- // resolveSchemaCompatibility(TypeSerializerSnapshot).
- // No @Override so this also compiles against Flink 1.x where the method does not exist.
- public TypeSerializerSchemaCompatibility<TimestampData> resolveSchemaCompatibility(
- TypeSerializerSnapshot<TimestampData> oldSerializerSnapshot) {
- if (!(oldSerializerSnapshot instanceof TimestampDataSerializerSnapshot)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- TimestampDataSerializerSnapshot snapshot =
- (TimestampDataSerializerSnapshot) oldSerializerSnapshot;
- if (previousPrecision != snapshot.previousPrecision) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- return TypeSerializerSchemaCompatibility.compatibleAsIs();
- }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ZonedTimestampDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ZonedTimestampDataSerializer.java
index 8676d903f..dc833e52a 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ZonedTimestampDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ZonedTimestampDataSerializer.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.runtime.serializer.data;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotAdapter;
import org.apache.flink.cdc.common.data.TimestampData;
import org.apache.flink.cdc.common.data.ZonedTimestampData;
import org.apache.flink.cdc.runtime.serializer.StringSerializer;
@@ -146,7 +147,7 @@ public class ZonedTimestampDataSerializer extends TypeSerializer<ZonedTimestampD
/** {@link TypeSerializerSnapshot} for {@link ZonedTimestampDataSerializer}. */
public static final class ZonedTimestampDataSerializerSnapshot
- implements TypeSerializerSnapshot<ZonedTimestampData> {
+ implements TypeSerializerSnapshotAdapter<ZonedTimestampData> {
private static final int CURRENT_VERSION = 1;
@@ -181,8 +182,7 @@ public class ZonedTimestampDataSerializer extends TypeSerializer<ZonedTimestampD
return new ZonedTimestampDataSerializer(previousPrecision);
}
- // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
- // compatibility
+ @Override
public TypeSerializerSchemaCompatibility<ZonedTimestampData> resolveSchemaCompatibility(
TypeSerializer<ZonedTimestampData> newSerializer) {
if (!(newSerializer instanceof ZonedTimestampDataSerializer)) {
@@ -197,19 +197,5 @@ public class ZonedTimestampDataSerializer extends TypeSerializer<ZonedTimestampD
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
-
- // Flink 2.x new abstract method - no @Override so this also compiles against Flink 1.x
- public TypeSerializerSchemaCompatibility<ZonedTimestampData> resolveSchemaCompatibility(
- TypeSerializerSnapshot<ZonedTimestampData> oldSerializerSnapshot) {
- if (!(oldSerializerSnapshot instanceof ZonedTimestampDataSerializerSnapshot)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- ZonedTimestampDataSerializerSnapshot snapshot =
- (ZonedTimestampDataSerializerSnapshot) oldSerializerSnapshot;
- if (previousPrecision != snapshot.previousPrecision) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
- return TypeSerializerSchemaCompatibility.compatibleAsIs();
- }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/utils/FlinkCompatibilityUtils.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/utils/FlinkCompatibilityUtils.java
deleted file mode 100644
index 974ad8641..000000000
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/utils/FlinkCompatibilityUtils.java
+++ /dev/null
@@ -1,153 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.runtime.utils;
-
-import org.apache.flink.api.common.typeutils.TypeSerializer;
-import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
-import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
-import org.apache.flink.cdc.common.annotation.Internal;
-import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.lang.reflect.Field;
-import java.lang.reflect.InvocationTargetException;
-import java.lang.reflect.Method;
-
-/** Utility class for handling Flink version compatibility issues. */
-@Internal
-public class FlinkCompatibilityUtils {
-
- private static final Logger LOG = LoggerFactory.getLogger(FlinkCompatibilityUtils.class);
-
- private FlinkCompatibilityUtils() {
- // Private constructor to prevent instantiation
- }
-
- /**
- * Sets the chaining strategy for an operator via reflection.
- *
- * <p>This method is needed because Flink 2.x removed the {@code chainingStrategy} field from
- * {@link AbstractStreamOperator}. This method attempts to set the field via reflection for
- * backward compatibility with Flink 1.x, and silently ignores the failure in Flink 2.x.
- *
- * @param operator the operator to set the chaining strategy for
- * @param strategy the chaining strategy to set
- */
- public static void setChainingStrategyIfAvailable(
- AbstractStreamOperator<?> operator, ChainingStrategy strategy) {
- try {
- Field field = AbstractStreamOperator.class.getDeclaredField("chainingStrategy");
- field.setAccessible(true);
- field.set(operator, strategy);
- LOG.debug("Successfully set chainingStrategy to {} via reflection", strategy);
- } catch (NoSuchFieldException e) {
- LOG.debug(
- "chainingStrategy field not available (likely Flink 2.x), skipping for operator {}",
- operator.getClass().getSimpleName());
- } catch (Exception e) {
- LOG.warn(
- "Failed to set chainingStrategy via reflection for operator {}",
- operator.getClass().getSimpleName(),
- e);
- }
- }
-
- /**
- * Resolves schema compatibility between a serializer snapshot and a new serializer, handling
- * API differences between Flink 1.x and 2.x.
- *
- * <p>In Flink 1.x, the method is {@code snapshot.resolveSchemaCompatibility(serializer)}. In
- * Flink 2.x, the method was changed to {@code
- * newSnapshot.resolveSchemaCompatibility(oldSnapshot)}.
- */
- @SuppressWarnings("unchecked")
- public static <T> TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
- TypeSerializerSnapshot<T> snapshot, TypeSerializer<T> serializer) {
- // Try Flink 1.x approach: snapshot.resolveSchemaCompatibility(serializer)
- try {
- Method m =
- snapshot.getClass()
- .getMethod("resolveSchemaCompatibility", TypeSerializer.class);
- return (TypeSerializerSchemaCompatibility<T>) m.invoke(snapshot, serializer);
- } catch (NoSuchMethodException ignored) {
- // fall through to Flink 2.x approach
- } catch (InvocationTargetException e) {
- Throwable cause = e.getCause();
- if (cause instanceof RuntimeException) {
- throw (RuntimeException) cause;
- }
- throw new RuntimeException(
- "Failed to resolve schema compatibility (Flink 1.x path)", cause);
- } catch (Exception e) {
- throw new RuntimeException(
- "Failed to resolve schema compatibility (Flink 1.x path)", e);
- }
-
- // Flink 2.x approach: newSnapshot.resolveSchemaCompatibility(oldSnapshot)
- try {
- TypeSerializerSnapshot<T> newSnapshot = serializer.snapshotConfiguration();
- Method m =
- newSnapshot
- .getClass()
- .getMethod("resolveSchemaCompatibility", TypeSerializerSnapshot.class);
- return (TypeSerializerSchemaCompatibility<T>) m.invoke(newSnapshot, snapshot);
- } catch (InvocationTargetException e) {
- Throwable cause = e.getCause();
- if (cause instanceof RuntimeException) {
- throw (RuntimeException) cause;
- }
- throw new RuntimeException(
- "Failed to resolve schema compatibility (Flink 2.x path)", cause);
- } catch (Exception e) {
- throw new RuntimeException(
- "Failed to resolve schema compatibility (Flink 2.x path)", e);
- }
- }
-
- /**
- * Calls {@link AbstractStreamOperator#setup} via reflection to handle access level changes
- * between Flink 1.x (public) and Flink 2.x (protected).
- */
- public static void setupOperator(
- AbstractStreamOperator<?> operator, Object containingTask, Object config, Object output)
- throws Exception {
- Method setupMethod = null;
- for (Method m : AbstractStreamOperator.class.getDeclaredMethods()) {
- if ("setup".equals(m.getName()) && m.getParameterCount() == 3) {
- setupMethod = m;
- break;
- }
- }
- if (setupMethod == null) {
- throw new NoSuchMethodException("Cannot find setup method in AbstractStreamOperator");
- }
- setupMethod.setAccessible(true);
- try {
- setupMethod.invoke(operator, containingTask, config, output);
- } catch (InvocationTargetException e) {
- Throwable cause = e.getCause();
- if (cause instanceof Exception) {
- throw (Exception) cause;
- }
- throw new RuntimeException(cause);
- }
- }
-}
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaEvolveTest.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaEvolveTest.java
index 1b4de204c..7ece5d20c 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaEvolveTest.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaEvolveTest.java
@@ -32,9 +32,9 @@ import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.schema.Column;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.types.DataTypes;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.schema.common.SchemaTestBase;
import org.apache.flink.cdc.runtime.testutils.operators.DistributedEventOperatorTestHarness;
-import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
import org.apache.flink.util.function.BiConsumerWithException;
@@ -378,7 +378,10 @@ public class SchemaEvolveTest extends SchemaTestBase {
"Unexpected schema change events occurred in EXCEPTION mode. Job will fail now.");
}
- protected static <OP extends AbstractStreamOperator<E>, E extends Event, T extends Throwable>
+ protected static <
+ OP extends AbstractStreamOperatorAdapter<E>,
+ E extends Event,
+ T extends Throwable>
LinkedList<StreamRecord<E>> runInHarness(
Supplier<OP> opCreator,
Function<OP, DistributedEventOperatorTestHarness<OP, E>> harnessCreator,
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkOperatorAdapter.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkOperatorAdapter.java
index 0a9e60b7f..61c561348 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkOperatorAdapter.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkOperatorAdapter.java
@@ -24,11 +24,11 @@ import org.apache.flink.cdc.common.event.FlushEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEventType;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.state.StateInitializationContext;
import org.apache.flink.runtime.state.StateSnapshotContext;
import org.apache.flink.streaming.api.graph.StreamConfig;
-import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.BoundedOneInput;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
@@ -46,7 +46,7 @@ import java.util.Set;
* The DataSinkOperatorAdapter class acts as an adapter for testing the core schema evolution
* process in both {@link DataSinkWriterOperator} and {@link DataSinkFunctionOperator}.
*/
-public class DataSinkOperatorAdapter extends AbstractStreamOperator<Event>
+public class DataSinkOperatorAdapter extends AbstractStreamOperatorAdapter<Event>
implements OneInputStreamOperator<Event, Event>, BoundedOneInput {
private SchemaEvolutionClient schemaEvolutionClient;
@@ -59,15 +59,7 @@ public class DataSinkOperatorAdapter extends AbstractStreamOperator<Event>
public DataSinkOperatorAdapter() {
this.schemaOperatorID = new OperatorID();
this.processedTableIds = new HashSet<>();
- // Try to set chainingStrategy via reflection for backward compatibility with Flink 1.x
- try {
- java.lang.reflect.Field field =
- AbstractStreamOperator.class.getDeclaredField("chainingStrategy");
- field.setAccessible(true);
- field.set(this, ChainingStrategy.ALWAYS);
- } catch (Exception e) {
- // Ignore if chainingStrategy doesn't exist in Flink 2.x
- }
+ this.chainingStrategy = ChainingStrategy.ALWAYS;
}
@Override
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/EnumSerializerTest.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/EnumSerializerTest.java
index c7fe72b1b..f18fd0173 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/EnumSerializerTest.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/EnumSerializerTest.java
@@ -22,7 +22,6 @@ import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotSerializationUtil;
import org.apache.flink.cdc.common.utils.InstantiationUtil;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.core.memory.DataInputViewStreamWrapper;
import org.apache.flink.core.memory.DataOutputViewStreamWrapper;
import org.apache.flink.util.TestLogger;
@@ -86,7 +85,7 @@ class EnumSerializerTest extends TestLogger {
EnumSerializer.EnumSerializerSnapshot serializerSnapshot =
new EnumSerializer.EnumSerializerSnapshot(PublicEnum.class, mockPreviousOrder);
TypeSerializerSchemaCompatibility compatibility =
- FlinkCompatibilityUtils.resolveSchemaCompatibility(serializerSnapshot, serializer);
+ serializerSnapshot.resolveSchemaCompatibility(serializer);
Assertions.assertThat(compatibility.isCompatibleWithReconfiguredSerializer()).isTrue();
// after reconfiguration, the order should be first the original BAR, PAULA, NATHANIEL,
@@ -132,7 +131,7 @@ class EnumSerializerTest extends TestLogger {
}
TypeSerializerSchemaCompatibility<PublicEnum> compatResult =
- FlinkCompatibilityUtils.resolveSchemaCompatibility(restoredConfig, serializer);
+ restoredConfig.resolveSchemaCompatibility(serializer.snapshotConfiguration());
Assertions.assertThat(compatResult.isCompatibleAsIs()).isTrue();
Assertions.assertThat(serializer.getValueToOrdinal().get(PublicEnum.FOO).intValue())
@@ -218,7 +217,7 @@ class EnumSerializerTest extends TestLogger {
EnumSerializer.EnumSerializerSnapshot serializerSnapshot =
new EnumSerializer.EnumSerializerSnapshot(PublicEnum.class, mockPreviousOrder);
TypeSerializerSchemaCompatibility compatibility =
- FlinkCompatibilityUtils.resolveSchemaCompatibility(serializerSnapshot, serializer);
+ serializerSnapshot.resolveSchemaCompatibility(serializer);
Assertions.assertThat(compatibility.isCompatibleWithReconfiguredSerializer()).isTrue();
// verify that after the serializer was read, the reconfigured constant ordering is
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/SerializerTestBase.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/SerializerTestBase.java
index d50bb8a39..a3ebd9056 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/SerializerTestBase.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/SerializerTestBase.java
@@ -23,7 +23,6 @@ import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotSerializationUtil;
import org.apache.flink.api.java.typeutils.runtime.NullableSerializer;
import org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.core.memory.DataInputDeserializer;
import org.apache.flink.core.memory.DataInputView;
import org.apache.flink.core.memory.DataInputViewStreamWrapper;
@@ -153,7 +152,7 @@ public abstract class SerializerTestBase<T> {
}
TypeSerializerSchemaCompatibility<T> strategy =
- FlinkCompatibilityUtils.resolveSchemaCompatibility(restoredConfig, getSerializer());
+ restoredConfig.resolveSchemaCompatibility(getSerializer().snapshotConfiguration());
final TypeSerializer<T> restoreSerializer;
if (strategy.isCompatibleAsIs()) {
restoreSerializer = restoredConfig.restoreSerializer();
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/DistributedEventOperatorTestHarness.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/DistributedEventOperatorTestHarness.java
index ab86809e1..cbc1f407d 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/DistributedEventOperatorTestHarness.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/DistributedEventOperatorTestHarness.java
@@ -22,6 +22,7 @@ import org.apache.flink.cdc.common.event.FlushEvent;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.schema.common.event.FlushSuccessEvent;
import org.apache.flink.cdc.runtime.operators.schema.common.event.GetEvolvedSchemaRequest;
import org.apache.flink.cdc.runtime.operators.schema.common.event.GetEvolvedSchemaResponse;
@@ -30,16 +31,21 @@ import org.apache.flink.cdc.runtime.operators.schema.distributed.SchemaCoordinat
import org.apache.flink.cdc.runtime.operators.sink.SchemaEvolutionClient;
import org.apache.flink.cdc.runtime.testutils.schema.CollectingMetadataApplier;
import org.apache.flink.cdc.runtime.testutils.schema.TestingSchemaRegistryGateway;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.event.WatermarkEvent;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.runtime.operators.testutils.DummyEnvironment;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.RecordAttributes;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
import org.apache.flink.streaming.util.MockStreamConfig;
+import org.apache.flink.util.OutputTag;
import org.apache.flink.util.SerializedValue;
import java.io.IOException;
@@ -62,7 +68,7 @@ import static org.apache.flink.cdc.runtime.operators.schema.common.CoordinationR
* @param <E> Type of the event emitted by the operator
*/
public class DistributedEventOperatorTestHarness<
- OP extends AbstractStreamOperator<E>, E extends Event>
+ OP extends AbstractStreamOperatorAdapter<E>, E extends Event>
implements AutoCloseable {
public static final OperatorID SCHEMA_OPERATOR_ID = new OperatorID(15213L, 15513L);
@@ -150,73 +156,72 @@ public class DistributedEventOperatorTestHarness<
// -------------------------------------- Helper functions -------------------------------
- @SuppressWarnings("unchecked")
private void initializeOperator() throws Exception {
- Output<StreamRecord<E>> output =
- (Output<StreamRecord<E>>)
- java.lang.reflect.Proxy.newProxyInstance(
- Output.class.getClassLoader(),
- new Class<?>[] {Output.class},
- new EventCollectingOutputHandler<>(
- outputRecords, schemaRegistryGateway));
- FlinkCompatibilityUtils.setupOperator(
- operator,
+ operator.setup(
new MockStreamTask(schemaRegistryGateway),
new MockStreamConfig(new Configuration(), numOutputs),
- output);
+ new EventCollectingOutput<>(outputRecords, schemaRegistryGateway));
schemaRegistryGateway.sendOperatorEventToCoordinator(
SINK_OPERATOR_ID, new SerializedValue<>(new SinkWriterRegisterEvent(0)));
}
// ---------------------------------------- Helper classes ---------------------------------
- private static class EventCollectingOutputHandler<E extends Event>
- implements java.lang.reflect.InvocationHandler {
+ private static class EventCollectingOutput<E extends Event> implements Output<StreamRecord<E>> {
private final LinkedList<StreamRecord<E>> outputRecords;
+
private final TestingSchemaRegistryGateway schemaRegistryGateway;
- EventCollectingOutputHandler(
+ public EventCollectingOutput(
LinkedList<StreamRecord<E>> outputRecords,
TestingSchemaRegistryGateway schemaRegistryGateway) {
this.outputRecords = outputRecords;
this.schemaRegistryGateway = schemaRegistryGateway;
}
- @SuppressWarnings("unchecked")
@Override
- public Object invoke(Object proxy, java.lang.reflect.Method method, Object[] args)
- throws Throwable {
- switch (method.getName()) {
- case "collect":
- if (args != null && args.length == 1 && args[0] instanceof StreamRecord) {
- StreamRecord<E> record = (StreamRecord<E>) args[0];
- outputRecords.add(record);
- Event event = record.getValue();
- if (event instanceof FlushEvent) {
- try {
- schemaRegistryGateway.sendOperatorEventToCoordinator(
- SINK_OPERATOR_ID,
- new SerializedValue<>(new FlushSuccessEvent(0, 0)));
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
- return null;
- }
- throw new UnsupportedOperationException();
- case "emitRecordAttributes":
- case "close":
- return null;
- case "toString":
- return "EventCollectingOutput";
- case "hashCode":
- return System.identityHashCode(proxy);
- case "equals":
- return proxy == args[0];
- default:
- throw new UnsupportedOperationException(method.getName());
+ public void collect(StreamRecord<E> record) {
+ outputRecords.add(record);
+ Event event = record.getValue();
+ if (event instanceof FlushEvent) {
+ try {
+ schemaRegistryGateway.sendOperatorEventToCoordinator(
+ SINK_OPERATOR_ID, new SerializedValue<>(new FlushSuccessEvent(0, 0)));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
}
}
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ throw new UnsupportedOperationException();
+ }
+
+ public void emitWatermark(WatermarkEvent mark) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void emitWatermarkStatus(WatermarkStatus watermarkStatus) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public <X> void collect(OutputTag<X> outputTag, StreamRecord<X> record) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void emitRecordAttributes(RecordAttributes recordAttributes) {}
+
+ @Override
+ public void close() {}
}
private static class MockStreamTask extends StreamTask<Event, AbstractStreamOperator<Event>> {
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/RegularEventOperatorTestHarness.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/RegularEventOperatorTestHarness.java
index 6b4facdbb..8a3a21ba6 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/RegularEventOperatorTestHarness.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/RegularEventOperatorTestHarness.java
@@ -26,6 +26,7 @@ import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.pipeline.PipelineOptions;
import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.runtime.operators.AbstractStreamOperatorAdapter;
import org.apache.flink.cdc.runtime.operators.schema.common.CoordinationResponseUtils;
import org.apache.flink.cdc.runtime.operators.schema.common.event.FlushSuccessEvent;
import org.apache.flink.cdc.runtime.operators.schema.common.event.GetEvolvedSchemaRequest;
@@ -39,16 +40,21 @@ import org.apache.flink.cdc.runtime.operators.schema.regular.event.SchemaChangeR
import org.apache.flink.cdc.runtime.operators.sink.SchemaEvolutionClient;
import org.apache.flink.cdc.runtime.testutils.schema.CollectingMetadataApplier;
import org.apache.flink.cdc.runtime.testutils.schema.TestingSchemaRegistryGateway;
-import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.event.WatermarkEvent;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.runtime.operators.testutils.DummyEnvironment;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.RecordAttributes;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
import org.apache.flink.streaming.util.MockStreamConfig;
+import org.apache.flink.util.OutputTag;
import org.apache.flink.util.SerializedValue;
import org.slf4j.Logger;
@@ -78,7 +84,8 @@ import static org.apache.flink.cdc.runtime.operators.schema.common.CoordinationR
* @param <OP> Type of the operator
* @param <E> Type of the event emitted by the operator
*/
-public class RegularEventOperatorTestHarness<OP extends AbstractStreamOperator<E>, E extends Event>
+public class RegularEventOperatorTestHarness<
+ OP extends AbstractStreamOperatorAdapter<E>, E extends Event>
implements AutoCloseable {
private static final Logger LOG =
LoggerFactory.getLogger(RegularEventOperatorTestHarness.class);
@@ -122,7 +129,7 @@ public class RegularEventOperatorTestHarness<OP extends AbstractStreamOperator<E
schemaRegistryGateway = new TestingSchemaRegistryGateway(schemaRegistry);
}
- public static <OP extends AbstractStreamOperator<E>, E extends Event>
+ public static <OP extends AbstractStreamOperatorAdapter<E>, E extends Event>
RegularEventOperatorTestHarness<OP, E> with(OP operator, int numOutputs) {
return new RegularEventOperatorTestHarness<>(
operator,
@@ -134,7 +141,7 @@ public class RegularEventOperatorTestHarness<OP extends AbstractStreamOperator<E
Collections.emptySet());
}
- public static <OP extends AbstractStreamOperator<E>, E extends Event>
+ public static <OP extends AbstractStreamOperatorAdapter<E>, E extends Event>
RegularEventOperatorTestHarness<OP, E> withDuration(
OP operator, int numOutputs, Duration evolveDuration) {
return new RegularEventOperatorTestHarness<>(
@@ -147,7 +154,7 @@ public class RegularEventOperatorTestHarness<OP extends AbstractStreamOperator<E
Collections.emptySet());
}
- public static <OP extends AbstractStreamOperator<E>, E extends Event>
+ public static <OP extends AbstractStreamOperatorAdapter<E>, E extends Event>
RegularEventOperatorTestHarness<OP, E> withDurationAndBehavior(
OP operator,
int numOutputs,
@@ -163,7 +170,7 @@ public class RegularEventOperatorTestHarness<OP extends AbstractStreamOperator<E
Collections.emptySet());
}
- public static <OP extends AbstractStreamOperator<E>, E extends Event>
+ public static <OP extends AbstractStreamOperatorAdapter<E>, E extends Event>
RegularEventOperatorTestHarness<OP, E> withDurationAndFineGrainedBehavior(
OP operator,
int numOutputs,
@@ -180,7 +187,7 @@ public class RegularEventOperatorTestHarness<OP extends AbstractStreamOperator<E
Collections.emptySet());
}
- public static <OP extends AbstractStreamOperator<E>, E extends Event>
+ public static <OP extends AbstractStreamOperatorAdapter<E>, E extends Event>
RegularEventOperatorTestHarness<OP, E> withDurationAndFineGrainedBehaviorWithError(
OP operator,
int numOutputs,
@@ -281,73 +288,72 @@ public class RegularEventOperatorTestHarness<OP extends AbstractStreamOperator<E
// -------------------------------------- Helper functions -------------------------------
- @SuppressWarnings("unchecked")
private void initializeOperator() throws Exception {
- Output<StreamRecord<E>> output =
- (Output<StreamRecord<E>>)
- java.lang.reflect.Proxy.newProxyInstance(
- Output.class.getClassLoader(),
- new Class<?>[] {Output.class},
- new EventCollectingOutputHandler<>(
- outputRecords, schemaRegistryGateway));
- FlinkCompatibilityUtils.setupOperator(
- operator,
+ operator.setup(
new MockStreamTask(schemaRegistryGateway),
new MockStreamConfig(new Configuration(), numOutputs),
- output);
+ new EventCollectingOutput<>(outputRecords, schemaRegistryGateway));
schemaRegistryGateway.sendOperatorEventToCoordinator(
SINK_OPERATOR_ID, new SerializedValue<>(new SinkWriterRegisterEvent(0)));
}
// ---------------------------------------- Helper classes ---------------------------------
- private static class EventCollectingOutputHandler<E extends Event>
- implements java.lang.reflect.InvocationHandler {
+ private static class EventCollectingOutput<E extends Event> implements Output<StreamRecord<E>> {
private final LinkedList<StreamRecord<E>> outputRecords;
+
private final TestingSchemaRegistryGateway schemaRegistryGateway;
- EventCollectingOutputHandler(
+ public EventCollectingOutput(
LinkedList<StreamRecord<E>> outputRecords,
TestingSchemaRegistryGateway schemaRegistryGateway) {
this.outputRecords = outputRecords;
this.schemaRegistryGateway = schemaRegistryGateway;
}
- @SuppressWarnings("unchecked")
@Override
- public Object invoke(Object proxy, java.lang.reflect.Method method, Object[] args)
- throws Throwable {
- switch (method.getName()) {
- case "collect":
- if (args != null && args.length == 1 && args[0] instanceof StreamRecord) {
- StreamRecord<E> record = (StreamRecord<E>) args[0];
- outputRecords.add(record);
- Event event = record.getValue();
- if (event instanceof FlushEvent) {
- try {
- schemaRegistryGateway.sendOperatorEventToCoordinator(
- SINK_OPERATOR_ID,
- new SerializedValue<>(new FlushSuccessEvent(0, 0)));
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
- return null;
- }
- throw new UnsupportedOperationException();
- case "emitRecordAttributes":
- case "close":
- return null;
- case "toString":
- return "EventCollectingOutput";
- case "hashCode":
- return System.identityHashCode(proxy);
- case "equals":
- return proxy == args[0];
- default:
- throw new UnsupportedOperationException(method.getName());
+ public void collect(StreamRecord<E> record) {
+ outputRecords.add(record);
+ Event event = record.getValue();
+ if (event instanceof FlushEvent) {
+ try {
+ schemaRegistryGateway.sendOperatorEventToCoordinator(
+ SINK_OPERATOR_ID, new SerializedValue<>(new FlushSuccessEvent(0, 0)));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
}
}
+
+ @Override
+ public void emitWatermark(Watermark mark) {
+ throw new UnsupportedOperationException();
+ }
+
+ public void emitWatermark(WatermarkEvent mark) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void emitWatermarkStatus(WatermarkStatus watermarkStatus) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public <X> void collect(OutputTag<X> outputTag, StreamRecord<X> record) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void emitLatencyMarker(LatencyMarker latencyMarker) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void emitRecordAttributes(RecordAttributes recordAttributes) {}
+
+ @Override
+ public void close() {}
}
private static class MockStreamTask extends StreamTask<Event, AbstractStreamOperator<Event>> {
diff --git a/pom.xml b/pom.xml
index 295538df5..a9f49f51b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -33,6 +33,7 @@ limitations under the License.
<packaging>pom</packaging>
<modules>
+ <module>flink-cdc-flink1.20-compat</module>
<module>flink-cdc-flink2.2-compat</module>
<module>flink-cdc-cli</module>
<module>flink-cdc-common</module>
(flink-cdc) 09/12: Update java doc.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 10e3747d86057a8832ae740b391260e2e63179d5
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Tue Mar 10 23:25:49 2026 +0800
Update java doc.
---
flink-cdc-flink1.20-compat/pom.xml | 62 +---------------------
.../typeutils/TypeSerializerSnapshotAdapter.java | 8 ++-
.../operators/AbstractStreamOperatorAdapter.java | 7 ++-
.../apache/flink/runtime/event/WatermarkEvent.java | 8 ++-
flink-cdc-flink2.2-compat/pom.xml | 62 +---------------------
.../typeutils/TypeSerializerSnapshotAdapter.java | 8 ++-
.../org/apache/flink/api/connector/sink2/Sink.java | 12 +++--
.../connector/sink2/TwoPhaseCommittingSink.java | 8 ++-
.../operators/AbstractStreamOperatorAdapter.java | 7 ++-
.../apache/flink/core/execution/RestoreMode.java | 8 +--
.../runtime/jobgraph/SavepointConfigOptions.java | 9 ++--
.../connector/sink2/WithPostCommitTopology.java | 8 ++-
.../api/connector/sink2/WithPreCommitTopology.java | 8 ++-
.../api/connector/sink2/WithPreWriteTopology.java | 7 ++-
.../streaming/api/functions/sink/SinkFunction.java | 8 ++-
.../source/RichParallelSourceFunction.java | 9 ++--
.../api/functions/source/SourceFunction.java | 8 ++-
.../api/operators/YieldingOperatorFactory.java | 8 ++-
.../flink/table/api/ValidationException.java | 9 ++--
19 files changed, 113 insertions(+), 151 deletions(-)
diff --git a/flink-cdc-flink1.20-compat/pom.xml b/flink-cdc-flink1.20-compat/pom.xml
index b65c53497..45a99cccd 100644
--- a/flink-cdc-flink1.20-compat/pom.xml
+++ b/flink-cdc-flink1.20-compat/pom.xml
@@ -32,64 +32,4 @@ limitations under the License.
<shaded.guava.version>${flink.1.x.shaded.guava.version}</shaded.guava.version>
</properties>
- <dependencies>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-common</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-base</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-java-bridge</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-core</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- <version>${slf4j.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-shaded-guava</artifactId>
- <version>${shaded.guava.version}</version>
- </dependency>
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-jar-plugin</artifactId>
- <executions>
- <execution>
- <id>test-jar</id>
- <goals>
- <goal>test-jar</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
-
-</project>
\ No newline at end of file
+</project>
diff --git a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
index a74e14c97..b868b4fba 100644
--- a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
+++ b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
@@ -17,5 +17,11 @@
package org.apache.flink.api.common.typeutils;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+import org.apache.flink.annotation.Internal;
+
+/**
+ * Compatibility adapter for Flink 1.20. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public interface TypeSerializerSnapshotAdapter<T> extends TypeSerializerSnapshot<T> {}
diff --git a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
index b4fe16f5e..73c578b04 100644
--- a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
+++ b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
@@ -17,7 +17,12 @@
package org.apache.flink.cdc.runtime.operators;
+import org.apache.flink.annotation.Internal;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+/**
+ * Compatibility adapter for Flink 1.20. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public class AbstractStreamOperatorAdapter<OUT> extends AbstractStreamOperator<OUT> {}
diff --git a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/runtime/event/WatermarkEvent.java b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/runtime/event/WatermarkEvent.java
index afa423894..b9d7f7fb6 100644
--- a/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/runtime/event/WatermarkEvent.java
+++ b/flink-cdc-flink1.20-compat/src/main/java/org/apache/flink/runtime/event/WatermarkEvent.java
@@ -17,5 +17,11 @@
package org.apache.flink.runtime.event;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+import org.apache.flink.annotation.Internal;
+
+/**
+ * Compatibility adapter for Flink 1.20. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public class WatermarkEvent {}
diff --git a/flink-cdc-flink2.2-compat/pom.xml b/flink-cdc-flink2.2-compat/pom.xml
index ffed8f2dc..5e755b2ee 100644
--- a/flink-cdc-flink2.2-compat/pom.xml
+++ b/flink-cdc-flink2.2-compat/pom.xml
@@ -32,64 +32,4 @@ limitations under the License.
<shaded.guava.version>${flink.2.x.shaded.guava.version}</shaded.guava.version>
</properties>
- <dependencies>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-common</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-base</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-java-bridge</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-core</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java</artifactId>
- <version>${flink.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- <version>${slf4j.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-shaded-guava</artifactId>
- <version>${shaded.guava.version}</version>
- </dependency>
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-jar-plugin</artifactId>
- <executions>
- <execution>
- <id>test-jar</id>
- <goals>
- <goal>test-jar</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
-
-</project>
\ No newline at end of file
+</project>
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
index a969801ee..5060d9219 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSnapshotAdapter.java
@@ -17,7 +17,13 @@
package org.apache.flink.api.common.typeutils;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+import org.apache.flink.annotation.Internal;
+
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public interface TypeSerializerSnapshotAdapter<T> extends TypeSerializerSnapshot<T> {
TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/Sink.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/Sink.java
index f4b5d6d01..0b06d3f51 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/Sink.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/Sink.java
@@ -18,8 +18,7 @@
package org.apache.flink.api.connector.sink2;
import org.apache.flink.annotation.Experimental;
-import org.apache.flink.annotation.Public;
-import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.Internal;
import org.apache.flink.api.common.operators.MailboxExecutor;
import org.apache.flink.api.common.operators.ProcessingTimeService;
import org.apache.flink.api.common.serialization.SerializationSchema;
@@ -32,8 +31,11 @@ import java.io.Serializable;
import java.util.Optional;
import java.util.function.Consumer;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
-@Public
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public interface Sink<InputT> extends Serializable {
SinkWriter<InputT> createWriter(WriterInitContext var1) throws IOException;
@@ -48,7 +50,7 @@ public interface Sink<InputT> extends Serializable {
* @deprecated
*/
@Deprecated
- @PublicEvolving
+ @Internal
public interface InitContext extends org.apache.flink.api.connector.sink2.InitContext {
UserCodeClassLoader getUserCodeClassLoader();
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java
index 86d0636a5..cae29dbfe 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java
@@ -17,6 +17,12 @@
package org.apache.flink.api.connector.sink2;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+import org.apache.flink.annotation.Internal;
+
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public interface TwoPhaseCommittingSink<InputT, CommT>
extends Sink<InputT>, SupportsCommitter<CommT> {}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
index f8cdb6f25..e893db104 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/cdc/runtime/operators/AbstractStreamOperatorAdapter.java
@@ -17,6 +17,7 @@
package org.apache.flink.cdc.runtime.operators;
+import org.apache.flink.annotation.Internal;
import org.apache.flink.streaming.api.graph.StreamConfig;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
@@ -24,7 +25,11 @@ import org.apache.flink.streaming.api.operators.Output;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
import org.apache.flink.streaming.runtime.tasks.StreamTask;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public class AbstractStreamOperatorAdapter<OUT> extends AbstractStreamOperator<OUT> {
protected ChainingStrategy chainingStrategy;
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/core/execution/RestoreMode.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/core/execution/RestoreMode.java
index 80227ea9b..6e7cca53f 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/core/execution/RestoreMode.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/core/execution/RestoreMode.java
@@ -18,16 +18,18 @@
package org.apache.flink.core.execution;
import org.apache.flink.annotation.Internal;
-import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.configuration.DescribedEnum;
import org.apache.flink.configuration.description.InlineElement;
import org.apache.flink.configuration.description.TextElement;
/**
- * Copy from <a
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ *
+ * <p>Copy from <a
* href="https://github.com/apache/flink/blob/release-1.20.3/flink-core/src/main/java/org/apache/flink/core/execution/RestoreMode.java">...</a>.
*/
-@PublicEvolving
+@Internal
public enum RestoreMode implements DescribedEnum {
CLAIM(
"Flink will take ownership of the given snapshot. It will clean the snapshot once it is subsumed by newer ones."),
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java
index f281c0ee4..83494815c 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java
@@ -17,19 +17,22 @@
package org.apache.flink.runtime.jobgraph;
-import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.Internal;
import org.apache.flink.annotation.docs.Documentation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ConfigOptions;
import org.apache.flink.core.execution.RestoreMode;
/**
- * Copy from <a
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ *
+ * <p>Copy from <a
* href="https://github.com/apache/flink/blob/release-1.20.3/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java">...</a>.
*/
@Deprecated
@Documentation.ExcludeFromDocumentation("Hidden for deprecated.")
-@PublicEvolving
+@Internal
public class SavepointConfigOptions {
public static final ConfigOption<String> SAVEPOINT_PATH =
ConfigOptions.key("execution.savepoint.path")
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPostCommitTopology.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPostCommitTopology.java
index 1f0cb5c86..2fe0127b9 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPostCommitTopology.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPostCommitTopology.java
@@ -17,5 +17,11 @@
package org.apache.flink.streaming.api.connector.sink2;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+import org.apache.flink.annotation.Internal;
+
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public interface WithPostCommitTopology<InputT, CommT> extends SupportsPostCommitTopology<CommT> {}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreCommitTopology.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreCommitTopology.java
index eda582e47..de18bdb86 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreCommitTopology.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreCommitTopology.java
@@ -17,6 +17,12 @@
package org.apache.flink.streaming.api.connector.sink2;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+import org.apache.flink.annotation.Internal;
+
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public interface WithPreCommitTopology<InputT, CommT>
extends SupportsPreCommitTopology<CommT, CommT> {}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreWriteTopology.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreWriteTopology.java
index 719c47459..8a1d8c2f2 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreWriteTopology.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreWriteTopology.java
@@ -17,8 +17,13 @@
package org.apache.flink.streaming.api.connector.sink2;
+import org.apache.flink.annotation.Internal;
import org.apache.flink.api.connector.sink2.Sink;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public interface WithPreWriteTopology<InputT>
extends Sink<InputT>, SupportsPreWriteTopology<InputT> {}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java
index e4f457374..ff1a27611 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java
@@ -17,6 +17,12 @@
package org.apache.flink.streaming.api.functions.sink;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+import org.apache.flink.annotation.Internal;
+
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public interface SinkFunction<IN>
extends org.apache.flink.streaming.api.functions.sink.legacy.SinkFunction<IN> {}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/RichParallelSourceFunction.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/RichParallelSourceFunction.java
index 127ab3cc9..c45677998 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/RichParallelSourceFunction.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/RichParallelSourceFunction.java
@@ -17,15 +17,18 @@
package org.apache.flink.streaming.api.functions.source;
-import org.apache.flink.annotation.Public;
+import org.apache.flink.annotation.Internal;
import org.apache.flink.api.common.functions.AbstractRichFunction;
import org.apache.flink.api.common.functions.OpenContext;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.functions.source.legacy.ParallelSourceFunction;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
@Deprecated
-@Public
+@Internal
public abstract class RichParallelSourceFunction<OUT> extends AbstractRichFunction
implements ParallelSourceFunction<OUT>, SourceFunction<OUT> {
private static final long serialVersionUID = 1L;
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java
index d61e8a991..6147d72d2 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java
@@ -17,6 +17,12 @@
package org.apache.flink.streaming.api.functions.source;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+import org.apache.flink.annotation.Internal;
+
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public interface SourceFunction<T>
extends org.apache.flink.streaming.api.functions.source.legacy.SourceFunction<T> {}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/operators/YieldingOperatorFactory.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/operators/YieldingOperatorFactory.java
index 34aaac426..0cc855d45 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/operators/YieldingOperatorFactory.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/operators/YieldingOperatorFactory.java
@@ -17,6 +17,12 @@
package org.apache.flink.streaming.api.operators;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+import org.apache.flink.annotation.Internal;
+
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public interface YieldingOperatorFactory<OUT>
extends org.apache.flink.streaming.api.operators.legacy.YieldingOperatorFactory<OUT> {}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/table/api/ValidationException.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/table/api/ValidationException.java
index fc6a8e37c..20cc03305 100644
--- a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/table/api/ValidationException.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/table/api/ValidationException.java
@@ -17,12 +17,15 @@
package org.apache.flink.table.api;
-import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.Internal;
import javax.annotation.Nullable;
-/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
-@PublicEvolving
+/**
+ * Compatibility adapter for Flink 2.2. This class is part of the multi-version compatibility layer
+ * that allows Flink CDC to work across different Flink versions.
+ */
+@Internal
public class ValidationException extends RuntimeException {
public ValidationException(String message) {
(flink-cdc) 01/12: Add support for Flink 2.x.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 7c28071eabf01f75933edbfa51376a51ea06d5c5
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Sat Mar 7 18:02:36 2026 +0800
Add support for Flink 2.x.
Co-authored-by: xiaoxiong.duan@zznode.com <xi...@zznode.com>
---
.github/workflows/flink_cdc_base.yml | 2 +-
.github/workflows/flink_cdc_ci.yml | 22 +
.github/workflows/flink_cdc_ci_nightly.yml | 24 +
.github/workflows/modules.py | 23 +-
flink-cdc-cli/pom.xml | 21 +
.../java/org/apache/flink/cdc/cli/CliFrontend.java | 8 +-
.../flink/cdc/cli/utils/ConfigurationUtils.java | 6 +-
.../org/apache/flink/cdc/cli/CliFrontendTest.java | 9 +-
flink-cdc-common/pom.xml | 33 ++
.../ValidationException.java} | 24 +-
.../flink/cdc/common/factories/FactoryHelper.java | 2 +-
.../flink/cdc/common/sink/EventSinkProvider.java | 1 -
.../cdc/common/source/EventSourceProvider.java | 1 -
.../cdc/common/factories/FactoryHelperTests.java | 2 +-
flink-cdc-composer/pom.xml | 38 +-
.../flink/translator/DataSinkTranslator.java | 261 +++++++--
.../flink/translator/DataSourceTranslator.java | 67 ++-
.../testsource/source/DistributedDataSource.java | 5 +-
.../testsource/source/DistributedSource.java | 587 +++++++++++++++++++++
.../source/DistributedSourceFunction.java | 276 ----------
.../sink/ElasticsearchDataSinkFactory.java | 2 +-
.../sink/ElasticsearchDataSinkFactoryTest.java | 2 +-
.../fluss/factory/FlussDataSinkFactoryTest.java | 2 +-
.../hudi/sink/HudiDataSinkFactoryTest.java | 2 +-
.../iceberg/sink/IcebergDataSinkFactoryTest.java | 2 +-
.../kafka/sink/KafkaDataSinkFactoryTest.java | 2 +-
.../mysql/factory/MySqlDataSourceFactory.java | 2 +-
.../mysql/source/MySqlDataSourceFactoryTest.java | 2 +-
.../oracle/factory/OracleDataSourceFactory.java | 2 +-
.../paimon/sink/PaimonDataSinkFactoryTest.java | 2 +-
.../factory/PostgresDataSourceFactory.java | 2 +-
.../factory/PostgresDataSourceFactoryTest.java | 1 +
.../sink/StarRocksDataSinkFactoryTest.java | 2 +-
.../pom.xml | 131 ++---
.../cdc/connectors/values/sink/ValuesDataSink.java | 16 +-
.../org.apache.flink.cdc.common.factories.Factory | 16 +
.../cdc/connectors/values/sink/ValuesDataSink.java | 23 +-
.../values/sink/ValuesDataSinkFunction.java | 101 ----
.../flink-cdc-pipeline-connectors/pom.xml | 1 +
flink-cdc-dist/pom.xml | 13 +
.../flink-cdc-pipeline-e2e-tests-2.x/pom.xml | 454 ++++++++++++++++
.../flink/cdc/pipeline/tests/ValuesE2eITCase.java | 260 +++++++++
.../tests/utils/PipelineTestEnvironment.java | 12 +-
.../cdc/pipeline/tests/utils/TarballFetcher.java | 151 ++++++
.../src/test/resources/docker/mysql/my.cnf | 64 +++
.../src/test/resources/docker/mysql/setup.sql | 30 ++
.../src/test/resources/log4j2-test.properties | 25 +
.../cdc/pipeline/tests/MaxComputeE2eITCase.java | 2 +-
.../cdc/pipeline/tests/MySqlToHudiE2eITCase.java | 2 +-
.../tests/utils/PipelineTestEnvironment.java | 2 +-
.../tests/utils/FlinkContainerTestEnvironment.java | 2 +-
flink-cdc-e2e-tests/pom.xml | 1 +
flink-cdc-runtime/pom.xml | 32 ++
.../schema/distributed/SchemaOperator.java | 2 +-
.../schema/regular/BatchSchemaOperator.java | 3 +-
.../operators/schema/regular/SchemaOperator.java | 5 +-
.../sink/BatchDataSinkFunctionOperator.java | 3 +-
.../sink/BatchDataSinkWriterOperator.java | 136 ++++-
.../operators/sink/DataSinkFunctionOperator.java | 9 +-
.../operators/sink/DataSinkWriterOperator.java | 250 ++++++++-
.../sink/DataSinkWriterOperatorFactory.java | 14 +-
.../operators/transform/PreTransformOperator.java | 26 +-
.../BatchRegularPrePartitionOperator.java | 3 +-
.../DistributedPrePartitionOperator.java | 5 +-
.../partitioning/RegularPrePartitionOperator.java | 3 +-
.../cdc/runtime/serializer/EnumSerializer.java | 29 +-
.../NestedSerializersSnapshotDelegate.java | 36 +-
.../serializer/NullableSerializerWrapper.java | 18 +-
.../serializer/data/ArrayDataSerializer.java | 18 +-
.../serializer/data/DecimalDataSerializer.java | 19 +-
.../data/LocalZonedTimestampDataSerializer.java | 20 +-
.../runtime/serializer/data/MapDataSerializer.java | 19 +-
.../serializer/data/TimestampDataSerializer.java | 19 +-
.../data/ZonedTimestampDataSerializer.java | 17 +-
.../flink/cdc/runtime/typeutils/EventTypeInfo.java | 10 +-
.../typeutils/PartitioningEventTypeInfo.java | 10 +-
.../cdc/runtime/utils/FlinkCompatibilityUtils.java | 153 ++++++
.../operators/sink/DataSinkOperatorAdapter.java | 16 +-
.../cdc/runtime/serializer/EnumSerializerTest.java | 7 +-
.../cdc/runtime/serializer/SerializerTestBase.java | 3 +-
.../DistributedEventOperatorTestHarness.java | 93 ++--
.../operators/RegularEventOperatorTestHarness.java | 93 ++--
pom.xml | 50 +-
83 files changed, 3116 insertions(+), 748 deletions(-)
diff --git a/.github/workflows/flink_cdc_base.yml b/.github/workflows/flink_cdc_base.yml
index c5157abc9..982ea81e3 100644
--- a/.github/workflows/flink_cdc_base.yml
+++ b/.github/workflows/flink_cdc_base.yml
@@ -108,7 +108,7 @@ jobs:
build_maven_parameter="${build_maven_parameter:+$build_maven_parameter }${{ inputs.custom-maven-parameter }}"
- mvn --no-snapshot-updates -B -DskipTests -pl $compile_modules -am install && mvn --no-snapshot-updates -B $build_maven_parameter -pl $modules -DspecifiedParallelism=${{ inputs.parallelism }} -Duser.timezone=$jvm_timezone verify
+ mvn --no-snapshot-updates -B -DskipTests ${{ inputs.custom-maven-parameter }} -pl $compile_modules -am install && mvn --no-snapshot-updates -B $build_maven_parameter -pl $modules -DspecifiedParallelism=${{ inputs.parallelism }} -Duser.timezone=$jvm_timezone verify
- name: Print JVM thread dumps when cancelled
if: ${{ failure() }}
diff --git a/.github/workflows/flink_cdc_ci.yml b/.github/workflows/flink_cdc_ci.yml
index 5398ca43e..66d2f71be 100644
--- a/.github/workflows/flink_cdc_ci.yml
+++ b/.github/workflows/flink_cdc_ci.yml
@@ -63,6 +63,14 @@ jobs:
with:
java-versions: "[11]"
modules: "['core']"
+ common_2_x:
+ name: Common Unit Tests 2.x
+ uses: ./.github/workflows/flink_cdc_base.yml
+ with:
+ java-versions: "[11]"
+ flink-versions: "['2.2.0']"
+ custom-maven-parameter: "-Pflink2"
+ modules: "['core_2.x']"
pipeline-ut:
name: Pipeline Unit Tests
uses: ./.github/workflows/flink_cdc_base.yml
@@ -88,6 +96,20 @@ jobs:
flink-versions: "['1.19.3', '1.20.3']"
modules: "['pipeline_e2e']"
parallelism: ${{ matrix.parallelism }}
+ pipeline_e2e_2_x:
+ strategy:
+ max-parallel: 2
+ fail-fast: false
+ matrix:
+ parallelism: [ 1, 4 ]
+ name: Pipeline E2E Tests 2.x (${{ matrix.parallelism }}-Parallelism)
+ uses: ./.github/workflows/flink_cdc_base.yml
+ with:
+ java-versions: "[11]"
+ flink-versions: "['2.2.0']"
+ custom-maven-parameter: "-Pflink2"
+ modules: "['pipeline_e2e_2.x']"
+ parallelism: ${{ matrix.parallelism }}
source_e2e:
name: Source E2E Tests
uses: ./.github/workflows/flink_cdc_base.yml
diff --git a/.github/workflows/flink_cdc_ci_nightly.yml b/.github/workflows/flink_cdc_ci_nightly.yml
index d48016764..5b01fc62c 100644
--- a/.github/workflows/flink_cdc_ci_nightly.yml
+++ b/.github/workflows/flink_cdc_ci_nightly.yml
@@ -54,6 +54,15 @@ jobs:
with:
java-versions: "[17]"
modules: "['core']"
+ common_2_x:
+ if: github.repository == 'apache/flink-cdc'
+ name: Common Unit Tests 2.x
+ uses: ./.github/workflows/flink_cdc_base.yml
+ with:
+ java-versions: "[17]"
+ flink-versions: "['2.2.0']"
+ custom-maven-parameter: "-Pflink2"
+ modules: "['core_2.x']"
pipeline-ut:
if: github.repository == 'apache/flink-cdc'
name: Pipeline Unit Tests
@@ -82,6 +91,21 @@ jobs:
flink-versions: "['1.19.3', '1.20.3']"
modules: "['pipeline_e2e']"
parallelism: ${{ matrix.parallelism }}
+ pipeline_e2e_2_x:
+ if: github.repository == 'apache/flink-cdc'
+ strategy:
+ max-parallel: 2
+ fail-fast: false
+ matrix:
+ parallelism: [ 1, 4 ]
+ name: Pipeline E2E Tests 2.x (${{ matrix.parallelism }}-Parallelism)
+ uses: ./.github/workflows/flink_cdc_base.yml
+ with:
+ java-versions: "[17]"
+ custom-maven-parameter: "-Pflink2"
+ flink-versions: "['2.2.0']"
+ modules: "['pipeline_e2e_2.x']"
+ parallelism: ${{ matrix.parallelism }}
source_e2e:
if: github.repository == 'apache/flink-cdc'
name: Source E2E Tests
diff --git a/.github/workflows/modules.py b/.github/workflows/modules.py
index d2d051c76..ffbcf0742 100755
--- a/.github/workflows/modules.py
+++ b/.github/workflows/modules.py
@@ -17,6 +17,15 @@ MODULES_CORE = [
"flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values"
]
+MODULES_CORE_2_X = [
+ "flink-cdc-cli",
+ "flink-cdc-common",
+ "flink-cdc-composer",
+ "flink-cdc-runtime",
+ "flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values",
+ "flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x"
+]
+
MODULES_PIPELINE_CONNECTORS = [
"flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris",
"flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch",
@@ -136,12 +145,17 @@ MODULES_PIPELINE_E2E = [
"flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests"
]
+MODULES_PIPELINE_E2E_2_X = [
+ "flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x"
+]
+
MODULES_SOURCE_E2E = [
"flink-cdc-e2e-tests/flink-cdc-source-e2e-tests"
]
ALL_MODULES = set(
MODULES_CORE +
+ MODULES_CORE_2_X +
MODULES_PIPELINE_CONNECTORS +
MODULES_MYSQL_SOURCE +
MODULES_MYSQL_PIPELINE +
@@ -169,13 +183,18 @@ ALL_MODULES = set(
MODULES_SOURCE_E2E
)
+# Modules that require the flink2 Maven profile to be activated
+ALL_MODULES_FLINK2 = ALL_MODULES | set(MODULES_PIPELINE_E2E_2_X)
+
test_modules = set()
compile_modules = set()
for module in INPUT_MODULES.split(', '):
- module_list = set(globals()['MODULES_' + module.upper().replace('-', '_')])
+ module_list = set(globals()['MODULES_' + module.upper().replace('-', '_').replace('.', '_')])
test_modules |= module_list
- if module == 'source_e2e' or module == 'pipeline_e2e':
+ if module == 'pipeline_e2e_2.x':
+ compile_modules |= ALL_MODULES_FLINK2
+ elif module == 'source_e2e' or module == 'pipeline_e2e':
compile_modules |= ALL_MODULES
else:
compile_modules |= module_list
diff --git a/flink-cdc-cli/pom.xml b/flink-cdc-cli/pom.xml
index 35a69cbd2..7934cf576 100644
--- a/flink-cdc-cli/pom.xml
+++ b/flink-cdc-cli/pom.xml
@@ -82,4 +82,25 @@ limitations under the License.
</dependencies>
+ <profiles>
+ <profile>
+ <id>flink2</id>
+ <properties>
+ <flink.version>${flink.2.x.version}</flink.version>
+ </properties>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+
</project>
\ No newline at end of file
diff --git a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java
index 5f327eca2..fdabbee77 100644
--- a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java
+++ b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java
@@ -27,7 +27,6 @@ import org.apache.flink.cdc.common.utils.StringUtils;
import org.apache.flink.cdc.composer.PipelineExecution;
import org.apache.flink.configuration.DeploymentOptions;
import org.apache.flink.core.fs.Path;
-import org.apache.flink.runtime.jobgraph.SavepointConfigOptions;
import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
import org.apache.flink.shaded.guava31.com.google.common.base.Joiner;
@@ -173,7 +172,12 @@ public class CliFrontend {
commandLine.getOptionValue(SAVEPOINT_CLAIM_MODE),
ConfigurationUtils.getClaimModeClass());
} else {
- restoreMode = SavepointConfigOptions.RESTORE_MODE.defaultValue();
+ try {
+ restoreMode =
+ ConfigurationUtils.getClaimModeClass().getField("DEFAULT").get(null);
+ } catch (NoSuchFieldException | IllegalAccessException e) {
+ throw new RuntimeException("Failed to get default restore mode.", e);
+ }
}
// allowNonRestoredState is always false because all operators are predefined.
diff --git a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java
index 7280c0679..75b8c9019 100644
--- a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java
+++ b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java
@@ -72,7 +72,11 @@ public class ConfigurationUtils {
try {
return Class.forName("org.apache.flink.runtime.jobgraph.RestoreMode");
} catch (ClassNotFoundException e) {
- throw new RuntimeException(e);
+ try {
+ return Class.forName("org.apache.flink.core.execution.RecoveryClaimMode");
+ } catch (ClassNotFoundException classNotFoundException) {
+ throw new RuntimeException(classNotFoundException);
+ }
}
}
}
diff --git a/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java b/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java
index b9f42cefc..ec7084a91 100644
--- a/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java
+++ b/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java
@@ -17,10 +17,10 @@
package org.apache.flink.cdc.cli;
+import org.apache.flink.cdc.cli.utils.ConfigurationUtils;
import org.apache.flink.cdc.composer.PipelineComposer;
import org.apache.flink.cdc.composer.PipelineExecution;
import org.apache.flink.cdc.composer.definition.PipelineDef;
-import org.apache.flink.core.execution.RestoreMode;
import org.apache.flink.core.fs.Path;
import org.apache.flink.shaded.guava31.com.google.common.io.Resources;
@@ -107,7 +107,8 @@ class CliFrontendTest {
"-n");
assertThat(executor.getFlinkConfig().get(SAVEPOINT_PATH))
.isEqualTo(flinkHome() + "/savepoints/savepoint-1");
- assertThat(executor.getFlinkConfig().get(RESTORE_MODE)).isEqualTo(RestoreMode.NO_CLAIM);
+ assertThat(executor.getFlinkConfig().get(RESTORE_MODE))
+ .isEqualTo(getRestoreModeEnum("NO_CLAIM"));
assertThat(executor.getFlinkConfig().get(SAVEPOINT_IGNORE_UNCLAIMED_STATE)).isTrue();
}
@@ -225,6 +226,10 @@ class CliFrontendTest {
return CliFrontend.createExecutor(parser.parse(cliOptions, args));
}
+ private Object getRestoreModeEnum(String name) throws Exception {
+ return Enum.valueOf((Class<Enum>) ConfigurationUtils.getClaimModeClass(), name);
+ }
+
private String pipelineDef() throws Exception {
URL resource = Resources.getResource("definitions/pipeline-definition-full.yaml");
return Paths.get(resource.toURI()).toString();
diff --git a/flink-cdc-common/pom.xml b/flink-cdc-common/pom.xml
index 95f7d5fe0..4e56ca8d4 100644
--- a/flink-cdc-common/pom.xml
+++ b/flink-cdc-common/pom.xml
@@ -44,4 +44,37 @@ limitations under the License.
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>flink2</id>
+ <properties>
+ <flink.version>${flink.2.x.version}</flink.version>
+ </properties>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <configuration>
+ <excludes>
+ <!-- SinkFunction is moved to legacy package in Flink 2.x -->
+ <exclude>org/apache/flink/cdc/common/sink/FlinkSinkFunctionProvider.java</exclude>
+ <!-- SourceFunction is moved to legacy package in Flink 2.x -->
+ <exclude>org/apache/flink/cdc/common/source/FlinkSourceFunctionProvider.java</exclude>
+ </excludes>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+
</project>
\ No newline at end of file
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/ValidationException.java
similarity index 58%
copy from flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
copy to flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/ValidationException.java
index ec1d34b67..adfbbfa16 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/ValidationException.java
@@ -15,16 +15,26 @@
* limitations under the License.
*/
-package org.apache.flink.cdc.common.sink;
+package org.apache.flink.cdc.common.exceptions;
-import org.apache.flink.api.connector.sink2.Sink;
import org.apache.flink.cdc.common.annotation.PublicEvolving;
-import org.apache.flink.streaming.api.functions.sink.SinkFunction;
+
+import javax.annotation.Nullable;
/**
- * A marker interface used to provide an event sink for writing change events to external systems.
- * We can reuse exiting Flink {@link Sink} and Flink {@link SinkFunction} implementation, and we can
- * support our own {@code EventSink} Implementation in the future.
+ * Exception for validation errors (e.g. invalid options, unsupported options). Defined in
+ * flink-cdc-common to avoid runtime dependency on Flink table API (e.g. {@code
+ * org.apache.flink.table.api.ValidationException}), which may not be on the classpath when running
+ * with different Flink versions.
*/
@PublicEvolving
-public interface EventSinkProvider {}
+public class ValidationException extends RuntimeException {
+
+ public ValidationException(String message) {
+ super(message);
+ }
+
+ public ValidationException(String message, @Nullable Throwable cause) {
+ super(message, cause);
+ }
+}
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/factories/FactoryHelper.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/factories/FactoryHelper.java
index 8e4ff0227..f9b53c973 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/factories/FactoryHelper.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/factories/FactoryHelper.java
@@ -21,9 +21,9 @@ import org.apache.flink.cdc.common.annotation.PublicEvolving;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.configuration.FallbackKey;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.utils.Preconditions;
import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.table.api.ValidationException;
import java.util.Arrays;
import java.util.HashMap;
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
index ec1d34b67..4e1114971 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
@@ -19,7 +19,6 @@ package org.apache.flink.cdc.common.sink;
import org.apache.flink.api.connector.sink2.Sink;
import org.apache.flink.cdc.common.annotation.PublicEvolving;
-import org.apache.flink.streaming.api.functions.sink.SinkFunction;
/**
* A marker interface used to provide an event sink for writing change events to external systems.
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source/EventSourceProvider.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source/EventSourceProvider.java
index bf33f839e..f94747f30 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source/EventSourceProvider.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source/EventSourceProvider.java
@@ -19,7 +19,6 @@ package org.apache.flink.cdc.common.source;
import org.apache.flink.api.connector.source.Source;
import org.apache.flink.cdc.common.annotation.PublicEvolving;
-import org.apache.flink.streaming.api.functions.source.SourceFunction;
/**
* A marker interface used to provide an event source for reading events from external systems. We
diff --git a/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/factories/FactoryHelperTests.java b/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/factories/FactoryHelperTests.java
index a7be9c3f5..d04d958a9 100644
--- a/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/factories/FactoryHelperTests.java
+++ b/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/factories/FactoryHelperTests.java
@@ -20,7 +20,7 @@ package org.apache.flink.cdc.common.factories;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.ConfigOptions;
import org.apache.flink.cdc.common.configuration.Configuration;
-import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.Sets;
diff --git a/flink-cdc-composer/pom.xml b/flink-cdc-composer/pom.xml
index ca2812acf..37210f834 100644
--- a/flink-cdc-composer/pom.xml
+++ b/flink-cdc-composer/pom.xml
@@ -27,6 +27,10 @@ limitations under the License.
<artifactId>flink-cdc-composer</artifactId>
+ <properties>
+ <values.connector.artifactId>flink-cdc-pipeline-connector-values</values.connector.artifactId>
+ </properties>
+
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
@@ -40,7 +44,7 @@ limitations under the License.
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-pipeline-connector-values</artifactId>
+ <artifactId>${values.connector.artifactId}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
@@ -96,4 +100,36 @@ limitations under the License.
</dependency>
</dependencies>
+ <profiles>
+ <profile>
+ <id>flink2</id>
+ <properties>
+ <flink.version>${flink.2.x.version}</flink.version>
+ <values.connector.artifactId>flink-cdc-pipeline-connector-values-2.x</values.connector.artifactId>
+ </properties>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <configuration>
+ <testExcludes>
+ <!-- Sink V2 topology interfaces (WithPreWriteTopology etc.) are removed in Flink 2.x -->
+ <testExclude>org/apache/flink/cdc/composer/flink/translator/DataSinkTranslatorTest.java</testExclude>
+ </testExcludes>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+
</project>
\ No newline at end of file
diff --git a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
index aa530300b..c53d9f275 100644
--- a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
+++ b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
@@ -19,7 +19,6 @@ package org.apache.flink.cdc.composer.flink.translator;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.connector.sink2.Sink;
-import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.annotation.VisibleForTesting;
import org.apache.flink.cdc.common.configuration.Configuration;
@@ -28,38 +27,59 @@ import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.common.sink.EventSinkProvider;
-import org.apache.flink.cdc.common.sink.FlinkSinkFunctionProvider;
import org.apache.flink.cdc.common.sink.FlinkSinkProvider;
import org.apache.flink.cdc.composer.definition.SinkDef;
import org.apache.flink.cdc.composer.flink.FlinkEnvironmentUtils;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
-import org.apache.flink.cdc.runtime.operators.sink.BatchDataSinkFunctionOperator;
-import org.apache.flink.cdc.runtime.operators.sink.DataSinkFunctionOperator;
import org.apache.flink.cdc.runtime.operators.sink.DataSinkWriterOperatorFactory;
import org.apache.flink.core.io.SimpleVersionedSerializer;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
-import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
-import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
-import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-import org.apache.flink.streaming.api.functions.sink.SinkFunction;
import org.apache.flink.streaming.api.operators.OneInputStreamOperatorFactory;
-import org.apache.flink.streaming.api.operators.StreamSink;
-import org.apache.flink.streaming.api.transformations.LegacySinkTransformation;
-import org.apache.flink.streaming.api.transformations.PhysicalTransformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
/** Translator used to build {@link DataSink} for given {@link DataStream}. */
@Internal
public class DataSinkTranslator {
+ private static final Logger LOG = LoggerFactory.getLogger(DataSinkTranslator.class);
+
private static final String SINK_WRITER_PREFIX = "Sink Writer: ";
private static final String SINK_COMMITTER_PREFIX = "Sink Committer: ";
+ private static final String FLINK_SINK_FUNCTION_PROVIDER_CLASS =
+ "org.apache.flink.cdc.common.sink.FlinkSinkFunctionProvider";
+ private static final String SINK_FUNCTION_CLASS =
+ "org.apache.flink.streaming.api.functions.sink.SinkFunction";
+ private static final String STREAM_SINK_CLASS =
+ "org.apache.flink.streaming.api.operators.StreamSink";
+ private static final String DATA_SINK_FUNCTION_OPERATOR_CLASS =
+ "org.apache.flink.cdc.runtime.operators.sink.DataSinkFunctionOperator";
+ private static final String BATCH_DATA_SINK_FUNCTION_OPERATOR_CLASS =
+ "org.apache.flink.cdc.runtime.operators.sink.BatchDataSinkFunctionOperator";
+ private static final String LEGACY_SINK_TRANSFORMATION_CLASS =
+ "org.apache.flink.streaming.api.transformations.LegacySinkTransformation";
+ // Sink V2 commit topology classes (not available in Flink 2.x)
+ private static final String TWO_PHASE_COMMITTING_SINK_CLASS =
+ "org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink";
+ private static final String SUPPORTS_COMMITTER_CLASS =
+ "org.apache.flink.api.connector.sink2.SupportsCommitter";
+ private static final String WITH_PRE_WRITE_TOPOLOGY_CLASS =
+ "org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology";
+ private static final String WITH_PRE_COMMIT_TOPOLOGY_CLASS =
+ "org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology";
+ private static final String WITH_POST_COMMIT_TOPOLOGY_CLASS =
+ "org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology";
+
public DataSink createDataSink(
SinkDef sinkDef, Configuration pipelineConfig, StreamExecutionEnvironment env) {
// Search the data sink factory
@@ -98,20 +118,20 @@ public class DataSinkTranslator {
OperatorUidGenerator operatorUidGenerator) {
// Get sink provider
EventSinkProvider eventSinkProvider = dataSink.getEventSinkProvider();
+ if (eventSinkProvider == null) {
+ return;
+ }
String sinkName = generateSinkName(sinkDef);
if (eventSinkProvider instanceof FlinkSinkProvider) {
- // Sink V2
+ // Sink V2 (works with both Flink 1.x and 2.x)
FlinkSinkProvider sinkProvider = (FlinkSinkProvider) eventSinkProvider;
Sink<Event> sink = sinkProvider.getSink();
sinkTo(input, sink, sinkName, isBatchMode, schemaOperatorID, operatorUidGenerator);
- } else if (eventSinkProvider instanceof FlinkSinkFunctionProvider) {
- // SinkFunction
- FlinkSinkFunctionProvider sinkFunctionProvider =
- (FlinkSinkFunctionProvider) eventSinkProvider;
- SinkFunction<Event> sinkFunction = sinkFunctionProvider.getSinkFunction();
- sinkTo(
+ } else {
+ // Try legacy SinkFunction path via reflection (Flink 1.x only)
+ sinkToLegacy(
input,
- sinkFunction,
+ eventSinkProvider,
sinkName,
isBatchMode,
schemaOperatorID,
@@ -128,12 +148,10 @@ public class DataSinkTranslator {
OperatorID schemaOperatorID,
OperatorUidGenerator operatorUidGenerator) {
DataStream<Event> stream = input;
- // Pre-write topology
- if (sink instanceof WithPreWriteTopology) {
- stream = ((WithPreWriteTopology<Event>) sink).addPreWriteTopology(stream);
- }
+ // Pre-write topology (use reflection since WithPreWriteTopology is removed in Flink 2.x)
+ stream = tryAddPreWriteTopology(sink, stream);
- if (sink instanceof TwoPhaseCommittingSink) {
+ if (isTwoPhaseCommittingSink(sink)) {
addCommittingTopology(
sink, stream, sinkName, isBatchMode, schemaOperatorID, operatorUidGenerator);
} else {
@@ -146,29 +164,138 @@ public class DataSinkTranslator {
}
}
- private void sinkTo(
+ /**
+ * Checks if the sink supports two-phase committing. In Flink 1.x, this is indicated by
+ * implementing {@code TwoPhaseCommittingSink}. In Flink 2.x, the equivalent interface is {@code
+ * SupportsCommitter}.
+ */
+ private static boolean isTwoPhaseCommittingSink(Sink<Event> sink) {
+ // Try Flink 1.x: TwoPhaseCommittingSink
+ try {
+ Class<?> tpcClass = Class.forName(TWO_PHASE_COMMITTING_SINK_CLASS);
+ if (tpcClass.isInstance(sink)) {
+ return true;
+ }
+ } catch (ClassNotFoundException ignored) {
+ // Not available in Flink 2.x
+ }
+ // Try Flink 2.x: SupportsCommitter
+ try {
+ Class<?> scClass = Class.forName(SUPPORTS_COMMITTER_CLASS);
+ if (scClass.isInstance(sink)) {
+ return true;
+ }
+ } catch (ClassNotFoundException ignored) {
+ // Not available in Flink 1.x
+ }
+ return false;
+ }
+
+ /**
+ * Tries to add a pre-write topology via reflection. In Flink 1.x, this checks for {@code
+ * WithPreWriteTopology}. In Flink 2.x, this interface does not exist.
+ */
+ @SuppressWarnings("unchecked")
+ private static DataStream<Event> tryAddPreWriteTopology(
+ Sink<Event> sink, DataStream<Event> stream) {
+ try {
+ Class<?> clazz = Class.forName(WITH_PRE_WRITE_TOPOLOGY_CLASS);
+ if (clazz.isInstance(sink)) {
+ Method method = clazz.getMethod("addPreWriteTopology", DataStream.class);
+ return (DataStream<Event>) method.invoke(sink, stream);
+ }
+ } catch (ClassNotFoundException ignored) {
+ // WithPreWriteTopology not available (Flink 2.x)
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to add pre-write topology via reflection", e);
+ }
+ return stream;
+ }
+
+ /**
+ * Sink to a legacy SinkFunction via reflection. This method handles the case where
+ * FlinkSinkFunctionProvider, SinkFunction, StreamSink, DataSinkFunctionOperator,
+ * BatchDataSinkFunctionOperator, and LegacySinkTransformation are not available in Flink 2.x.
+ */
+ private void sinkToLegacy(
DataStream<Event> input,
- SinkFunction<Event> sinkFunction,
+ EventSinkProvider eventSinkProvider,
String sinkName,
boolean isBatchMode,
OperatorID schemaOperatorID,
OperatorUidGenerator operatorUidGenerator) {
- StreamSink<Event> sinkOperator;
- if (isBatchMode) {
- sinkOperator = new BatchDataSinkFunctionOperator(sinkFunction);
- } else {
- sinkOperator = new DataSinkFunctionOperator(sinkFunction, schemaOperatorID);
+ try {
+ // Check if FlinkSinkFunctionProvider is available
+ Class<?> sinkFuncProviderClass = Class.forName(FLINK_SINK_FUNCTION_PROVIDER_CLASS);
+ if (!sinkFuncProviderClass.isInstance(eventSinkProvider)) {
+ throw new IllegalStateException(
+ String.format(
+ "Unsupported EventSinkProvider type \"%s\"",
+ eventSinkProvider.getClass().getCanonicalName()));
+ }
+
+ // Get SinkFunction via reflection
+ Object sinkFunction =
+ sinkFuncProviderClass.getMethod("getSinkFunction").invoke(eventSinkProvider);
+
+ // Create the operator (StreamSink subclass)
+ Class<?> sinkFunctionClass = Class.forName(SINK_FUNCTION_CLASS);
+ Object sinkOperator;
+ if (isBatchMode) {
+ Class<?> batchOpClass = Class.forName(BATCH_DATA_SINK_FUNCTION_OPERATOR_CLASS);
+ sinkOperator =
+ batchOpClass.getConstructor(sinkFunctionClass).newInstance(sinkFunction);
+ } else {
+ Class<?> opClass = Class.forName(DATA_SINK_FUNCTION_OPERATOR_CLASS);
+ sinkOperator =
+ opClass.getConstructor(sinkFunctionClass, OperatorID.class)
+ .newInstance(sinkFunction, schemaOperatorID);
+ }
+
+ // Create LegacySinkTransformation via reflection
+ Class<?> streamSinkClass = Class.forName(STREAM_SINK_CLASS);
+ Class<?> legacySinkTransClass = Class.forName(LEGACY_SINK_TRANSFORMATION_CLASS);
+ StreamExecutionEnvironment executionEnvironment = input.getExecutionEnvironment();
+
+ // Find the constructor: LegacySinkTransformation(Transformation, String, StreamSink,
+ // int, boolean)
+ Constructor<?> constructor = null;
+ for (Constructor<?> c : legacySinkTransClass.getDeclaredConstructors()) {
+ Class<?>[] paramTypes = c.getParameterTypes();
+ if (paramTypes.length == 5 && streamSinkClass.isAssignableFrom(paramTypes[2])) {
+ constructor = c;
+ break;
+ }
+ }
+ if (constructor == null) {
+ throw new NoSuchMethodException(
+ "Cannot find suitable constructor for LegacySinkTransformation");
+ }
+
+ Object transformation =
+ constructor.newInstance(
+ input.getTransformation(),
+ SINK_WRITER_PREFIX + sinkName,
+ sinkOperator,
+ executionEnvironment.getParallelism(),
+ false);
+
+ // Set UID
+ Method setUidMethod = transformation.getClass().getMethod("setUid", String.class);
+ setUidMethod.invoke(transformation, operatorUidGenerator.generateUid("sink-writer"));
+
+ // Add operator to environment
+ executionEnvironment.addOperator(
+ (org.apache.flink.api.dag.Transformation<?>) transformation);
+
+ } catch (ClassNotFoundException e) {
+ throw new IllegalStateException(
+ "Legacy SinkFunction API is not available. This is expected in Flink 2.x. "
+ + "Please use FlinkSinkProvider (Sink V2 API) instead.",
+ e);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to create legacy sink via reflection", e);
}
- final StreamExecutionEnvironment executionEnvironment = input.getExecutionEnvironment();
- PhysicalTransformation<Event> transformation =
- new LegacySinkTransformation<>(
- input.getTransformation(),
- SINK_WRITER_PREFIX + sinkName,
- sinkOperator,
- executionEnvironment.getParallelism(),
- false);
- transformation.setUid(operatorUidGenerator.generateUid("sink-writer"));
- executionEnvironment.addOperator(transformation);
}
private <CommT> void addCommittingTopology(
@@ -190,10 +317,8 @@ public class DataSinkTranslator {
.uid(operatorUidGenerator.generateUid("sink-writer"));
DataStream<CommittableMessage<CommT>> preCommitted = written;
- if (sink instanceof WithPreCommitTopology) {
- preCommitted =
- ((WithPreCommitTopology<Event, CommT>) sink).addPreCommitTopology(written);
- }
+ // Pre-commit topology (use reflection since WithPreCommitTopology is removed in Flink 2.x)
+ preCommitted = tryAddPreCommitTopology(sink, written, preCommitted);
// TODO: Hard coding checkpoint
boolean isCheckpointingEnabled = true;
@@ -206,8 +331,50 @@ public class DataSinkTranslator {
sink, isBatchMode, isCheckpointingEnabled))
.uid(operatorUidGenerator.generateUid("sink-committer"));
- if (sink instanceof WithPostCommitTopology) {
- ((WithPostCommitTopology<Event, CommT>) sink).addPostCommitTopology(committed);
+ // Post-commit topology (use reflection since WithPostCommitTopology is removed in Flink
+ // 2.x)
+ tryAddPostCommitTopology(sink, committed);
+ }
+
+ /**
+ * Tries to add a pre-commit topology via reflection. In Flink 1.x, this checks for {@code
+ * WithPreCommitTopology}. In Flink 2.x, this interface does not exist.
+ */
+ @SuppressWarnings("unchecked")
+ private static <CommT> DataStream<CommittableMessage<CommT>> tryAddPreCommitTopology(
+ Sink<Event> sink,
+ DataStream<CommittableMessage<CommT>> written,
+ DataStream<CommittableMessage<CommT>> fallback) {
+ try {
+ Class<?> clazz = Class.forName(WITH_PRE_COMMIT_TOPOLOGY_CLASS);
+ if (clazz.isInstance(sink)) {
+ Method method = clazz.getMethod("addPreCommitTopology", DataStream.class);
+ return (DataStream<CommittableMessage<CommT>>) method.invoke(sink, written);
+ }
+ } catch (ClassNotFoundException ignored) {
+ // WithPreCommitTopology not available (Flink 2.x)
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to add pre-commit topology via reflection", e);
+ }
+ return fallback;
+ }
+
+ /**
+ * Tries to add a post-commit topology via reflection. In Flink 1.x, this checks for {@code
+ * WithPostCommitTopology}. In Flink 2.x, this interface does not exist.
+ */
+ private static <CommT> void tryAddPostCommitTopology(
+ Sink<Event> sink, DataStream<CommittableMessage<CommT>> committed) {
+ try {
+ Class<?> clazz = Class.forName(WITH_POST_COMMIT_TOPOLOGY_CLASS);
+ if (clazz.isInstance(sink)) {
+ Method method = clazz.getMethod("addPostCommitTopology", DataStream.class);
+ method.invoke(sink, committed);
+ }
+ } catch (ClassNotFoundException ignored) {
+ // WithPostCommitTopology not available (Flink 2.x)
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to add post-commit topology via reflection", e);
}
}
diff --git a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java
index e4f684f8a..280f7f5ad 100644
--- a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java
+++ b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java
@@ -18,6 +18,7 @@
package org.apache.flink.cdc.composer.flink.translator;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.Event;
@@ -25,7 +26,6 @@ import org.apache.flink.cdc.common.factories.DataSourceFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.source.DataSource;
import org.apache.flink.cdc.common.source.EventSourceProvider;
-import org.apache.flink.cdc.common.source.FlinkSourceFunctionProvider;
import org.apache.flink.cdc.common.source.FlinkSourceProvider;
import org.apache.flink.cdc.composer.definition.SourceDef;
import org.apache.flink.cdc.composer.flink.FlinkEnvironmentUtils;
@@ -35,10 +35,22 @@ import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Method;
+
/** Translator used to build {@link DataSource} which will generate a {@link DataStream}. */
@Internal
public class DataSourceTranslator {
+ private static final Logger LOG = LoggerFactory.getLogger(DataSourceTranslator.class);
+
+ private static final String FLINK_SOURCE_FUNCTION_PROVIDER_CLASS =
+ "org.apache.flink.cdc.common.source.FlinkSourceFunctionProvider";
+ private static final String SOURCE_FUNCTION_CLASS =
+ "org.apache.flink.streaming.api.functions.source.SourceFunction";
+
public DataStream<Event> translate(
SourceDef sourceDef,
DataSource dataSource,
@@ -57,7 +69,7 @@ public class DataSourceTranslator {
StreamExecutionEnvironment env,
EventSourceProvider eventSourceProvider,
SourceDef sourceDef) {
- // Source
+ // Source (New Source API, works with both Flink 1.x and 2.x)
if (eventSourceProvider instanceof FlinkSourceProvider) {
FlinkSourceProvider sourceProvider = (FlinkSourceProvider) eventSourceProvider;
return env.fromSource(
@@ -67,14 +79,13 @@ public class DataSourceTranslator {
new EventTypeInfo());
}
- // SourceFunction
- if (eventSourceProvider instanceof FlinkSourceFunctionProvider) {
- FlinkSourceFunctionProvider sourceFunctionProvider =
- (FlinkSourceFunctionProvider) eventSourceProvider;
- DataStreamSource<Event> stream =
- env.addSource(sourceFunctionProvider.getSourceFunction(), new EventTypeInfo());
- sourceDef.getName().ifPresent(stream::name);
- return stream;
+ // SourceFunction (Legacy API, Flink 1.x only)
+ // Use reflection since FlinkSourceFunctionProvider and SourceFunction are not available
+ // in Flink 2.x builds.
+ DataStreamSource<Event> legacyResult =
+ tryCreateLegacyDataStreamSource(env, eventSourceProvider, sourceDef);
+ if (legacyResult != null) {
+ return legacyResult;
}
// Unknown provider type
@@ -84,6 +95,42 @@ public class DataSourceTranslator {
eventSourceProvider.getClass().getCanonicalName()));
}
+ @SuppressWarnings("unchecked")
+ private DataStreamSource<Event> tryCreateLegacyDataStreamSource(
+ StreamExecutionEnvironment env,
+ EventSourceProvider eventSourceProvider,
+ SourceDef sourceDef) {
+ try {
+ Class<?> sourceFuncProviderClass = Class.forName(FLINK_SOURCE_FUNCTION_PROVIDER_CLASS);
+ if (!sourceFuncProviderClass.isInstance(eventSourceProvider)) {
+ return null;
+ }
+ // Get the SourceFunction via reflection
+ Object sourceFunction =
+ sourceFuncProviderClass
+ .getMethod("getSourceFunction")
+ .invoke(eventSourceProvider);
+ // Call env.addSource(sourceFunction, typeInfo) via reflection
+ Class<?> sourceFunctionClass = Class.forName(SOURCE_FUNCTION_CLASS);
+ Method addSourceMethod =
+ StreamExecutionEnvironment.class.getMethod(
+ "addSource", sourceFunctionClass, TypeInformation.class);
+ DataStreamSource<Event> stream =
+ (DataStreamSource<Event>)
+ addSourceMethod.invoke(env, sourceFunction, new EventTypeInfo());
+ sourceDef.getName().ifPresent(stream::name);
+ return stream;
+ } catch (ClassNotFoundException e) {
+ LOG.debug(
+ "FlinkSourceFunctionProvider or SourceFunction not available (likely Flink 2.x), "
+ + "skipping legacy source function path.");
+ return null;
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to create legacy data stream source via reflection", e);
+ }
+ }
+
public DataSource createDataSource(
SourceDef sourceDef, Configuration pipelineConfig, StreamExecutionEnvironment env) {
// Search the data source factory
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedDataSource.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedDataSource.java
index dd3952039..910239423 100644
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedDataSource.java
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedDataSource.java
@@ -20,7 +20,7 @@ package org.apache.flink.cdc.composer.testsource.source;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.source.DataSource;
import org.apache.flink.cdc.common.source.EventSourceProvider;
-import org.apache.flink.cdc.common.source.FlinkSourceFunctionProvider;
+import org.apache.flink.cdc.common.source.FlinkSourceProvider;
import org.apache.flink.cdc.common.source.MetadataAccessor;
/** A {@link DataSource} that emits single-table-in-multi-partition data records. */
@@ -37,8 +37,7 @@ public class DistributedDataSource implements DataSource {
@Override
public EventSourceProvider getEventSourceProvider() {
- return FlinkSourceFunctionProvider.of(
- new DistributedSourceFunction(tableCount, distributedTables));
+ return FlinkSourceProvider.of(new DistributedSource(tableCount, distributedTables));
}
@Override
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSource.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSource.java
new file mode 100644
index 000000000..9e012b110
--- /dev/null
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSource.java
@@ -0,0 +1,587 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.composer.testsource.source;
+
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.flink.api.connector.source.SplitEnumerator;
+import org.apache.flink.api.connector.source.SplitEnumeratorContext;
+import org.apache.flink.cdc.common.data.DecimalData;
+import org.apache.flink.cdc.common.data.LocalZonedTimestampData;
+import org.apache.flink.cdc.common.data.TimeData;
+import org.apache.flink.cdc.common.data.TimestampData;
+import org.apache.flink.cdc.common.data.ZonedTimestampData;
+import org.apache.flink.cdc.common.data.binary.BinaryRecordData;
+import org.apache.flink.cdc.common.data.binary.BinaryStringData;
+import org.apache.flink.cdc.common.event.AddColumnEvent;
+import org.apache.flink.cdc.common.event.CreateTableEvent;
+import org.apache.flink.cdc.common.event.DataChangeEvent;
+import org.apache.flink.cdc.common.event.Event;
+import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.schema.Column;
+import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.types.DataType;
+import org.apache.flink.cdc.common.types.DataTypes;
+import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.runtime.typeutils.BinaryRecordDataGenerator;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputViewStreamWrapper;
+import org.apache.flink.core.memory.DataOutputViewStreamWrapper;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.sql.Timestamp;
+import java.time.Instant;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/** Distributed source using FLIP-27 Source API for testing purposes only. */
+public class DistributedSource
+ implements Source<
+ Event,
+ DistributedSource.DistributedSourceSplit,
+ Collection<DistributedSource.DistributedSourceSplit>> {
+
+ private static final long serialVersionUID = 1L;
+
+ private final int numOfTables;
+ private final boolean distributedTables;
+
+ public DistributedSource(int numOfTables, boolean distributedTables) {
+ this.numOfTables = numOfTables;
+ this.distributedTables = distributedTables;
+ }
+
+ @Override
+ public Boundedness getBoundedness() {
+ return Boundedness.BOUNDED;
+ }
+
+ @Override
+ public SplitEnumerator<DistributedSourceSplit, Collection<DistributedSourceSplit>>
+ createEnumerator(SplitEnumeratorContext<DistributedSourceSplit> enumContext) {
+ return new DistributedSourceEnumerator(enumContext, new ArrayList<>());
+ }
+
+ @Override
+ public SplitEnumerator<DistributedSourceSplit, Collection<DistributedSourceSplit>>
+ restoreEnumerator(
+ SplitEnumeratorContext<DistributedSourceSplit> enumContext,
+ Collection<DistributedSourceSplit> checkpoint) {
+ return new DistributedSourceEnumerator(enumContext, new ArrayList<>(checkpoint));
+ }
+
+ @Override
+ public SimpleVersionedSerializer<DistributedSourceSplit> getSplitSerializer() {
+ return new DistributedSourceSplitSerializer();
+ }
+
+ @Override
+ public SimpleVersionedSerializer<Collection<DistributedSourceSplit>>
+ getEnumeratorCheckpointSerializer() {
+ return new DistributedSourceEnumeratorSerializer();
+ }
+
+ @Override
+ public SourceReader<Event, DistributedSourceSplit> createReader(
+ SourceReaderContext readerContext) {
+ return new DistributedSourceReader(readerContext, numOfTables, distributedTables);
+ }
+
+ // ------------------------------ Split ------------------------------
+
+ /** Split for the distributed source, one per subtask. */
+ static class DistributedSourceSplit implements SourceSplit {
+
+ final int splitIndex;
+ final int parallelism;
+ final int eventOffset;
+
+ DistributedSourceSplit(int splitIndex, int parallelism, int eventOffset) {
+ this.splitIndex = splitIndex;
+ this.parallelism = parallelism;
+ this.eventOffset = eventOffset;
+ }
+
+ @Override
+ public String splitId() {
+ return "distributed-split-" + splitIndex;
+ }
+ }
+
+ // ------------------------------ Enumerator ------------------------------
+
+ /** Enumerator that creates one split per subtask and assigns them by subtask ID. */
+ private static class DistributedSourceEnumerator
+ implements SplitEnumerator<DistributedSourceSplit, Collection<DistributedSourceSplit>> {
+
+ private final SplitEnumeratorContext<DistributedSourceSplit> context;
+ private final List<DistributedSourceSplit> pendingSplits;
+
+ DistributedSourceEnumerator(
+ SplitEnumeratorContext<DistributedSourceSplit> context,
+ List<DistributedSourceSplit> pendingSplits) {
+ this.context = context;
+ this.pendingSplits = pendingSplits;
+ }
+
+ @Override
+ public void start() {
+ if (pendingSplits.isEmpty()) {
+ int parallelism = context.currentParallelism();
+ for (int i = 0; i < parallelism; i++) {
+ pendingSplits.add(new DistributedSourceSplit(i, parallelism, 0));
+ }
+ }
+ }
+
+ @Override
+ public void handleSplitRequest(int subtaskId, String requesterHostname) {
+ // No-op: splits are assigned in addReader via push model
+ }
+
+ @Override
+ public void addSplitsBack(List<DistributedSourceSplit> splits, int subtaskId) {
+ pendingSplits.addAll(splits);
+ }
+
+ @Override
+ public void addReader(int subtaskId) {
+ DistributedSourceSplit assigned = null;
+ for (int i = 0; i < pendingSplits.size(); i++) {
+ if (pendingSplits.get(i).splitIndex == subtaskId) {
+ assigned = pendingSplits.remove(i);
+ break;
+ }
+ }
+ if (assigned != null) {
+ context.assignSplit(assigned, subtaskId);
+ }
+ context.signalNoMoreSplits(subtaskId);
+ }
+
+ @Override
+ public Collection<DistributedSourceSplit> snapshotState(long checkpointId) {
+ return new ArrayList<>(pendingSplits);
+ }
+
+ @Override
+ public void close() {
+ // No-op
+ }
+ }
+
+ // ------------------------------ Reader ------------------------------
+
+ /** Reader that generates events using the same logic as the old DistributedSourceFunction. */
+ private static class DistributedSourceReader
+ implements SourceReader<Event, DistributedSourceSplit> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DistributedSourceReader.class);
+
+ private final SourceReaderContext context;
+ private final int numOfTables;
+ private final boolean distributedTables;
+ private final int subTaskId;
+
+ private int parallelism;
+ private List<Event> events;
+ private int currentIndex;
+ private boolean splitReceived;
+ private boolean finished;
+ private CompletableFuture<Void> availability;
+
+ DistributedSourceReader(
+ SourceReaderContext context, int numOfTables, boolean distributedTables) {
+ this.context = context;
+ this.numOfTables = numOfTables;
+ this.distributedTables = distributedTables;
+ this.subTaskId = context.getIndexOfSubtask();
+ this.splitReceived = false;
+ this.finished = false;
+ this.availability = new CompletableFuture<>();
+ }
+
+ @Override
+ public void start() {
+ // Wait for splits to be assigned
+ }
+
+ @Override
+ public InputStatus pollNext(ReaderOutput<Event> output) throws InterruptedException {
+ if (!splitReceived) {
+ return InputStatus.NOTHING_AVAILABLE;
+ }
+ if (currentIndex < events.size()) {
+ Event event = events.get(currentIndex++);
+ LOG.info("{}> Emitting event {}", subTaskId, event);
+ output.collect(event);
+ return currentIndex < events.size()
+ ? InputStatus.MORE_AVAILABLE
+ : InputStatus.END_OF_INPUT;
+ }
+ if (parallelism > 1 && !finished) {
+ // To allow test running correctly, we need to wait for downstream schema
+ // evolutions to finish before closing any subTask.
+ Thread.sleep(10000);
+ finished = true;
+ }
+ return InputStatus.END_OF_INPUT;
+ }
+
+ @Override
+ public List<DistributedSourceSplit> snapshotState(long checkpointId) {
+ if (!splitReceived || finished) {
+ return Collections.emptyList();
+ }
+ return Collections.singletonList(
+ new DistributedSourceSplit(subTaskId, parallelism, currentIndex));
+ }
+
+ @Override
+ public CompletableFuture<Void> isAvailable() {
+ return availability;
+ }
+
+ @Override
+ public void addSplits(List<DistributedSourceSplit> splits) {
+ for (DistributedSourceSplit split : splits) {
+ this.parallelism = split.parallelism;
+ this.events = generateAllEvents(split);
+ this.currentIndex = split.eventOffset;
+ }
+ this.splitReceived = true;
+ this.availability.complete(null);
+ }
+
+ @Override
+ public void notifyNoMoreSplits() {
+ // If no splits were assigned (shouldn't happen), mark as finished
+ if (!splitReceived) {
+ finished = true;
+ availability.complete(null);
+ }
+ }
+
+ @Override
+ public void close() {
+ // No-op
+ }
+
+ // ----- Event generation logic (migrated from DistributedSourceFunction) -----
+
+ private List<Event> generateAllEvents(DistributedSourceSplit split) {
+ int parallelism = split.parallelism;
+ int iotaCounter = 0;
+
+ // Initialize dummy data types (same as original open())
+ Map<DataType, Object> dummyDataTypes = new LinkedHashMap<>();
+ dummyDataTypes.put(DataTypes.BOOLEAN(), true);
+ dummyDataTypes.put(DataTypes.TINYINT(), (byte) 17);
+ dummyDataTypes.put(DataTypes.SMALLINT(), (short) 34);
+ dummyDataTypes.put(DataTypes.INT(), (int) 68);
+ dummyDataTypes.put(DataTypes.BIGINT(), (long) 136);
+ dummyDataTypes.put(DataTypes.FLOAT(), (float) 272.0);
+ dummyDataTypes.put(DataTypes.DOUBLE(), (double) 544.0);
+ dummyDataTypes.put(
+ DataTypes.DECIMAL(17, 11),
+ DecimalData.fromBigDecimal(new BigDecimal("1088.000"), 17, 11));
+ dummyDataTypes.put(DataTypes.CHAR(17), BinaryStringData.fromString("Alice"));
+ dummyDataTypes.put(DataTypes.VARCHAR(17), BinaryStringData.fromString("Bob"));
+ dummyDataTypes.put(DataTypes.BINARY(17), "Cicada".getBytes());
+ dummyDataTypes.put(DataTypes.VARBINARY(17), "Derrida".getBytes());
+ dummyDataTypes.put(DataTypes.TIME(9), TimeData.fromMillisOfDay(64801000));
+ dummyDataTypes.put(
+ DataTypes.TIMESTAMP(9),
+ TimestampData.fromTimestamp(Timestamp.valueOf("2020-07-17 18:00:00")));
+ dummyDataTypes.put(
+ DataTypes.TIMESTAMP_TZ(9),
+ ZonedTimestampData.of(364800000, 123456, "Asia/Shanghai"));
+ dummyDataTypes.put(
+ DataTypes.TIMESTAMP_LTZ(9),
+ LocalZonedTimestampData.fromInstant(toInstant("2019-12-31 18:00:00")));
+
+ // Build tables list
+ List<TableId> tables;
+ if (distributedTables) {
+ tables =
+ IntStream.range(0, numOfTables)
+ .mapToObj(
+ idx ->
+ TableId.tableId(
+ "default_namespace",
+ "default_database",
+ "table_" + idx))
+ .collect(Collectors.toList());
+ } else {
+ tables =
+ IntStream.range(0, numOfTables)
+ .mapToObj(
+ idx ->
+ TableId.tableId(
+ "default_namespace_subtask_" + subTaskId,
+ "default_database",
+ "table_" + idx))
+ .collect(Collectors.toList());
+ }
+
+ // Collect all events
+ List<Event> allEvents = new ArrayList<>();
+ Map<TableId, Schema> headSchemaMap = new HashMap<>();
+
+ // Use a mutable counter wrapper for iota in lambdas
+ int[] iotaRef = {iotaCounter};
+
+ // Phase 1: Create tables and initial data
+ Consumer<TableId> createTableConsumer =
+ tableId -> {
+ Schema initialSchema =
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.STRING())
+ .primaryKey("id")
+ .partitionKey("id")
+ .build();
+ CreateTableEvent createTableEvent =
+ new CreateTableEvent(tableId, initialSchema);
+ headSchemaMap.compute(
+ tableId,
+ (tbl, schema) ->
+ SchemaUtils.applySchemaChangeEvent(
+ schema, createTableEvent));
+ allEvents.add(createTableEvent);
+ for (int i = 0; i < 10; i++) {
+ allEvents.add(
+ DataChangeEvent.insertEvent(
+ tableId,
+ generateBinRec(
+ headSchemaMap.get(tableId),
+ dummyDataTypes,
+ subTaskId,
+ iotaRef)));
+ }
+ };
+
+ if (parallelism > 1) {
+ Collections.shuffle(tables);
+ }
+ tables.forEach(createTableConsumer);
+
+ // Phase 2: Add columns for each data type
+ List<DataType> fullTypes = new ArrayList<>(dummyDataTypes.keySet());
+ if (parallelism > 1) {
+ Collections.shuffle(fullTypes);
+ }
+
+ for (DataType colType : fullTypes) {
+ // Global column
+ if (parallelism > 1) {
+ Collections.shuffle(tables);
+ }
+ for (TableId tableId : tables) {
+ AddColumnEvent addColumnEvent =
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "col_"
+ + colType.getClass()
+ .getSimpleName()
+ .toLowerCase(),
+ colType))));
+ headSchemaMap.compute(
+ tableId,
+ (tbl, schema) ->
+ SchemaUtils.applySchemaChangeEvent(schema, addColumnEvent));
+ allEvents.add(addColumnEvent);
+ allEvents.add(
+ DataChangeEvent.insertEvent(
+ tableId,
+ generateBinRec(
+ headSchemaMap.get(tableId),
+ dummyDataTypes,
+ subTaskId,
+ iotaRef)));
+ }
+
+ // Subtask-specific column
+ if (parallelism > 1) {
+ Collections.shuffle(tables);
+ }
+ for (TableId tableId : tables) {
+ AddColumnEvent addColumnEvent =
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "subtask_"
+ + subTaskId
+ + "_col_"
+ + colType.getClass()
+ .getSimpleName()
+ .toLowerCase(),
+ colType))));
+ headSchemaMap.compute(
+ tableId,
+ (tbl, schema) ->
+ SchemaUtils.applySchemaChangeEvent(schema, addColumnEvent));
+ allEvents.add(addColumnEvent);
+ allEvents.add(
+ DataChangeEvent.insertEvent(
+ tableId,
+ generateBinRec(
+ headSchemaMap.get(tableId),
+ dummyDataTypes,
+ subTaskId,
+ iotaRef)));
+ }
+ }
+
+ return allEvents;
+ }
+
+ private static BinaryRecordData generateBinRec(
+ Schema schema, Map<DataType, Object> dummyDataTypes, int subTaskId, int[] iotaRef) {
+ BinaryRecordDataGenerator generator =
+ new BinaryRecordDataGenerator(
+ schema.getColumnDataTypes().toArray(new DataType[0]));
+
+ int arity = schema.getColumnDataTypes().size();
+ List<DataType> rowTypes = schema.getColumnDataTypes();
+ Object[] rowObjects = new Object[arity];
+
+ for (int i = 0; i < arity; i++) {
+ DataType type = rowTypes.get(i);
+ if (Objects.equals(type, DataTypes.STRING())) {
+ rowObjects[i] =
+ BinaryStringData.fromString(
+ String.format("__$%d$%d$__", subTaskId, iotaRef[0]++));
+ } else {
+ rowObjects[i] = dummyDataTypes.get(type);
+ }
+ }
+ return generator.generate(rowObjects);
+ }
+
+ private static Instant toInstant(String ts) {
+ return Timestamp.valueOf(ts).toLocalDateTime().atZone(ZoneId.of("UTC")).toInstant();
+ }
+ }
+
+ // ------------------------------ Serializers ------------------------------
+
+ /** Serializer for {@link DistributedSourceSplit}. */
+ private static class DistributedSourceSplitSerializer
+ implements SimpleVersionedSerializer<DistributedSourceSplit> {
+
+ private static final int VERSION = 1;
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public byte[] serialize(DistributedSourceSplit split) throws IOException {
+ try (ByteArrayOutputStream bao = new ByteArrayOutputStream(12)) {
+ DataOutputViewStreamWrapper view = new DataOutputViewStreamWrapper(bao);
+ view.writeInt(split.splitIndex);
+ view.writeInt(split.parallelism);
+ view.writeInt(split.eventOffset);
+ return bao.toByteArray();
+ }
+ }
+
+ @Override
+ public DistributedSourceSplit deserialize(int version, byte[] serialized)
+ throws IOException {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(serialized)) {
+ DataInputViewStreamWrapper view = new DataInputViewStreamWrapper(bis);
+ int splitIndex = view.readInt();
+ int parallelism = view.readInt();
+ int eventOffset = view.readInt();
+ return new DistributedSourceSplit(splitIndex, parallelism, eventOffset);
+ }
+ }
+ }
+
+ /** Serializer for the enumerator checkpoint (collection of splits). */
+ private static class DistributedSourceEnumeratorSerializer
+ implements SimpleVersionedSerializer<Collection<DistributedSourceSplit>> {
+
+ private static final int VERSION = 1;
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public byte[] serialize(Collection<DistributedSourceSplit> splits) throws IOException {
+ try (ByteArrayOutputStream bao = new ByteArrayOutputStream(256);
+ DataOutputViewStreamWrapper view = new DataOutputViewStreamWrapper(bao)) {
+ view.writeInt(splits.size());
+ for (DistributedSourceSplit split : splits) {
+ view.writeInt(split.splitIndex);
+ view.writeInt(split.parallelism);
+ view.writeInt(split.eventOffset);
+ }
+ return bao.toByteArray();
+ }
+ }
+
+ @Override
+ public Collection<DistributedSourceSplit> deserialize(int version, byte[] serialized)
+ throws IOException {
+ try (ByteArrayInputStream bis = new ByteArrayInputStream(serialized);
+ DataInputViewStreamWrapper view = new DataInputViewStreamWrapper(bis)) {
+ int size = view.readInt();
+ List<DistributedSourceSplit> splits = new ArrayList<>(size);
+ for (int i = 0; i < size; i++) {
+ int splitIndex = view.readInt();
+ int parallelism = view.readInt();
+ int eventOffset = view.readInt();
+ splits.add(new DistributedSourceSplit(splitIndex, parallelism, eventOffset));
+ }
+ return splits;
+ }
+ }
+ }
+}
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSourceFunction.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSourceFunction.java
deleted file mode 100644
index 58bbe3f42..000000000
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSourceFunction.java
+++ /dev/null
@@ -1,276 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.composer.testsource.source;
-
-import org.apache.flink.cdc.common.data.DecimalData;
-import org.apache.flink.cdc.common.data.LocalZonedTimestampData;
-import org.apache.flink.cdc.common.data.TimeData;
-import org.apache.flink.cdc.common.data.TimestampData;
-import org.apache.flink.cdc.common.data.ZonedTimestampData;
-import org.apache.flink.cdc.common.data.binary.BinaryRecordData;
-import org.apache.flink.cdc.common.data.binary.BinaryStringData;
-import org.apache.flink.cdc.common.event.AddColumnEvent;
-import org.apache.flink.cdc.common.event.CreateTableEvent;
-import org.apache.flink.cdc.common.event.DataChangeEvent;
-import org.apache.flink.cdc.common.event.Event;
-import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.schema.Column;
-import org.apache.flink.cdc.common.schema.Schema;
-import org.apache.flink.cdc.common.types.DataType;
-import org.apache.flink.cdc.common.types.DataTypes;
-import org.apache.flink.cdc.common.utils.SchemaUtils;
-import org.apache.flink.cdc.runtime.typeutils.BinaryRecordDataGenerator;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.math.BigDecimal;
-import java.sql.Timestamp;
-import java.time.Instant;
-import java.time.ZoneId;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.function.Consumer;
-import java.util.stream.Collectors;
-import java.util.stream.IntStream;
-
-/** Distributed source function for testing purposes only. */
-public class DistributedSourceFunction extends RichParallelSourceFunction<Event> {
-
- private static final Logger LOG = LoggerFactory.getLogger(DistributedSourceFunction.class);
-
- private int subTaskId;
- private int parallelism;
-
- private final int numOfTables;
- private final boolean distributedTables;
- private transient Map<DataType, Object> dummyDataTypes;
- private List<TableId> tables;
- private transient int iotaCounter;
-
- public DistributedSourceFunction(int numOfTables, boolean distributedTables) {
- this.numOfTables = numOfTables;
- this.distributedTables = distributedTables;
- }
-
- @Override
- public void open(Configuration parameters) throws Exception {
- super.open(parameters);
- iotaCounter = 0;
- subTaskId = getRuntimeContext().getIndexOfThisSubtask();
- parallelism = getRuntimeContext().getNumberOfParallelSubtasks();
- if (distributedTables) {
- tables =
- IntStream.range(0, numOfTables)
- .mapToObj(
- idx ->
- TableId.tableId(
- "default_namespace",
- "default_database",
- "table_" + idx))
- .collect(Collectors.toList());
- } else {
- tables =
- IntStream.range(0, numOfTables)
- .mapToObj(
- idx ->
- TableId.tableId(
- "default_namespace_subtask_" + subTaskId,
- "default_database",
- "table_" + idx))
- .collect(Collectors.toList());
- }
- dummyDataTypes = new LinkedHashMap<>();
- dummyDataTypes.put(DataTypes.BOOLEAN(), true);
- dummyDataTypes.put(DataTypes.TINYINT(), (byte) 17);
- dummyDataTypes.put(DataTypes.SMALLINT(), (short) 34);
- dummyDataTypes.put(DataTypes.INT(), (int) 68);
- dummyDataTypes.put(DataTypes.BIGINT(), (long) 136);
- dummyDataTypes.put(DataTypes.FLOAT(), (float) 272.0);
- dummyDataTypes.put(DataTypes.DOUBLE(), (double) 544.0);
- dummyDataTypes.put(
- DataTypes.DECIMAL(17, 11),
- DecimalData.fromBigDecimal(new BigDecimal("1088.000"), 17, 11));
- dummyDataTypes.put(DataTypes.CHAR(17), BinaryStringData.fromString("Alice"));
- dummyDataTypes.put(DataTypes.VARCHAR(17), BinaryStringData.fromString("Bob"));
- dummyDataTypes.put(DataTypes.BINARY(17), "Cicada".getBytes());
- dummyDataTypes.put(DataTypes.VARBINARY(17), "Derrida".getBytes());
- dummyDataTypes.put(DataTypes.TIME(9), TimeData.fromMillisOfDay(64801000));
- dummyDataTypes.put(
- DataTypes.TIMESTAMP(9),
- TimestampData.fromTimestamp(Timestamp.valueOf("2020-07-17 18:00:00")));
- dummyDataTypes.put(
- DataTypes.TIMESTAMP_TZ(9),
- ZonedTimestampData.of(364800000, 123456, "Asia/Shanghai"));
- dummyDataTypes.put(
- DataTypes.TIMESTAMP_LTZ(9),
- LocalZonedTimestampData.fromInstant(toInstant("2019-12-31 18:00:00")));
- }
-
- // Generates statically incrementing data, could be used for data integrity verification.
- private BinaryStringData iota() {
- return BinaryStringData.fromString(String.format("__$%d$%d$__", subTaskId, iotaCounter++));
- }
-
- private void sendFromTables(Consumer<TableId> tableIdConsumer) {
- if (parallelism > 1) {
- // Inject a little randomness in multi-parallelism mode
- Collections.shuffle(tables);
- }
- tables.forEach(tableIdConsumer);
- }
-
- @Override
- public void run(SourceContext<Event> context) throws InterruptedException {
- Schema initialSchema =
- Schema.newBuilder()
- .physicalColumn("id", DataTypes.STRING())
- .primaryKey("id")
- .partitionKey("id")
- .build();
-
- Map<TableId, Schema> headSchemaMap = new HashMap<>();
-
- sendFromTables(
- tableId -> {
- CreateTableEvent createTableEvent =
- new CreateTableEvent(tableId, initialSchema);
- headSchemaMap.compute(
- tableId,
- (tbl, schema) ->
- SchemaUtils.applySchemaChangeEvent(schema, createTableEvent));
- collect(context, createTableEvent);
- for (int i = 0; i < 10; i++) {
- collect(
- context,
- DataChangeEvent.insertEvent(
- tableId, generateBinRec(headSchemaMap.get(tableId))));
- }
- });
-
- List<DataType> fullTypes = new ArrayList<>(dummyDataTypes.keySet());
- if (parallelism > 1) {
- // Inject randomness in multi-partition mode
- Collections.shuffle(fullTypes);
- }
- fullTypes.forEach(
- colType -> {
- sendFromTables(
- tableId -> {
- AddColumnEvent addColumnEvent =
- new AddColumnEvent(
- tableId,
- Collections.singletonList(
- new AddColumnEvent.ColumnWithPosition(
- Column.physicalColumn(
- "col_"
- + colType.getClass()
- .getSimpleName()
- .toLowerCase(),
- colType))));
-
- headSchemaMap.compute(
- tableId,
- (tbl, schema) ->
- SchemaUtils.applySchemaChangeEvent(
- schema, addColumnEvent));
- collect(context, addColumnEvent);
- collect(
- context,
- DataChangeEvent.insertEvent(
- tableId,
- generateBinRec(headSchemaMap.get(tableId))));
- });
-
- sendFromTables(
- tableId -> {
- AddColumnEvent addColumnEvent =
- new AddColumnEvent(
- tableId,
- Collections.singletonList(
- new AddColumnEvent.ColumnWithPosition(
- Column.physicalColumn(
- "subtask_"
- + subTaskId
- + "_col_"
- + colType.getClass()
- .getSimpleName()
- .toLowerCase(),
- colType))));
-
- headSchemaMap.compute(
- tableId,
- (tbl, schema) ->
- SchemaUtils.applySchemaChangeEvent(
- schema, addColumnEvent));
- collect(context, addColumnEvent);
- collect(
- context,
- DataChangeEvent.insertEvent(
- tableId,
- generateBinRec(headSchemaMap.get(tableId))));
- });
- });
-
- if (parallelism > 1) {
- // To allow test running correctly, we need to wait for downstream schema evolutions
- // to finish before closing any subTask.
- Thread.sleep(10000);
- }
- }
-
- @Override
- public void cancel() {
- // Do nothing
- }
-
- private BinaryRecordData generateBinRec(Schema schema) {
- BinaryRecordDataGenerator generator =
- new BinaryRecordDataGenerator(schema.getColumnDataTypes().toArray(new DataType[0]));
-
- int arity = schema.getColumnDataTypes().size();
- List<DataType> rowTypes = schema.getColumnDataTypes();
- Object[] rowObjects = new Object[arity];
-
- for (int i = 0; i < arity; i++) {
- DataType type = rowTypes.get(i);
- if (Objects.equals(type, DataTypes.STRING())) {
- rowObjects[i] = iota();
- } else {
- rowObjects[i] = dummyDataTypes.get(type);
- }
- }
- return generator.generate(rowObjects);
- }
-
- private void collect(SourceContext<Event> sourceContext, Event event) {
- LOG.info("{}> Emitting event {}", subTaskId, event);
- sourceContext.collect(event);
- }
-
- private Instant toInstant(String ts) {
- return Timestamp.valueOf(ts).toLocalDateTime().atZone(ZoneId.of("UTC")).toInstant();
- }
-}
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/main/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactory.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/main/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactory.java
index 14229dcc3..c7e34183a 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/main/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactory.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/main/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactory.java
@@ -20,13 +20,13 @@ package org.apache.flink.cdc.connectors.elasticsearch.sink;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.pipeline.PipelineOptions;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.connectors.elasticsearch.config.ElasticsearchSinkOptions;
import org.apache.flink.cdc.connectors.elasticsearch.v2.NetworkConfig;
-import org.apache.flink.table.api.ValidationException;
import org.apache.http.HttpHost;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/test/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/test/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactoryTest.java
index ef3691d31..a5443d5c8 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/test/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/test/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactoryTest.java
@@ -21,11 +21,11 @@ package org.apache.flink.cdc.connectors.elasticsearch.sink;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/test/java/org/apache/flink/cdc/connectors/fluss/factory/FlussDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/test/java/org/apache/flink/cdc/connectors/fluss/factory/FlussDataSinkFactoryTest.java
index 49ec01d30..f84714ac7 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/test/java/org/apache/flink/cdc/connectors/fluss/factory/FlussDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/test/java/org/apache/flink/cdc/connectors/fluss/factory/FlussDataSinkFactoryTest.java
@@ -18,13 +18,13 @@
package org.apache.flink.cdc.connectors.fluss.factory;
import org.apache.flink.cdc.common.configuration.Configuration;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
import org.apache.flink.cdc.connectors.fluss.sink.FlussDataSink;
import org.apache.flink.cdc.connectors.fluss.sink.FlussDataSinkOptions;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-hudi/src/test/java/org/apache/flink/cdc/connectors/hudi/sink/HudiDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-hudi/src/test/java/org/apache/flink/cdc/connectors/hudi/sink/HudiDataSinkFactoryTest.java
index d6c141626..89d77209d 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-hudi/src/test/java/org/apache/flink/cdc/connectors/hudi/sink/HudiDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-hudi/src/test/java/org/apache/flink/cdc/connectors/hudi/sink/HudiDataSinkFactoryTest.java
@@ -19,11 +19,11 @@ package org.apache.flink.cdc.connectors.hudi.sink;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkFactoryTest.java
index 10f89e16d..1752fed14 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkFactoryTest.java
@@ -20,13 +20,13 @@ package org.apache.flink.cdc.connectors.iceberg.sink;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.CreateTableEvent;
import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.common.types.DataTypes;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/test/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/test/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSinkFactoryTest.java
index f7da80dd2..3342166c8 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/test/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/test/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSinkFactoryTest.java
@@ -18,11 +18,11 @@
package org.apache.flink.cdc.connectors.kafka.sink;
import org.apache.flink.cdc.common.configuration.Configuration;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/factory/MySqlDataSourceFactory.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/factory/MySqlDataSourceFactory.java
index 1b3540da0..0ab054379 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/factory/MySqlDataSourceFactory.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/factory/MySqlDataSourceFactory.java
@@ -21,6 +21,7 @@ import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSourceFactory;
import org.apache.flink.cdc.common.factories.Factory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
@@ -39,7 +40,6 @@ import org.apache.flink.cdc.connectors.mysql.table.StartupOptions;
import org.apache.flink.cdc.connectors.mysql.utils.MySqlSchemaUtils;
import org.apache.flink.cdc.connectors.mysql.utils.OptionUtils;
import org.apache.flink.cdc.debezium.table.DebeziumOptions;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.table.catalog.ObjectPath;
import com.mysql.cj.conf.PropertyKey;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlDataSourceFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlDataSourceFactoryTest.java
index 75fb6590d..03f677ec0 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlDataSourceFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlDataSourceFactoryTest.java
@@ -19,10 +19,10 @@ package org.apache.flink.cdc.connectors.mysql.source;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.Factory;
import org.apache.flink.cdc.connectors.mysql.factory.MySqlDataSourceFactory;
import org.apache.flink.cdc.connectors.mysql.testutils.UniqueDatabase;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.table.catalog.ObjectPath;
import org.junit.jupiter.api.Test;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-oracle/src/main/java/org/apache/flink/cdc/connectors/oracle/factory/OracleDataSourceFactory.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-oracle/src/main/java/org/apache/flink/cdc/connectors/oracle/factory/OracleDataSourceFactory.java
index a2b4647fc..7771dcc1f 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-oracle/src/main/java/org/apache/flink/cdc/connectors/oracle/factory/OracleDataSourceFactory.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-oracle/src/main/java/org/apache/flink/cdc/connectors/oracle/factory/OracleDataSourceFactory.java
@@ -21,6 +21,7 @@ import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSourceFactory;
import org.apache.flink.cdc.common.factories.Factory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
@@ -35,7 +36,6 @@ import org.apache.flink.cdc.connectors.oracle.source.config.OracleSourceConfig;
import org.apache.flink.cdc.connectors.oracle.source.config.OracleSourceConfigFactory;
import org.apache.flink.cdc.connectors.oracle.table.OracleReadableMetaData;
import org.apache.flink.cdc.connectors.oracle.utils.OracleSchemaUtils;
-import org.apache.flink.table.api.ValidationException;
import java.time.Duration;
import java.util.ArrayList;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonDataSinkFactoryTest.java
index dcbbdd682..5f3f929ca 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonDataSinkFactoryTest.java
@@ -19,13 +19,13 @@ package org.apache.flink.cdc.connectors.paimon.sink;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.common.sink.FlinkSinkProvider;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
import org.apache.flink.cdc.connectors.paimon.sink.v2.PaimonEventSink;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactory.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactory.java
index 578b6e0f8..569395ea1 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactory.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactory.java
@@ -21,6 +21,7 @@ import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSourceFactory;
import org.apache.flink.cdc.common.factories.Factory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
@@ -34,7 +35,6 @@ import org.apache.flink.cdc.connectors.postgres.source.PostgresSourceBuilder;
import org.apache.flink.cdc.connectors.postgres.source.config.PostgresSourceConfigFactory;
import org.apache.flink.cdc.connectors.postgres.table.PostgreSQLReadableMetadata;
import org.apache.flink.cdc.connectors.postgres.utils.PostgresSchemaUtils;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.table.data.RowData;
import org.slf4j.Logger;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/test/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/test/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactoryTest.java
index c3cce43bc..f3780e5fe 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/test/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/test/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactoryTest.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.connectors.postgres.factory;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.Factory;
import org.apache.flink.cdc.common.source.SupportedMetadataColumn;
import org.apache.flink.cdc.connectors.postgres.PostgresTestBase;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/test/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/test/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksDataSinkFactoryTest.java
index 4e7278254..f00b346d8 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/test/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/test/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksDataSinkFactoryTest.java
@@ -19,11 +19,11 @@ package org.apache.flink.cdc.connectors.starrocks.sink;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-dist/pom.xml b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/pom.xml
similarity index 52%
copy from flink-cdc-dist/pom.xml
copy to flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/pom.xml
index 132bea2cd..ebfee3e24 100644
--- a/flink-cdc-dist/pom.xml
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/pom.xml
@@ -19,113 +19,114 @@ limitations under the License.
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
- <artifactId>flink-cdc-parent</artifactId>
+ <artifactId>flink-cdc-pipeline-connectors</artifactId>
<groupId>org.apache.flink</groupId>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>
- <artifactId>flink-cdc-dist</artifactId>
+ <artifactId>flink-cdc-pipeline-connector-values-2.x</artifactId>
<properties>
- <japicmp.skip>true</japicmp.skip>
+ <flink.version>${flink.2.x.version}</flink.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-common</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-runtime</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-cli</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-composer</artifactId>
+ <artifactId>flink-cdc-pipeline-connector-values</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>test-jar</id>
+ <goals>
+ <goal>test-jar</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${maven.shade.plugin.version}</version>
<executions>
<execution>
- <id>shade-dist</id>
+ <id>shade-flink</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
+ <shadeTestJar>false</shadeTestJar>
+ <artifactSet>
+ <includes>
+ <include>*:*</include>
+ </includes>
+ </artifactSet>
<filters>
<filter>
- <artifact>*:*</artifact>
+ <artifact>org.apache.kafka:*</artifact>
<excludes>
- <exclude>META-INF/*.SF</exclude>
- <exclude>META-INF/*.DSA</exclude>
- <exclude>META-INF/*.RSA</exclude>
+ <exclude>kafka/kafka-version.properties</exclude>
+ <exclude>LICENSE</exclude>
+ <!-- Does not contain anything relevant.
+ Cites a binary dependency on jersey, but this is neither reflected in the
+ dependency graph, nor are any jersey files bundled. -->
+ <exclude>NOTICE</exclude>
+ <exclude>common/**</exclude>
</excludes>
</filter>
</filters>
- <shadeTestJar>false</shadeTestJar>
- <createDependencyReducedPom>false</createDependencyReducedPom>
- <shadedArtifactAttached>false</shadedArtifactAttached>
- <finalName>${project.artifactId}-${revision}</finalName>
- <artifactSet>
- <includes>
- <include>*</include>
- </includes>
- </artifactSet>
<relocations>
<relocation>
- <pattern>org.apache.commons.cli</pattern>
- <shadedPattern>org.apache.flink.cdc.shaded.com.apache.commons.cli</shadedPattern>
+ <pattern>org.apache.kafka</pattern>
+ <shadedPattern>
+ org.apache.flink.cdc.connectors.shaded.org.apache.kafka
+ </shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.google</pattern>
+ <shadedPattern>
+ org.apache.flink.cdc.connectors.shaded.com.google
+ </shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.fasterxml</pattern>
+ <shadedPattern>
+ org.apache.flink.cdc.connectors.shaded.com.fasterxml
+ </shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-assembly-plugin</artifactId>
- <executions>
- <execution>
- <id>dist</id>
- <goals>
- <goal>single</goal>
- </goals>
- <phase>package</phase>
- <configuration>
- <tarLongFileMode>gnu</tarLongFileMode>
- <appendAssemblyId>false</appendAssemblyId>
- <attach>false</attach>
- <finalName>flink-cdc-${project.version}-bin</finalName>
- <descriptors>
- <descriptor>${basedir}/src/main/assembly/assembly.xml</descriptor>
- </descriptors>
- </configuration>
- </execution>
- </executions>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-deploy-plugin</artifactId>
- <configuration>
- <skip>true</skip>
- </configuration>
- </plugin>
</plugins>
</build>
-</project>
+
+ <profiles>
+ <profile>
+ <id>flink2</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+</project>
\ No newline at end of file
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
similarity index 92%
copy from flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
copy to flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
index d5511b7ba..72b5b1fa3 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
@@ -19,6 +19,7 @@ package org.apache.flink.cdc.connectors.values.sink;
import org.apache.flink.api.connector.sink2.Sink;
import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.api.connector.sink2.WriterInitContext;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.data.RecordData;
import org.apache.flink.cdc.common.event.ChangeEvent;
@@ -30,12 +31,12 @@ import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.common.sink.EventSinkProvider;
-import org.apache.flink.cdc.common.sink.FlinkSinkFunctionProvider;
import org.apache.flink.cdc.common.sink.FlinkSinkProvider;
import org.apache.flink.cdc.common.sink.MetadataApplier;
import org.apache.flink.cdc.common.utils.SchemaUtils;
import org.apache.flink.cdc.connectors.values.ValuesDatabase;
+import java.io.IOException;
import java.io.Serializable;
import java.util.HashMap;
import java.util.List;
@@ -67,12 +68,7 @@ public class ValuesDataSink implements DataSink, Serializable {
@Override
public EventSinkProvider getEventSinkProvider() {
- if (SinkApi.SINK_V2.equals(sinkApi)) {
- return FlinkSinkProvider.of(new ValuesSink(materializedInMemory, print));
- } else {
- return FlinkSinkFunctionProvider.of(
- new ValuesDataSinkFunction(materializedInMemory, print));
- }
+ return FlinkSinkProvider.of(new ValuesSink(materializedInMemory, print));
}
@Override
@@ -97,12 +93,12 @@ public class ValuesDataSink implements DataSink, Serializable {
}
@Override
- public SinkWriter<Event> createWriter(InitContext context) {
+ public SinkWriter<Event> createWriter(WriterInitContext context) throws IOException {
return new ValuesSinkWriter(
materializedInMemory,
print,
- context.getSubtaskId(),
- context.getNumberOfParallelSubtasks());
+ context.getTaskInfo().getIndexOfThisSubtask(),
+ context.getTaskInfo().getNumberOfParallelSubtasks());
}
}
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/resources/META-INF/services/org.apache.flink.cdc.common.factories.Factory b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/resources/META-INF/services/org.apache.flink.cdc.common.factories.Factory
new file mode 100644
index 000000000..12658f8d1
--- /dev/null
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/resources/META-INF/services/org.apache.flink.cdc.common.factories.Factory
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+org.apache.flink.cdc.connectors.values.factory.ValuesDataFactory
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
index d5511b7ba..3cc498cfa 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
@@ -19,6 +19,7 @@ package org.apache.flink.cdc.connectors.values.sink;
import org.apache.flink.api.connector.sink2.Sink;
import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.api.connector.sink2.WriterInitContext;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.data.RecordData;
import org.apache.flink.cdc.common.event.ChangeEvent;
@@ -30,12 +31,12 @@ import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.common.sink.EventSinkProvider;
-import org.apache.flink.cdc.common.sink.FlinkSinkFunctionProvider;
import org.apache.flink.cdc.common.sink.FlinkSinkProvider;
import org.apache.flink.cdc.common.sink.MetadataApplier;
import org.apache.flink.cdc.common.utils.SchemaUtils;
import org.apache.flink.cdc.connectors.values.ValuesDatabase;
+import java.io.IOException;
import java.io.Serializable;
import java.util.HashMap;
import java.util.List;
@@ -67,12 +68,7 @@ public class ValuesDataSink implements DataSink, Serializable {
@Override
public EventSinkProvider getEventSinkProvider() {
- if (SinkApi.SINK_V2.equals(sinkApi)) {
- return FlinkSinkProvider.of(new ValuesSink(materializedInMemory, print));
- } else {
- return FlinkSinkFunctionProvider.of(
- new ValuesDataSinkFunction(materializedInMemory, print));
- }
+ return FlinkSinkProvider.of(new ValuesSink(materializedInMemory, print));
}
@Override
@@ -96,13 +92,20 @@ public class ValuesDataSink implements DataSink, Serializable {
this.print = print;
}
- @Override
public SinkWriter<Event> createWriter(InitContext context) {
return new ValuesSinkWriter(
materializedInMemory,
print,
- context.getSubtaskId(),
- context.getNumberOfParallelSubtasks());
+ context.getTaskInfo().getIndexOfThisSubtask(),
+ context.getTaskInfo().getNumberOfParallelSubtasks());
+ }
+
+ public SinkWriter<Event> createWriter(WriterInitContext context) throws IOException {
+ return new ValuesSinkWriter(
+ materializedInMemory,
+ print,
+ context.getTaskInfo().getIndexOfThisSubtask(),
+ context.getTaskInfo().getNumberOfParallelSubtasks());
}
}
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSinkFunction.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSinkFunction.java
deleted file mode 100644
index b33a2d0e9..000000000
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSinkFunction.java
+++ /dev/null
@@ -1,101 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.connectors.values.sink;
-
-import org.apache.flink.api.common.eventtime.Watermark;
-import org.apache.flink.cdc.common.data.RecordData;
-import org.apache.flink.cdc.common.event.ChangeEvent;
-import org.apache.flink.cdc.common.event.CreateTableEvent;
-import org.apache.flink.cdc.common.event.DataChangeEvent;
-import org.apache.flink.cdc.common.event.Event;
-import org.apache.flink.cdc.common.event.SchemaChangeEvent;
-import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.schema.Schema;
-import org.apache.flink.cdc.common.utils.SchemaUtils;
-import org.apache.flink.cdc.connectors.values.ValuesDatabase;
-import org.apache.flink.streaming.api.functions.sink.SinkFunction;
-
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-/** An e2e {@link SinkFunction} implementation that print all {@link DataChangeEvent} out. */
-public class ValuesDataSinkFunction implements SinkFunction<Event> {
- private final boolean materializedInMemory;
-
- private final boolean print;
-
- /**
- * keep the relationship of TableId and Schema as write method may rely on the schema
- * information of DataChangeEvent.
- */
- private final Map<TableId, Schema> schemaMaps;
-
- private final Map<TableId, List<RecordData.FieldGetter>> fieldGetterMaps;
-
- public ValuesDataSinkFunction(boolean materializedInMemory, boolean print) {
- this.materializedInMemory = materializedInMemory;
- this.print = print;
- schemaMaps = new HashMap<>();
- fieldGetterMaps = new HashMap<>();
- }
-
- @Override
- public void invoke(Event event, Context context) throws Exception {
- if (event instanceof SchemaChangeEvent) {
- SchemaChangeEvent schemaChangeEvent = (SchemaChangeEvent) event;
- TableId tableId = schemaChangeEvent.tableId();
- if (event instanceof CreateTableEvent) {
- Schema schema = ((CreateTableEvent) event).getSchema();
- schemaMaps.put(tableId, schema);
- fieldGetterMaps.put(tableId, SchemaUtils.createFieldGetters(schema));
- } else {
- if (!schemaMaps.containsKey(tableId)) {
- throw new RuntimeException("schema of " + tableId + " is not existed.");
- }
- Schema schema =
- SchemaUtils.applySchemaChangeEvent(
- schemaMaps.get(tableId), schemaChangeEvent);
- schemaMaps.put(tableId, schema);
- fieldGetterMaps.put(tableId, SchemaUtils.createFieldGetters(schema));
- }
- } else if (materializedInMemory && event instanceof DataChangeEvent) {
- ValuesDatabase.applyDataChangeEvent((DataChangeEvent) event);
- }
-
- if (print) {
- TableId tableId = ((ChangeEvent) event).tableId();
- // print the detail message to console for verification.
- System.out.println(
- ValuesDataSinkHelper.convertEventToStr(
- event,
- schemaMaps.get(tableId).getColumnDataTypes(),
- fieldGetterMaps.get(tableId)));
- }
- }
-
- @Override
- public void writeWatermark(Watermark watermark) throws Exception {
- SinkFunction.super.writeWatermark(watermark);
- }
-
- @Override
- public void finish() throws Exception {
- SinkFunction.super.finish();
- }
-}
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/pom.xml b/flink-cdc-connect/flink-cdc-pipeline-connectors/pom.xml
index f7afcf980..d1bb4db33 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/pom.xml
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/pom.xml
@@ -42,6 +42,7 @@ limitations under the License.
<module>flink-cdc-pipeline-connector-iceberg</module>
<module>flink-cdc-pipeline-connector-fluss</module>
<module>flink-cdc-pipeline-connector-hudi</module>
+ <module>flink-cdc-pipeline-connector-values-2.x</module>
</modules>
<dependencies>
diff --git a/flink-cdc-dist/pom.xml b/flink-cdc-dist/pom.xml
index 132bea2cd..703e3138c 100644
--- a/flink-cdc-dist/pom.xml
+++ b/flink-cdc-dist/pom.xml
@@ -54,6 +54,19 @@ limitations under the License.
</dependency>
</dependencies>
+ <profiles>
+ <profile>
+ <id>flink2</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-shaded-guava</artifactId>
+ <version>${flink.2.x.shaded.guava.version}</version>
+ </dependency>
+ </dependencies>
+ </profile>
+ </profiles>
+
<build>
<plugins>
<plugin>
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/pom.xml b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/pom.xml
new file mode 100644
index 000000000..2d5a92045
--- /dev/null
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/pom.xml
@@ -0,0 +1,454 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>flink-cdc-e2e-tests</artifactId>
+ <groupId>org.apache.flink</groupId>
+ <version>${revision}</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>flink-cdc-pipeline-e2e-tests-2.x</artifactId>
+
+ <properties>
+ <flink.version>${flink.2.x.version}</flink.version>
+ <mysql.driver.version>8.0.27</mysql.driver.version>
+ <flink.release.download.skip>true</flink.release.download.skip>
+ <skipITs>true</skipITs>
+ <flink.release.name>flink-${flink.version}-bin-scala_${scala.binary.version}.tgz</flink.release.name>
+ <flink.release.mirror>https://dlcdn.apache.org/flink/flink-${flink.version}</flink.release.mirror>
+ <maven.plugin.download.version>1.6.8</maven.plugin.download.version>
+ <jmh.version>1.37</jmh.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-e2e-utils</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Drivers -->
+ <dependency>
+ <groupId>mysql</groupId>
+ <artifactId>mysql-connector-java</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java</artifactId>
+ </exclusion>
+ </exclusions>
+ <version>${mysql.driver.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- CDC connectors test utils - 2.x versions -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-dist</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-mysql-cdc</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-pipeline-connector-values-2.x</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-test-util</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-pipeline-udf-examples</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- testcontainers -->
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers</artifactId>
+ <version>${testcontainers.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <version>${testcontainers.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>mysql</artifactId>
+ <version>${testcontainers.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- benchmark -->
+ <dependency>
+ <groupId>org.openjdk.jmh</groupId>
+ <artifactId>jmh-core</artifactId>
+ <version>${jmh.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.openjdk.jmh</groupId>
+ <artifactId>jmh-generator-annprocess</artifactId>
+ <version>${jmh.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- test dependencies on TestContainers -->
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>postgresql</artifactId>
+ <version>${testcontainers.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- This is for testing Scala UDF.-->
+ <dependency>
+ <groupId>org.scala-lang</groupId>
+ <artifactId>scala-library</artifactId>
+ <version>${scala.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>kafka</artifactId>
+ <version>${testcontainers.version}</version>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- mini yarn -->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <!-- This dependency is no longer shipped with the JDK since Java 9.-->
+ <groupId>jdk.tools</groupId>
+ <artifactId>jdk.tools</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>ch.qos.reload4j</groupId>
+ <artifactId>reload4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-reload4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>jdk.tools</groupId>
+ <artifactId>jdk.tools</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <artifactId>jdk.tools</artifactId>
+ <groupId>jdk.tools</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>log4j</artifactId>
+ <groupId>log4j</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>slf4j-log4j12</artifactId>
+ <groupId>org.slf4j</groupId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-tests</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>test</scope>
+ <type>test-jar</type>
+ <exclusions>
+ <exclusion>
+ <artifactId>jdk.tools</artifactId>
+ <groupId>jdk.tools</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>log4j</artifactId>
+ <groupId>log4j</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>slf4j-log4j12</artifactId>
+ <groupId>org.slf4j</groupId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>test</scope>
+ <type>test-jar</type>
+ <exclusions>
+ <exclusion>
+ <artifactId>jdk.tools</artifactId>
+ <groupId>jdk.tools</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>log4j</artifactId>
+ <groupId>log4j</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>slf4j-log4j12</artifactId>
+ <groupId>org.slf4j</groupId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-yarn</artifactId>
+ <version>${flink.version}</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <testResources>
+ <testResource>
+ <directory>src/test/resources</directory>
+ <excludes>
+ <exclude>**/flink-cdc.sh</exclude>
+ <exclude>**/flink-cdc.yaml</exclude>
+ </excludes>
+ </testResource>
+ </testResources>
+ <plugins>
+ <!--
+ Skip tests in the test phase.
+ Tests will run in the integration-test phase after shading is complete.
+ -->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+ <skip>true</skip>
+ </configuration>
+ </plugin>
+
+ <!--
+ Configure failsafe to run integration tests AFTER the package phase.
+ This ensures tests use the shaded artifacts where Guava and Flink APIs
+ have been relocated.
+ -->
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-failsafe-plugin</artifactId>
+ <version>3.0.0-M5</version>
+ <executions>
+ <execution>
+ <id>end-to-end-tests</id>
+ <goals>
+ <goal>integration-test</goal>
+ <goal>verify</goal>
+ </goals>
+ <configuration>
+ <includes>
+ <include>**/*.*</include>
+ </includes>
+ <excludes>
+ <exclude>**/MysqlE2eWithYarnApplicationITCase.java</exclude>
+ </excludes>
+ <forkCount>1</forkCount>
+ <systemPropertyVariables>
+ <moduleDir>${project.basedir}</moduleDir>
+ </systemPropertyVariables>
+ </configuration>
+ </execution>
+ <execution>
+ <id>run-last-test</id>
+ <goals>
+ <goal>integration-test</goal>
+ <goal>verify</goal>
+ </goals>
+ <configuration>
+ <includes>
+ <include>**/MysqlE2eWithYarnApplicationITCase.java</include>
+ </includes>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.8</version>
+ <configuration>
+ <cacheDirectory>${maven.plugin.download.cache.path}</cacheDirectory>
+ <outputDirectory>${project.build.directory}</outputDirectory>
+ <readTimeOut>60000</readTimeOut>
+ <retries>3</retries>
+ <unpack>true</unpack>
+ </configuration>
+ <executions>
+ <execution>
+ <id>download-flink-release</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <phase>test</phase>
+ <configuration>
+ <skip>${flink.release.download.skip}</skip>
+ <url>${flink.release.mirror}/${flink.release.name}</url>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>copy-jars</id>
+ <phase>test</phase>
+ <goals>
+ <goal>copy</goal>
+ </goals>
+ </execution>
+ <execution>
+ <id>store-classpath-in-target-for-tests</id>
+ <phase>process-test-resources</phase>
+ <goals>
+ <goal>build-classpath</goal>
+ </goals>
+ <configuration>
+ <outputFile>${project.build.directory}/yarn.classpath</outputFile>
+ <excludeGroupIds>org.apache.flink</excludeGroupIds>
+ </configuration>
+ </execution>
+ </executions>
+ <configuration>
+ <artifactItems>
+
+ <artifactItem>
+ <groupId>mysql</groupId>
+ <artifactId>mysql-connector-java</artifactId>
+ <version>${mysql.driver.version}</version>
+ <destFileName>mysql-driver.jar</destFileName>
+ <type>jar</type>
+ <outputDirectory>${project.build.directory}/dependencies
+ </outputDirectory>
+ </artifactItem>
+
+ <artifactItem>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-dist</artifactId>
+ <version>${project.version}</version>
+ <destFileName>flink-cdc-dist.jar</destFileName>
+ <type>jar</type>
+ <outputDirectory>${project.build.directory}/dependencies
+ </outputDirectory>
+ </artifactItem>
+
+ <artifactItem>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-pipeline-connector-values-2.x</artifactId>
+ <version>${project.version}</version>
+ <destFileName>values-cdc-pipeline-connector.jar</destFileName>
+ <type>jar</type>
+ <outputDirectory>${project.build.directory}/dependencies
+ </outputDirectory>
+ </artifactItem>
+
+ <artifactItem>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-pipeline-udf-examples</artifactId>
+ <version>${project.version}</version>
+ <destFileName>udf-examples.jar</destFileName>
+ <type>jar</type>
+ <outputDirectory>${project.build.directory}/dependencies
+ </outputDirectory>
+ </artifactItem>
+
+ <artifactItem>
+ <groupId>org.scala-lang</groupId>
+ <artifactId>scala-library</artifactId>
+ <version>${scala.version}</version>
+ <destFileName>scala-library.jar</destFileName>
+ <type>jar</type>
+ <outputDirectory>${project.build.directory}/dependencies
+ </outputDirectory>
+ </artifactItem>
+ </artifactItems>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+
+ <profiles>
+ <profile>
+ <id>flink2</id>
+ <properties>
+ <flink.release.download.skip>false</flink.release.download.skip>
+ <skipITs>false</skipITs>
+ </properties>
+ </profile>
+ </profiles>
+</project>
\ No newline at end of file
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/ValuesE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/ValuesE2eITCase.java
new file mode 100644
index 000000000..7d863333d
--- /dev/null
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/ValuesE2eITCase.java
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.pipeline.tests;
+
+import org.apache.flink.cdc.pipeline.tests.utils.PipelineTestEnvironment;
+
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+
+/** End-to-end tests for values cdc pipeline job. */
+class ValuesE2eITCase extends PipelineTestEnvironment {
+ private static final Logger LOG = LoggerFactory.getLogger(ValuesE2eITCase.class);
+
+ @Test
+ void testValuesSingleSplitSingleTable() throws Exception {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: values\n"
+ + " event-set.id: SINGLE_SPLIT_SINGLE_TABLE\n"
+ + "\n"
+ + "sink:\n"
+ + " type: values\n"
+ + " print.enabled: true\n"
+ + "\n"
+ + "pipeline:\n"
+ + " parallelism: %d",
+ parallelism);
+
+ submitPipelineJob(pipelineJob);
+ waitUntilJobFinished(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+
+ validateResult(
+ "CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3], op=INSERT, meta=()}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
+ "DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1], after=[], op=DELETE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[2, ], after=[2, x], op=UPDATE, meta=()}");
+ }
+
+ @Test
+ void testValuesSingleSplitMultiTables() throws Exception {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: values\n"
+ + " event-set.id: SINGLE_SPLIT_MULTI_TABLES\n"
+ + "\n"
+ + "sink:\n"
+ + " type: values\n"
+ + " print.enabled: true\n"
+ + "\n"
+ + "pipeline:\n"
+ + " parallelism: %d",
+ parallelism);
+
+ submitPipelineJob(pipelineJob);
+ waitUntilJobFinished(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+
+ validateResult(
+ "CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
+ "CreateTableEvent{tableId=default_namespace.default_schema.table2, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3], op=INSERT, meta=()}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table2, before=[], after=[1, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table2, before=[], after=[2, 2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table2, before=[], after=[3, 3], op=INSERT, meta=()}",
+ "RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
+ "DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1], after=[], op=DELETE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[2, 2], after=[2, x], op=UPDATE, meta=()}");
+ }
+
+ @Test
+ void testValuesMultiSplitsSingleTable() throws Exception {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: values\n"
+ + " event-set.id: MULTI_SPLITS_SINGLE_TABLE\n"
+ + "\n"
+ + "sink:\n"
+ + " type: values\n"
+ + " print.enabled: true\n"
+ + "\n"
+ + "pipeline:\n"
+ + " parallelism: %d",
+ parallelism);
+
+ submitPipelineJob(pipelineJob);
+ waitUntilJobFinished(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+
+ validateResult(
+ "CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[4, 4], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[5, 5], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[6, 6], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[2, 2], after=[], op=DELETE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[4, 4], after=[], op=DELETE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[6, 6], after=[], op=DELETE, meta=()}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, ], after=[1, 1, x], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[3, 3, ], after=[3, 3, x], op=UPDATE, meta=()}");
+ }
+
+ @Test
+ void testValuesSingleSplitSingleTableWithDefaultValue() throws Exception {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: values\n"
+ + " event-set.id: SINGLE_SPLIT_SINGLE_TABLE_WITH_DEFAULT_VALUE\n"
+ + "\n"
+ + "sink:\n"
+ + " type: values\n"
+ + " print.enabled: true\n"
+ + "\n"
+ + "pipeline:\n"
+ + " parallelism: %d",
+ parallelism);
+
+ submitPipelineJob(pipelineJob);
+ waitUntilJobFinished(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+
+ validateResult(
+ "CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3], op=INSERT, meta=()}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
+ "DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1], after=[], op=DELETE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[2, ], after=[2, x], op=UPDATE, meta=()}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`colWithDefault` STRING 'flink', position=LAST, existedColumnName=null}]}",
+ "RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={colWithDefault=newColWithDefault}}");
+ }
+
+ @Test
+ void testValuesSingleSplitSingleBatchTable() throws Exception {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: values\n"
+ + " event-set.id: SINGLE_SPLIT_SINGLE_BATCH_TABLE\n"
+ + "\n"
+ + "sink:\n"
+ + " type: values\n"
+ + " print.enabled: true\n"
+ + "\n"
+ + "pipeline:\n"
+ + " parallelism: %d\n"
+ + " execution.runtime-mode: BATCH",
+ parallelism);
+
+ submitPipelineJob(pipelineJob);
+ waitUntilJobFinished(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+
+ validateResult(
+ "CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3], op=INSERT, meta=()}");
+ }
+
+ @Test
+ void testValuesSingleSplitMultiBatchTable() throws Exception {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: values\n"
+ + " event-set.id: SINGLE_SPLIT_MULTI_BATCH_TABLE\n"
+ + "\n"
+ + "sink:\n"
+ + " type: values\n"
+ + " print.enabled: true\n"
+ + "\n"
+ + "pipeline:\n"
+ + " parallelism: %d\n"
+ + " execution.runtime-mode: BATCH",
+ parallelism);
+
+ submitPipelineJob(pipelineJob);
+ waitUntilJobFinished(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+
+ validateResult(
+ "CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
+ "CreateTableEvent{tableId=default_namespace.default_schema.table2, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table2, before=[], after=[1, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table2, before=[], after=[2, 2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table2, before=[], after=[3, 3], op=INSERT, meta=()}");
+ }
+
+ @Test
+ void testValuesMultiSplitsSingleBatchTable() throws Exception {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: values\n"
+ + " event-set.id: MULTI_SPLITS_SINGLE_BATCH_TABLE\n"
+ + "\n"
+ + "sink:\n"
+ + " type: values\n"
+ + " print.enabled: true\n"
+ + "\n"
+ + "pipeline:\n"
+ + " parallelism: %d\n"
+ + " execution.runtime-mode: BATCH",
+ parallelism);
+
+ submitPipelineJob(pipelineJob);
+ waitUntilJobFinished(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+
+ validateResult(
+ "CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[4, 4], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[5, 5], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[6, 6], op=INSERT, meta=()}");
+ }
+}
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
similarity index 97%
copy from flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
copy to flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
index 641180580..948ae3b8f 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.pipeline.tests.utils;
import org.apache.flink.api.common.JobID;
import org.apache.flink.api.common.JobStatus;
import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.test.utils.TestUtils;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlContainer;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlVersion;
@@ -28,7 +29,6 @@ import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.RestOptions;
import org.apache.flink.runtime.client.JobStatusMessage;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.util.TestLogger;
import com.github.dockerjava.api.DockerClient;
@@ -66,7 +66,6 @@ import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
-import java.util.Objects;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.function.Function;
@@ -162,12 +161,7 @@ public abstract class PipelineTestEnvironment extends TestLogger {
protected String flinkVersion = getFlinkVersion();
public static String getFlinkVersion() {
- String flinkVersion = System.getProperty("specifiedFlinkVersion");
- if (Objects.isNull(flinkVersion)) {
- throw new IllegalArgumentException(
- "No Flink version specified to run this test. Please use -DspecifiedFlinkVersion to pass one.");
- }
- return flinkVersion;
+ return "2.2.0";
}
protected List<String> copyJarToFlinkLib() {
@@ -283,8 +277,6 @@ public abstract class PipelineTestEnvironment extends TestLogger {
containerPaths.add(containerPath);
}
- // Attach default MySQL and Values connectors
- containerPaths.add(version.workDir() + "/lib/mysql-cdc-pipeline-connector.jar");
containerPaths.add(version.workDir() + "/lib/values-cdc-pipeline-connector.jar");
StringBuilder sb = new StringBuilder();
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/TarballFetcher.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/TarballFetcher.java
new file mode 100644
index 000000000..8bd16d805
--- /dev/null
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/TarballFetcher.java
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.pipeline.tests.utils;
+
+import org.apache.flink.cdc.common.test.utils.TestUtils;
+
+import org.apache.commons.io.FileUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.utility.MountableFile;
+
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
+
+/** Obtain and downloads corresponding Flink CDC tarball files. */
+public abstract class TarballFetcher {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TarballFetcher.class);
+
+ public static void fetchAll(GenericContainer<?> container) throws Exception {
+ fetch(container, CdcVersion.values());
+ }
+
+ public static void fetchLatest(GenericContainer<?> container) throws Exception {
+ fetch(container, CdcVersion.SNAPSHOT);
+ }
+
+ public static void fetch(GenericContainer<?> container, CdcVersion... versions)
+ throws Exception {
+ for (CdcVersion version : versions) {
+ TarballFetcher.fetchInternal(container, version);
+ }
+ }
+
+ private static void fetchInternal(GenericContainer<?> container, CdcVersion version)
+ throws Exception {
+ LOG.info("Trying to download CDC tarball @ {}...", version);
+ if (CdcVersion.SNAPSHOT.equals(version)) {
+ LOG.info("CDC {} is a snapshot version, we should fetch it locally...", version);
+
+ container.copyFileToContainer(
+ MountableFile.forHostPath(
+ TestUtils.getResource("flink-cdc.sh", "flink-cdc-dist", "src"), 0755),
+ version.workDir() + "/bin/flink-cdc.sh");
+ container.copyFileToContainer(
+ MountableFile.forHostPath(
+ TestUtils.getResource("flink-cdc.yaml", "flink-cdc-dist", "src"), 0755),
+ version.workDir() + "/conf/flink-cdc.yaml");
+ container.copyFileToContainer(
+ MountableFile.forHostPath(TestUtils.getResource("flink-cdc-dist.jar")),
+ version.workDir() + "/lib/flink-cdc-dist.jar");
+ container.copyFileToContainer(
+ MountableFile.forHostPath(
+ TestUtils.getResource("values-cdc-pipeline-connector.jar")),
+ version.workDir() + "/lib/values-cdc-pipeline-connector.jar");
+
+ } else {
+ LOG.info("CDC {} is a released version, download it from the Internet...", version);
+
+ String containerPath = "/tmp/tarball/" + version.getVersion() + ".tar.gz";
+ downloadAndCopyToContainer(container, version.tarballUrl(), containerPath);
+ container.execInContainer("mkdir", "-p", version.workDir());
+ container.execInContainer(
+ "tar", "-xzvf", containerPath, "-C", version.workDir(), "--strip-components=1");
+
+ downloadAndCopyToContainer(
+ container,
+ version.connectorJarUrl("values"),
+ version.workDir() + "/lib/values-cdc-pipeline-connector.jar");
+ }
+ }
+
+ private static void downloadAndCopyToContainer(
+ GenericContainer<?> container, String url, String containerPath) throws Exception {
+ Path tempFile = Files.createTempFile("download-", ".tmp");
+ FileUtils.copyURLToFile(
+ new URL(url),
+ tempFile.toFile(),
+ (int) Duration.ofMinutes(1).toMillis(),
+ (int) Duration.ofMinutes(5).toMillis());
+ container.copyFileToContainer(MountableFile.forHostPath(tempFile), containerPath);
+ }
+
+ /** Enum for all released Flink CDC version tags. */
+ public enum CdcVersion {
+ V3_1_1("3.1.1"),
+ V3_2_0("3.2.0"),
+ V3_2_1("3.2.1"),
+ V3_3_0("3.3.0"),
+ V3_4_0("3.4.0"),
+ V3_5_0("3.5.0"),
+ SNAPSHOT("SNAPSHOT");
+
+ private final String version;
+
+ CdcVersion(String version) {
+ this.version = version;
+ }
+
+ public String getVersion() {
+ return version;
+ }
+
+ public static List<CdcVersion> getAllVersions() {
+ return Arrays.asList(CdcVersion.values());
+ }
+
+ public static List<CdcVersion> getVersionsSince(CdcVersion version) {
+ return getAllVersions()
+ .subList(getAllVersions().indexOf(version), getAllVersions().size());
+ }
+
+ public String tarballUrl() {
+ return "https://dlcdn.apache.org/flink/flink-cdc-"
+ + version
+ + "/flink-cdc-"
+ + version
+ + "-bin.tar.gz";
+ }
+
+ public String workDir() {
+ return "/tmp/cdc/" + version;
+ }
+
+ public String connectorJarUrl(String name) {
+ return String.format(
+ "https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-%s/%s/flink-cdc-pipeline-connector-%s-%s.jar",
+ name, version, name, version);
+ }
+ }
+}
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/my.cnf b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/my.cnf
new file mode 100644
index 000000000..26d1abad6
--- /dev/null
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/my.cnf
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# For advice on how to change settings please see
+# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
+
+[mysqld]
+#
+# Remove leading # and set to the amount of RAM for the most important data
+# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
+# innodb_buffer_pool_size = 128M
+#
+# Remove leading # to turn on a very important data integrity option: logging
+# changes to the binary log between backups.
+# log_bin
+#
+# Remove leading # to set options mainly useful for reporting servers.
+# The server defaults are faster for transactions and fast SELECTs.
+# Adjust sizes as needed, experiment to find the optimal values.
+# join_buffer_size = 128M
+# sort_buffer_size = 2M
+# read_rnd_buffer_size = 2M
+skip-host-cache
+skip-name-resolve
+#datadir=/var/lib/mysql
+#socket=/var/lib/mysql/mysql.sock
+#secure-file-priv=/var/lib/mysql-files
+secure-file-priv=/var/lib/mysql
+user=mysql
+
+# Disabling symbolic-links is recommended to prevent assorted security risks
+symbolic-links=0
+
+#log-error=/var/log/mysqld.log
+#pid-file=/var/run/mysqld/mysqld.pid
+
+# ----------------------------------------------
+# Enable the binlog for replication & CDC
+# ----------------------------------------------
+
+# Enable binary replication log and set the prefix, expiration, and log format.
+# The prefix is arbitrary, expiration can be short for integration tests but would
+# be longer on a production system. Row-level info is required for ingest to work.
+# Server ID is required, but this will vary on production systems
+server-id = 223344
+log_bin = mysql-bin
+expire_logs_days = 1
+binlog_format = row
+
+# enable gtid mode
+gtid_mode = on
+enforce_gtid_consistency = on
\ No newline at end of file
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/setup.sql b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/setup.sql
new file mode 100644
index 000000000..3c0cccb4b
--- /dev/null
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/setup.sql
@@ -0,0 +1,30 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+-- In production you would almost certainly limit the replication user must be on the follower (slave) machine,
+-- to prevent other clients accessing the log from other machines. For example, 'replicator'@'follower.acme.com'.
+-- However, in this database we'll grant 2 users different privileges:
+--
+-- 1) 'flinkuser' - all privileges required by the snapshot reader AND binlog reader (used for testing)
+-- 2) 'mysqluser' - all privileges
+--
+GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO 'flinkuser'@'%';
+CREATE USER 'mysqluser' IDENTIFIED BY 'mysqlpw';
+GRANT ALL PRIVILEGES ON *.* TO 'mysqluser'@'%';
+
+-- ----------------------------------------------------------------------------------------------------------------
+-- DATABASE: emptydb
+-- ----------------------------------------------------------------------------------------------------------------
+CREATE DATABASE emptydb;
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/log4j2-test.properties b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/log4j2-test.properties
new file mode 100644
index 000000000..32df1c025
--- /dev/null
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/log4j2-test.properties
@@ -0,0 +1,25 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+# http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+# Set root logger level to ERROR to not flood build logs
+# set manually to INFO for debugging purposes
+rootLogger.level=ERROR
+rootLogger.appenderRef.test.ref = TestLogger
+
+appender.testlogger.name = TestLogger
+appender.testlogger.type = CONSOLE
+appender.testlogger.target = SYSTEM_ERR
+appender.testlogger.layout.type = PatternLayout
+appender.testlogger.layout.pattern = %-4r [%t] %-5p %c - %m%n
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java
index 59d966bc2..24c72bd2f 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java
@@ -19,13 +19,13 @@ package org.apache.flink.cdc.pipeline.tests;
import org.apache.flink.api.common.JobStatus;
import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.test.utils.TestUtils;
import org.apache.flink.cdc.connectors.maxcompute.options.MaxComputeOptions;
import org.apache.flink.cdc.connectors.maxcompute.utils.MaxComputeUtils;
import org.apache.flink.cdc.pipeline.tests.utils.PipelineTestEnvironment;
import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.runtime.client.JobStatusMessage;
-import org.apache.flink.table.api.ValidationException;
import com.aliyun.odps.Instance;
import com.aliyun.odps.Odps;
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java
index 76936b698..3de85c4d1 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.pipeline.tests;
import org.apache.flink.api.common.JobID;
import org.apache.flink.api.common.JobStatus;
import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.test.utils.TestUtils;
import org.apache.flink.cdc.connectors.mysql.testutils.UniqueDatabase;
import org.apache.flink.cdc.pipeline.tests.utils.PipelineTestEnvironment;
@@ -27,7 +28,6 @@ import org.apache.flink.cdc.pipeline.tests.utils.TarballFetcher;
import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.core.execution.SavepointFormatType;
import org.apache.flink.runtime.client.JobStatusMessage;
-import org.apache.flink.table.api.ValidationException;
import org.apache.hudi.common.model.HoodieTableType;
import org.assertj.core.api.Assertions;
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
index 641180580..fa3ecec7d 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.pipeline.tests.utils;
import org.apache.flink.api.common.JobID;
import org.apache.flink.api.common.JobStatus;
import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.test.utils.TestUtils;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlContainer;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlVersion;
@@ -28,7 +29,6 @@ import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.RestOptions;
import org.apache.flink.runtime.client.JobStatusMessage;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.util.TestLogger;
import com.github.dockerjava.api.DockerClient;
diff --git a/flink-cdc-e2e-tests/flink-cdc-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/utils/FlinkContainerTestEnvironment.java b/flink-cdc-e2e-tests/flink-cdc-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/utils/FlinkContainerTestEnvironment.java
index 96173989c..0d1ae70f6 100644
--- a/flink-cdc-e2e-tests/flink-cdc-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/utils/FlinkContainerTestEnvironment.java
+++ b/flink-cdc-e2e-tests/flink-cdc-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/utils/FlinkContainerTestEnvironment.java
@@ -19,6 +19,7 @@ package org.apache.flink.cdc.connectors.tests.utils;
import org.apache.flink.api.common.JobStatus;
import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.test.utils.TestUtils;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlContainer;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlVersion;
@@ -28,7 +29,6 @@ import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.RestOptions;
import org.apache.flink.runtime.client.JobStatusMessage;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.util.TestLogger;
import com.fasterxml.jackson.core.Version;
diff --git a/flink-cdc-e2e-tests/pom.xml b/flink-cdc-e2e-tests/pom.xml
index 28d16beee..155c6b735 100644
--- a/flink-cdc-e2e-tests/pom.xml
+++ b/flink-cdc-e2e-tests/pom.xml
@@ -33,6 +33,7 @@ limitations under the License.
<module>flink-cdc-e2e-utils</module>
<module>flink-cdc-source-e2e-tests</module>
<module>flink-cdc-pipeline-e2e-tests</module>
+ <module>flink-cdc-pipeline-e2e-tests-2.x</module>
</modules>
<build>
diff --git a/flink-cdc-runtime/pom.xml b/flink-cdc-runtime/pom.xml
index bb1c78fbb..cb38487c3 100644
--- a/flink-cdc-runtime/pom.xml
+++ b/flink-cdc-runtime/pom.xml
@@ -103,4 +103,36 @@ limitations under the License.
</dependency>
</dependencies>
+ <profiles>
+ <profile>
+ <id>flink2</id>
+ <properties>
+ <flink.version>${flink.2.x.version}</flink.version>
+ </properties>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <configuration>
+ <excludes>
+ <!-- SinkFunction is moved to legacy package in Flink 2.x -->
+ <exclude>org/apache/flink/cdc/runtime/operators/sink/DataSinkFunctionOperator.java</exclude>
+ <exclude>org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkFunctionOperator.java</exclude>
+ </excludes>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+
</project>
\ No newline at end of file
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaOperator.java
index 15c0571ba..45f91afaf 100755
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/distributed/SchemaOperator.java
@@ -97,7 +97,7 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
@Override
public void open() throws Exception {
super.open();
- subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+ subTaskId = getRuntimeContext().getTaskInfo().getIndexOfThisSubtask();
upstreamSchemaTable = HashBasedTable.create();
evolvedSchemaMap = new HashMap<>();
tableIdRouter = new TableIdRouter(routingRules);
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/BatchSchemaOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/BatchSchemaOperator.java
index df990425a..da089e6c5 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/BatchSchemaOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/BatchSchemaOperator.java
@@ -30,6 +30,7 @@ import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.sink.MetadataApplier;
import org.apache.flink.cdc.runtime.operators.schema.common.SchemaDerivator;
import org.apache.flink.cdc.runtime.operators.schema.common.SchemaManager;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.streaming.api.graph.StreamConfig;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
@@ -70,7 +71,7 @@ public class BatchSchemaOperator extends AbstractStreamOperator<Event>
public BatchSchemaOperator(
List<RouteRule> routingRules, MetadataApplier metadataApplier, String timezone) {
- this.chainingStrategy = ChainingStrategy.ALWAYS;
+ FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
this.timezone = timezone;
this.routingRules = routingRules;
this.metadataApplier = metadataApplier;
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/SchemaOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/SchemaOperator.java
index 901ba168d..f35d62ce6 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/SchemaOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/regular/SchemaOperator.java
@@ -34,6 +34,7 @@ import org.apache.flink.cdc.runtime.operators.schema.common.SchemaDerivator;
import org.apache.flink.cdc.runtime.operators.schema.common.metrics.SchemaOperatorMetrics;
import org.apache.flink.cdc.runtime.operators.schema.regular.event.SchemaChangeRequest;
import org.apache.flink.cdc.runtime.operators.schema.regular.event.SchemaChangeResponse;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.runtime.operators.coordination.CoordinationRequest;
import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
@@ -109,7 +110,7 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
Duration rpcTimeOut,
SchemaChangeBehavior schemaChangeBehavior,
String timezone) {
- this.chainingStrategy = ChainingStrategy.ALWAYS;
+ FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
this.rpcTimeout = rpcTimeOut;
this.schemaChangeBehavior = schemaChangeBehavior;
this.timezone = timezone;
@@ -131,7 +132,7 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
this.schemaOperatorMetrics =
new SchemaOperatorMetrics(
getRuntimeContext().getMetricGroup(), schemaChangeBehavior);
- this.subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+ this.subTaskId = getRuntimeContext().getTaskInfo().getIndexOfThisSubtask();
this.originalSchemaMap = new HashMap<>();
this.evolvedSchemaMap = new HashMap<>();
this.router = new TableIdRouter(routingRules);
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkFunctionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkFunctionOperator.java
index 560909006..73a57f326 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkFunctionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkFunctionOperator.java
@@ -20,6 +20,7 @@ package org.apache.flink.cdc.runtime.operators.sink;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.event.Event;
import org.apache.flink.cdc.runtime.operators.sink.exception.SinkWrapperException;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.streaming.api.functions.sink.SinkFunction;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.StreamSink;
@@ -37,7 +38,7 @@ public class BatchDataSinkFunctionOperator extends StreamSink<Event> {
public BatchDataSinkFunctionOperator(SinkFunction<Event> userFunction) {
super(userFunction);
- this.chainingStrategy = ChainingStrategy.ALWAYS;
+ FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
}
@Override
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java
index 52900fef7..5f4bd9acf 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java
@@ -30,7 +30,6 @@ import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
import org.apache.flink.streaming.api.graph.StreamConfig;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.BoundedOneInput;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
import org.apache.flink.streaming.api.watermark.Watermark;
@@ -41,6 +40,7 @@ import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
import java.lang.reflect.Constructor;
import java.lang.reflect.Field;
+import java.lang.reflect.Method;
/**
* An operator that processes records to be written into a {@link Sink} in batch mode.
@@ -78,7 +78,6 @@ public class BatchDataSinkWriterOperator<CommT>
this.sink = sink;
this.processingTimeService = processingTimeService;
this.mailboxExecutor = mailboxExecutor;
- this.chainingStrategy = ChainingStrategy.ALWAYS;
}
@Override
@@ -88,40 +87,43 @@ public class BatchDataSinkWriterOperator<CommT>
Output<StreamRecord<CommittableMessage<CommT>>> output) {
super.setup(containingTask, config, output);
flinkWriterOperator = createFlinkWriterOperator();
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .setup(containingTask, config, output);
+ invokeFlinkWriterOperatorMethod(
+ "setup",
+ new Class<?>[] {StreamTask.class, StreamConfig.class, Output.class},
+ containingTask,
+ config,
+ output);
}
@Override
public void open() throws Exception {
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator().open();
+ invokeFlinkWriterOperatorMethod("open", new Class<?>[0]);
copySinkWriter = getFieldValue("sinkWriter");
}
@Override
public void initializeState(StateInitializationContext context) throws Exception {
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .initializeState(context);
+ invokeFlinkWriterOperatorMethod(
+ "initializeState", new Class<?>[] {StateInitializationContext.class}, context);
}
@Override
public void snapshotState(StateSnapshotContext context) throws Exception {
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .snapshotState(context);
+ invokeFlinkWriterOperatorMethod(
+ "snapshotState", new Class<?>[] {StateSnapshotContext.class}, context);
}
@Override
public void processWatermark(Watermark mark) throws Exception {
super.processWatermark(mark);
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .processWatermark(mark);
+ invokeFlinkWriterOperatorMethod("processWatermark", new Class<?>[] {Watermark.class}, mark);
}
@Override
public void processWatermarkStatus(WatermarkStatus watermarkStatus) throws Exception {
super.processWatermarkStatus(watermarkStatus);
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .processWatermarkStatus(watermarkStatus);
+ invokeFlinkWriterOperatorMethod(
+ "processWatermarkStatus", new Class<?>[] {WatermarkStatus.class}, watermarkStatus);
}
@Override
@@ -138,19 +140,18 @@ public class BatchDataSinkWriterOperator<CommT>
@Override
public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .prepareSnapshotPreBarrier(checkpointId);
+ invokeFlinkWriterOperatorMethod(
+ "prepareSnapshotPreBarrier", new Class<?>[] {long.class}, checkpointId);
}
@Override
public void close() throws Exception {
- this.<OneInputStreamOperator<Event, CommittableMessage<CommT>>>getFlinkWriterOperator()
- .close();
+ invokeFlinkWriterOperatorMethod("close", new Class<?>[0]);
}
@Override
public void endInput() throws Exception {
- this.<BoundedOneInput>getFlinkWriterOperator().endInput();
+ invokeFlinkWriterOperatorMethod("endInput", new Class<?>[0]);
}
// ----------------------------- Helper functions -------------------------------
@@ -168,11 +169,68 @@ public class BatchDataSinkWriterOperator<CommT>
.getUserCodeClassLoader()
.loadClass(
"org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator");
- Constructor<?> constructor =
- flinkWriterClass.getDeclaredConstructor(
- Sink.class, ProcessingTimeService.class, MailboxExecutor.class);
- constructor.setAccessible(true);
- return constructor.newInstance(sink, processingTimeService, mailboxExecutor);
+
+ // Try to find a constructor whose first three parameters are compatible with
+ // (Sink, ProcessingTimeService, MailboxExecutor). This makes the code resilient
+ // against Flink 2.x adding extra parameters to the constructor.
+ Constructor<?> target = null;
+ for (Constructor<?> c : flinkWriterClass.getDeclaredConstructors()) {
+ Class<?>[] p = c.getParameterTypes();
+ if (p.length >= 3
+ && Sink.class.isAssignableFrom(p[0])
+ && ProcessingTimeService.class.isAssignableFrom(p[1])
+ && MailboxExecutor.class.isAssignableFrom(p[2])) {
+ target = c;
+ break;
+ }
+ }
+
+ if (target == null) {
+ // Fallback: use the first declared constructor and best-effort argument mapping
+ // below. This covers Flink 2.2 where the constructor signature may have changed.
+ Constructor<?>[] all = flinkWriterClass.getDeclaredConstructors();
+ if (all.length == 0) {
+ throw new RuntimeException(
+ "No constructors found on SinkWriterOperator in Flink runtime");
+ }
+ target = all[0];
+ }
+
+ target.setAccessible(true);
+ Class<?>[] paramTypes = target.getParameterTypes();
+ Object[] args = new Object[paramTypes.length];
+ for (int i = 0; i < paramTypes.length; i++) {
+ Class<?> t = paramTypes[i];
+ if (Sink.class.isAssignableFrom(t)) {
+ args[i] = sink;
+ } else if (ProcessingTimeService.class.isAssignableFrom(t)) {
+ args[i] = processingTimeService;
+ } else if (MailboxExecutor.class.isAssignableFrom(t)) {
+ args[i] = mailboxExecutor;
+ } else if (t.isPrimitive()) {
+ if (t == boolean.class) {
+ args[i] = false;
+ } else if (t == char.class) {
+ args[i] = '\0';
+ } else if (t == byte.class) {
+ args[i] = (byte) 0;
+ } else if (t == short.class) {
+ args[i] = (short) 0;
+ } else if (t == int.class) {
+ args[i] = 0;
+ } else if (t == long.class) {
+ args[i] = 0L;
+ } else if (t == float.class) {
+ args[i] = 0.0f;
+ } else if (t == double.class) {
+ args[i] = 0.0d;
+ }
+ } else {
+ // Best effort: pass null for any extra, unknown parameters.
+ args[i] = null;
+ }
+ }
+ return target.newInstance(args);
} catch (Exception e) {
throw new RuntimeException("Failed to create SinkWriterOperator in Flink", e);
}
@@ -204,4 +262,36 @@ public class BatchDataSinkWriterOperator<CommT>
private <T> T getFlinkWriterOperator() {
return (T) flinkWriterOperator;
}
+
+ private void invokeFlinkWriterOperatorMethod(
+ String methodName, Class<?>[] parameterTypes, Object... args) {
+ try {
+ Method m = findMethod(flinkWriterOperator.getClass(), methodName, parameterTypes);
+ if (m == null) {
+ // Method does not exist in this Flink version (for example, open() signature
+ // changed); ignore for compatibility.
+ return;
+ }
+ m.setAccessible(true);
+ m.invoke(flinkWriterOperator, args);
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to invoke method "
+ + methodName
+ + " on wrapped flink writer operator "
+ + flinkWriterOperator.getClass().getName(),
+ e);
+ }
+ }
+
+ private static Method findMethod(Class<?> clazz, String methodName, Class<?>[] parameterTypes) {
+ while (clazz != null) {
+ try {
+ return clazz.getDeclaredMethod(methodName, parameterTypes);
+ } catch (NoSuchMethodException e) {
+ clazz = clazz.getSuperclass();
+ }
+ }
+ return null;
+ }
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkFunctionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkFunctionOperator.java
index df0e7802b..505f131e1 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkFunctionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkFunctionOperator.java
@@ -26,6 +26,7 @@ import org.apache.flink.cdc.common.event.SchemaChangeEventType;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.runtime.operators.sink.exception.SinkWrapperException;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.state.StateInitializationContext;
import org.apache.flink.streaming.api.functions.sink.SinkFunction;
@@ -62,7 +63,7 @@ public class DataSinkFunctionOperator extends StreamSink<Event> {
super(userFunction);
this.schemaOperatorID = schemaOperatorID;
processedTableIds = new HashSet<>();
- this.chainingStrategy = ChainingStrategy.ALWAYS;
+ FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
}
@Override
@@ -79,7 +80,8 @@ public class DataSinkFunctionOperator extends StreamSink<Event> {
@Override
public void initializeState(StateInitializationContext context) throws Exception {
- schemaEvolutionClient.registerSubtask(getRuntimeContext().getIndexOfThisSubtask());
+ schemaEvolutionClient.registerSubtask(
+ getRuntimeContext().getTaskInfo().getIndexOfThisSubtask());
super.initializeState(context);
}
@@ -135,7 +137,8 @@ public class DataSinkFunctionOperator extends StreamSink<Event> {
});
}
schemaEvolutionClient.notifyFlushSuccess(
- getRuntimeContext().getIndexOfThisSubtask(), event.getSourceSubTaskId());
+ getRuntimeContext().getTaskInfo().getIndexOfThisSubtask(),
+ event.getSourceSubTaskId());
}
private void emitLatestSchema(TableId tableId) throws Exception {
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
index 0432d1f54..6038045dd 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
@@ -36,7 +36,6 @@ import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
import org.apache.flink.streaming.api.graph.StreamConfig;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.BoundedOneInput;
-import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
import org.apache.flink.streaming.api.watermark.Watermark;
@@ -47,6 +46,7 @@ import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
import java.lang.reflect.Constructor;
import java.lang.reflect.Field;
+import java.lang.reflect.Method;
import java.util.HashSet;
import java.util.Optional;
import java.util.Set;
@@ -97,7 +97,6 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
this.mailboxExecutor = mailboxExecutor;
this.schemaOperatorID = schemaOperatorID;
this.processedTableIds = new HashSet<>();
- this.chainingStrategy = ChainingStrategy.ALWAYS;
}
@Override
@@ -107,8 +106,14 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
Output<StreamRecord<CommittableMessage<CommT>>> output) {
super.setup(containingTask, config, output);
flinkWriterOperator = createFlinkWriterOperator();
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .setup(containingTask, config, output);
+ // Call protected AbstractStreamOperator.setup(...) on the wrapped operator via reflection
+ // to avoid JVM verifier issues with protected access across packages.
+ invokeFlinkWriterOperatorMethod(
+ "setup",
+ new Class<?>[] {StreamTask.class, StreamConfig.class, Output.class},
+ containingTask,
+ config,
+ output);
schemaEvolutionClient =
new SchemaEvolutionClient(
containingTask.getEnvironment().getOperatorCoordinatorEventGateway(),
@@ -117,35 +122,109 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
@Override
public void open() throws Exception {
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator().open();
+ invokeFlinkWriterOperatorMethod("open", new Class<?>[0]);
copySinkWriter = getFieldValue("sinkWriter");
+ ensureEmitDownstream();
+ }
+
+ /**
+ * Flink 2.2 {@code SinkWriterOperator} sets {@code emitDownstream = sink instanceof
+ * SupportsCommitter}. Sinks like Paimon implement the older {@code TwoPhaseCommittingSink}
+ * rather than Flink 2.x {@code SupportsCommitter}, so committables are silently discarded. This
+ * method forces the flag to {@code true} and fills in the committable serializer so that the
+ * wrapped operator properly emits committables to the downstream committer.
+ *
+ * <p>This is only applied when the sink actually supports committing (implements either {@code
+ * TwoPhaseCommittingSink} or {@code SupportsCommitter}). For sinks that do not support
+ * committing, forcing {@code emitDownstream = true} would cause a {@link ClassCastException}
+ * when the wrapped operator tries to cast the writer to {@code CommittingSinkWriter}.
+ */
+ private void ensureEmitDownstream() {
+ if (!sinkSupportsCommitting()) {
+ return;
+ }
+ try {
+ Field emitField = findField(flinkWriterOperator.getClass(), "emitDownstream");
+ if (emitField == null) {
+ return;
+ }
+ emitField.setAccessible(true);
+ if (!emitField.getBoolean(flinkWriterOperator)) {
+ emitField.setBoolean(flinkWriterOperator, true);
+
+ Field serField = findField(flinkWriterOperator.getClass(), "committableSerializer");
+ if (serField != null) {
+ serField.setAccessible(true);
+ if (serField.get(flinkWriterOperator) == null) {
+ try {
+ Method getSerializer =
+ sink.getClass().getMethod("getCommittableSerializer");
+ getSerializer.setAccessible(true);
+ serField.set(flinkWriterOperator, getSerializer.invoke(sink));
+ } catch (NoSuchMethodException ignored) {
+ }
+ }
+ }
+ }
+ } catch (Exception e) {
+ LOG.warn("Could not force emitDownstream on wrapped SinkWriterOperator", e);
+ }
+ }
+
+ private boolean sinkSupportsCommitting() {
+ try {
+ Class<?> tpc =
+ Class.forName("org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink");
+ if (tpc.isInstance(sink)) {
+ return true;
+ }
+ } catch (ClassNotFoundException ignored) {
+ }
+ try {
+ Class<?> sc = Class.forName("org.apache.flink.api.connector.sink2.SupportsCommitter");
+ if (sc.isInstance(sink)) {
+ return true;
+ }
+ } catch (ClassNotFoundException ignored) {
+ }
+ return false;
+ }
+
+ private static Field findField(Class<?> clazz, String name) {
+ while (clazz != null) {
+ try {
+ return clazz.getDeclaredField(name);
+ } catch (NoSuchFieldException e) {
+ clazz = clazz.getSuperclass();
+ }
+ }
+ return null;
}
@Override
public void initializeState(StateInitializationContext context) throws Exception {
- schemaEvolutionClient.registerSubtask(getRuntimeContext().getIndexOfThisSubtask());
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .initializeState(context);
+ schemaEvolutionClient.registerSubtask(getSubtaskIndexCompat());
+ invokeFlinkWriterOperatorMethod(
+ "initializeState", new Class<?>[] {StateInitializationContext.class}, context);
}
@Override
public void snapshotState(StateSnapshotContext context) throws Exception {
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .snapshotState(context);
+ invokeFlinkWriterOperatorMethod(
+ "snapshotState", new Class<?>[] {StateSnapshotContext.class}, context);
}
@Override
public void processWatermark(Watermark mark) throws Exception {
super.processWatermark(mark);
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .processWatermark(mark);
+ invokeFlinkWriterOperatorMethod("processWatermark", new Class<?>[] {Watermark.class}, mark);
}
@Override
public void processWatermarkStatus(WatermarkStatus watermarkStatus) throws Exception {
super.processWatermarkStatus(watermarkStatus);
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .processWatermarkStatus(watermarkStatus);
+ invokeFlinkWriterOperatorMethod(
+ "processWatermarkStatus", new Class<?>[] {WatermarkStatus.class}, watermarkStatus);
}
@Override
@@ -188,19 +267,18 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
@Override
public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
- this.<AbstractStreamOperator<CommittableMessage<CommT>>>getFlinkWriterOperator()
- .prepareSnapshotPreBarrier(checkpointId);
+ invokeFlinkWriterOperatorMethod(
+ "prepareSnapshotPreBarrier", new Class<?>[] {long.class}, checkpointId);
}
@Override
public void close() throws Exception {
- this.<OneInputStreamOperator<Event, CommittableMessage<CommT>>>getFlinkWriterOperator()
- .close();
+ invokeFlinkWriterOperatorMethod("close", new Class<?>[0]);
}
@Override
public void endInput() throws Exception {
- this.<BoundedOneInput>getFlinkWriterOperator().endInput();
+ invokeFlinkWriterOperatorMethod("endInput", new Class<?>[0]);
}
// ----------------------------- Helper functions -------------------------------
@@ -223,7 +301,7 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
});
}
schemaEvolutionClient.notifyFlushSuccess(
- getRuntimeContext().getIndexOfThisSubtask(), event.getSourceSubTaskId());
+ getSubtaskIndexCompat(), event.getSourceSubTaskId());
}
private void emitLatestSchema(TableId tableId) throws Exception {
@@ -250,16 +328,106 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
.getUserCodeClassLoader()
.loadClass(
"org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator");
- Constructor<?> constructor =
- flinkWriterClass.getDeclaredConstructor(
- Sink.class, ProcessingTimeService.class, MailboxExecutor.class);
- constructor.setAccessible(true);
- return constructor.newInstance(sink, processingTimeService, mailboxExecutor);
+
+ // Try to find a constructor whose first three parameters are compatible with
+ // (Sink, ProcessingTimeService, MailboxExecutor). This makes the code resilient
+ // against Flink 2.x adding extra parameters to the constructor.
+ Constructor<?> target = null;
+ for (Constructor<?> c : flinkWriterClass.getDeclaredConstructors()) {
+ Class<?>[] p = c.getParameterTypes();
+ if (p.length >= 3
+ && Sink.class.isAssignableFrom(p[0])
+ && ProcessingTimeService.class.isAssignableFrom(p[1])
+ && MailboxExecutor.class.isAssignableFrom(p[2])) {
+ target = c;
+ break;
+ }
+ }
+
+ if (target == null) {
+ // Fallback: use the first declared constructor and best-effort argument mapping
+ // below. This covers Flink 2.2 where the constructor signature may have changed.
+ Constructor<?>[] all = flinkWriterClass.getDeclaredConstructors();
+ if (all.length == 0) {
+ throw new RuntimeException(
+ "No constructors found on SinkWriterOperator in Flink runtime");
+ }
+ target = all[0];
+ }
+
+ target.setAccessible(true);
+ Class<?>[] paramTypes = target.getParameterTypes();
+ Object[] args = new Object[paramTypes.length];
+ for (int i = 0; i < paramTypes.length; i++) {
+ Class<?> t = paramTypes[i];
+ if (Sink.class.isAssignableFrom(t)) {
+ args[i] = sink;
+ } else if (ProcessingTimeService.class.isAssignableFrom(t)) {
+ args[i] = processingTimeService;
+ } else if (MailboxExecutor.class.isAssignableFrom(t)) {
+ args[i] = mailboxExecutor;
+ } else if (t.isPrimitive()) {
+ if (t == boolean.class) {
+ args[i] = false;
+ } else if (t == char.class) {
+ args[i] = '\0';
+ } else if (t == byte.class) {
+ args[i] = (byte) 0;
+ } else if (t == short.class) {
+ args[i] = (short) 0;
+ } else if (t == int.class) {
+ args[i] = 0;
+ } else if (t == long.class) {
+ args[i] = 0L;
+ } else if (t == float.class) {
+ args[i] = 0.0f;
+ } else if (t == double.class) {
+ args[i] = 0.0d;
+ }
+ } else {
+ // Best effort: pass null for any extra, unknown parameters.
+ args[i] = null;
+ }
+ }
+ return target.newInstance(args);
} catch (Exception e) {
throw new RuntimeException("Failed to create SinkWriterOperator in Flink", e);
}
}
+ /**
+ * Obtains the subtask index in a way that is compatible with both Flink 1.x and 2.x.
+ *
+ * <p>Flink 2.x removed {@code StreamingRuntimeContext.getIndexOfThisSubtask()}. The correct
+ * replacement is {@code getRuntimeContext().getTaskInfo().getIndexOfThisSubtask()}.
+ *
+ * <p><b>IMPORTANT:</b> Returning a wrong (fixed) index here causes ALL sink subtasks to report
+ * the same ID to SchemaCoordinator. The coordinator then waits forever for the missing subtask
+ * IDs, causing a 3-minute timeout in SchemaOperator.
+ */
+ private int getSubtaskIndexCompat() {
+ try {
+ Object ctx = getRuntimeContext();
+ // Flink 2.x: getTaskInfo().getIndexOfThisSubtask()
+ try {
+ Method getTaskInfo = ctx.getClass().getMethod("getTaskInfo");
+ getTaskInfo.setAccessible(true);
+ Object taskInfo = getTaskInfo.invoke(ctx);
+ Method getIndex = taskInfo.getClass().getMethod("getIndexOfThisSubtask");
+ getIndex.setAccessible(true);
+ return (Integer) getIndex.invoke(taskInfo);
+ } catch (NoSuchMethodException ignored) {
+ // fall through to Flink 1.x path
+ }
+ // Flink 1.x: direct getIndexOfThisSubtask()
+ Method m = ctx.getClass().getMethod("getIndexOfThisSubtask");
+ m.setAccessible(true);
+ return (Integer) m.invoke(ctx);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to obtain subtask index from RuntimeContext", e);
+ }
+ }
+
/**
* Finds a field by name from its declaring class. This also searches for the field in super
* classes.
@@ -286,4 +454,36 @@ public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<Commit
private <T> T getFlinkWriterOperator() {
return (T) flinkWriterOperator;
}
+
+ private void invokeFlinkWriterOperatorMethod(
+ String methodName, Class<?>[] parameterTypes, Object... args) {
+ try {
+ Method m = findMethod(flinkWriterOperator.getClass(), methodName, parameterTypes);
+ if (m == null) {
+ // Method does not exist in this Flink version (for example, open() signature
+ // changed); ignore for compatibility.
+ return;
+ }
+ m.setAccessible(true);
+ m.invoke(flinkWriterOperator, args);
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to invoke method "
+ + methodName
+ + " on wrapped flink writer operator "
+ + flinkWriterOperator.getClass().getName(),
+ e);
+ }
+ }
+
+ private static Method findMethod(Class<?> clazz, String methodName, Class<?>[] parameterTypes) {
+ while (clazz != null) {
+ try {
+ return clazz.getDeclaredMethod(methodName, parameterTypes);
+ } catch (NoSuchMethodException e) {
+ clazz = clazz.getSuperclass();
+ }
+ }
+ return null;
+ }
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java
index 278518f97..a286ac2e7 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperatorFactory.java
@@ -23,17 +23,16 @@ import org.apache.flink.cdc.common.event.Event;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
import org.apache.flink.streaming.api.operators.AbstractStreamOperatorFactory;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperatorFactory;
import org.apache.flink.streaming.api.operators.StreamOperator;
import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
-import org.apache.flink.streaming.api.operators.YieldingOperatorFactory;
/** Operator factory for {@link DataSinkWriterOperator}. */
@Internal
public class DataSinkWriterOperatorFactory<CommT>
extends AbstractStreamOperatorFactory<CommittableMessage<CommT>>
- implements OneInputStreamOperatorFactory<Event, CommittableMessage<CommT>>,
- YieldingOperatorFactory<CommittableMessage<CommT>> {
+ implements OneInputStreamOperatorFactory<Event, CommittableMessage<CommT>> {
private final Sink<Event> sink;
private final boolean isBounded;
@@ -41,6 +40,7 @@ public class DataSinkWriterOperatorFactory<CommT>
public DataSinkWriterOperatorFactory(
Sink<Event> sink, boolean isBounded, OperatorID schemaOperatorID) {
+ setChainingStrategy(ChainingStrategy.ALWAYS);
this.sink = sink;
this.isBounded = isBounded;
this.schemaOperatorID = schemaOperatorID;
@@ -50,11 +50,10 @@ public class DataSinkWriterOperatorFactory<CommT>
@Override
public <T extends StreamOperator<CommittableMessage<CommT>>> T createStreamOperator(
StreamOperatorParameters<CommittableMessage<CommT>> parameters) {
-
if (isBounded) {
BatchDataSinkWriterOperator<CommT> writerOperator =
new BatchDataSinkWriterOperator<>(
- sink, processingTimeService, getMailboxExecutor());
+ sink, processingTimeService, parameters.getMailboxExecutor());
writerOperator.setup(
parameters.getContainingTask(),
parameters.getStreamConfig(),
@@ -63,7 +62,10 @@ public class DataSinkWriterOperatorFactory<CommT>
}
DataSinkWriterOperator<CommT> writerOperator =
new DataSinkWriterOperator<>(
- sink, processingTimeService, getMailboxExecutor(), schemaOperatorID);
+ sink,
+ processingTimeService,
+ parameters.getMailboxExecutor(),
+ schemaOperatorID);
writerOperator.setup(
parameters.getContainingTask(),
parameters.getStreamConfig(),
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java
index ff6cc512f..fc3509196 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java
@@ -35,6 +35,7 @@ import org.apache.flink.cdc.common.utils.Preconditions;
import org.apache.flink.cdc.common.utils.SchemaUtils;
import org.apache.flink.cdc.runtime.operators.transform.exceptions.TransformException;
import org.apache.flink.cdc.runtime.parser.TransformParser;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.state.StateInitializationContext;
import org.apache.flink.runtime.state.StateSnapshotContext;
import org.apache.flink.streaming.api.graph.StreamConfig;
@@ -85,7 +86,7 @@ public class PreTransformOperator extends AbstractStreamOperator<Event>
this.preTransformChangeInfoMap = new HashMap<>();
this.preTransformProcessorMap = new HashMap<>();
this.schemaMetadataTransformers = new ArrayList<>();
- this.chainingStrategy = ChainingStrategy.ALWAYS;
+ FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
this.transformRules = transformRules;
this.udfFunctions = udfFunctions;
@@ -161,6 +162,12 @@ public class PreTransformOperator extends AbstractStreamOperator<Event>
try {
processEvent(event);
} catch (Exception e) {
+ // If this exception originates from a chained downstream operator, rethrow as-is
+ // to avoid misleadingly wrapping it as a "pre-transform" error.
+ if (isChainedOperatorException(e)) {
+ throw e;
+ }
+
TableId tableId = null;
Schema schemaBefore = null;
Schema schemaAfter = null;
@@ -398,4 +405,21 @@ public class PreTransformOperator extends AbstractStreamOperator<Event>
this.transforms = null;
this.preTransformProcessorMap = null;
}
+
+ /**
+ * Checks if the given exception originates from a chained downstream operator. In Flink 2.x,
+ * all operators default to {@code ChainingStrategy.ALWAYS}, which means downstream operators
+ * like PostTransformOperator may be chained with PreTransformOperator. When a chained operator
+ * fails, the exception propagates back through the chain and should not be re-wrapped as a
+ * "pre-transform" error.
+ */
+ private static boolean isChainedOperatorException(Exception e) {
+ try {
+ return Class.forName(
+ "org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException")
+ .isInstance(e);
+ } catch (ClassNotFoundException ignored) {
+ return false;
+ }
+ }
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/BatchRegularPrePartitionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/BatchRegularPrePartitionOperator.java
index d6142868f..e273bd694 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/BatchRegularPrePartitionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/BatchRegularPrePartitionOperator.java
@@ -27,6 +27,7 @@ import org.apache.flink.cdc.common.function.HashFunctionProvider;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator;
import org.apache.flink.cdc.runtime.serializer.event.EventSerializer;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.state.StateSnapshotContext;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
@@ -58,7 +59,7 @@ public class BatchRegularPrePartitionOperator extends AbstractStreamOperator<Par
public BatchRegularPrePartitionOperator(
int downstreamParallelism, HashFunctionProvider<DataChangeEvent> hashFunctionProvider) {
- this.chainingStrategy = ChainingStrategy.ALWAYS;
+ FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
this.downstreamParallelism = downstreamParallelism;
this.hashFunctionProvider = hashFunctionProvider;
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/DistributedPrePartitionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/DistributedPrePartitionOperator.java
index bedbe8e8e..8b8fcf322 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/DistributedPrePartitionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/DistributedPrePartitionOperator.java
@@ -28,6 +28,7 @@ import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.utils.SchemaUtils;
import org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator;
import org.apache.flink.cdc.runtime.serializer.event.EventSerializer;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.state.StateSnapshotContext;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
@@ -55,7 +56,7 @@ public class DistributedPrePartitionOperator extends AbstractStreamOperator<Part
public DistributedPrePartitionOperator(
int downstreamParallelism, HashFunctionProvider<DataChangeEvent> hashFunctionProvider) {
- this.chainingStrategy = ChainingStrategy.ALWAYS;
+ FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
this.downstreamParallelism = downstreamParallelism;
this.hashFunctionProvider = hashFunctionProvider;
}
@@ -63,7 +64,7 @@ public class DistributedPrePartitionOperator extends AbstractStreamOperator<Part
@Override
public void open() throws Exception {
super.open();
- subTaskId = getRuntimeContext().getIndexOfThisSubtask();
+ subTaskId = getRuntimeContext().getTaskInfo().getIndexOfThisSubtask();
schemaMap = new HashMap<>();
hashFunctionMap = new HashMap<>();
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/RegularPrePartitionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/RegularPrePartitionOperator.java
index 38f177472..d65e3d755 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/RegularPrePartitionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/RegularPrePartitionOperator.java
@@ -29,6 +29,7 @@ import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.runtime.operators.schema.regular.SchemaOperator;
import org.apache.flink.cdc.runtime.operators.sink.SchemaEvolutionClient;
import org.apache.flink.cdc.runtime.serializer.event.EventSerializer;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.runtime.state.StateSnapshotContext;
@@ -67,7 +68,7 @@ public class RegularPrePartitionOperator extends AbstractStreamOperator<Partitio
OperatorID schemaOperatorId,
int downstreamParallelism,
HashFunctionProvider<DataChangeEvent> hashFunctionProvider) {
- this.chainingStrategy = ChainingStrategy.ALWAYS;
+ FlinkCompatibilityUtils.setChainingStrategyIfAvailable(this, ChainingStrategy.ALWAYS);
this.schemaOperatorId = schemaOperatorId;
this.downstreamParallelism = downstreamParallelism;
this.hashFunctionProvider = hashFunctionProvider;
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/EnumSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/EnumSerializer.java
index 5b93c320d..c78281089 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/EnumSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/EnumSerializer.java
@@ -238,7 +238,8 @@ public final class EnumSerializer<T extends Enum<T>> extends TypeSerializer<T> {
return new EnumSerializer<>(enumClass, previousEnums);
}
- @Override
+ // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
+ // compatibility
public TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
TypeSerializer<T> newSerializer) {
if (!(newSerializer instanceof EnumSerializer)) {
@@ -269,6 +270,32 @@ public final class EnumSerializer<T extends Enum<T>> extends TypeSerializer<T> {
return TypeSerializerSchemaCompatibility.compatibleWithReconfiguredSerializer(
reconfiguredSerializer);
}
+
+ // Flink 2.x new abstract method - no @Override so this also compiles against Flink 1.x
+ public TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
+ TypeSerializerSnapshot<T> oldSerializerSnapshot) {
+ if (!(oldSerializerSnapshot instanceof EnumSerializerSnapshot)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ @SuppressWarnings("unchecked")
+ EnumSerializerSnapshot<T> oldSnapshot =
+ (EnumSerializerSnapshot<T>) oldSerializerSnapshot;
+ if (!enumClass.equals(oldSnapshot.enumClass)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ if (Arrays.equals(oldSnapshot.previousEnums, previousEnums)) {
+ return TypeSerializerSchemaCompatibility.compatibleAsIs();
+ }
+ Set<T> reconfiguredEnumSet =
+ new LinkedHashSet<>(Arrays.asList(oldSnapshot.previousEnums));
+ reconfiguredEnumSet.addAll(Arrays.asList(previousEnums));
+ @SuppressWarnings("unchecked")
+ T[] reconfiguredEnums =
+ reconfiguredEnumSet.toArray(
+ (T[]) Array.newInstance(enumClass, reconfiguredEnumSet.size()));
+ return TypeSerializerSchemaCompatibility.compatibleWithReconfiguredSerializer(
+ new EnumSerializer<>(enumClass, reconfiguredEnums));
+ }
}
// --------------------------------------------------------------------------------------------
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NestedSerializersSnapshotDelegate.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NestedSerializersSnapshotDelegate.java
index 66c62630d..8eedaeef1 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NestedSerializersSnapshotDelegate.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NestedSerializersSnapshotDelegate.java
@@ -184,7 +184,14 @@ public class NestedSerializersSnapshotDelegate {
// Utilities
// ------------------------------------------------------------------------
- /** Utility method to conjure up a new scope for the generic parameters. */
+ /**
+ * Utility method to conjure up a new scope for the generic parameters.
+ *
+ * <p>In Flink 1.x, TypeSerializerSnapshot.resolveSchemaCompatibility(TypeSerializer) is the
+ * abstract method. In Flink 2.x, it was replaced by
+ * resolveSchemaCompatibility(TypeSerializerSnapshot). This method uses reflection to call the
+ * right variant at runtime.
+ */
@SuppressWarnings("unchecked")
private static <E> TypeSerializerSchemaCompatibility<E> resolveCompatibility(
TypeSerializer<?> serializer, TypeSerializerSnapshot<?> snapshot) {
@@ -192,7 +199,32 @@ public class NestedSerializersSnapshotDelegate {
TypeSerializer<E> typedSerializer = (TypeSerializer<E>) serializer;
TypeSerializerSnapshot<E> typedSnapshot = (TypeSerializerSnapshot<E>) snapshot;
- return typedSnapshot.resolveSchemaCompatibility(typedSerializer);
+ // Try Flink 1.x approach: oldSnapshot.resolveSchemaCompatibility(newSerializer)
+ try {
+ java.lang.reflect.Method m =
+ typedSnapshot
+ .getClass()
+ .getMethod("resolveSchemaCompatibility", TypeSerializer.class);
+ return (TypeSerializerSchemaCompatibility<E>) m.invoke(typedSnapshot, typedSerializer);
+ } catch (NoSuchMethodException ignored) {
+ // fall through to Flink 2.x approach
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to resolve schema compatibility (Flink 1.x path)", e);
+ }
+
+ // Flink 2.x approach: newSnapshot.resolveSchemaCompatibility(oldSnapshot)
+ try {
+ TypeSerializerSnapshot<E> newSnapshot = typedSerializer.snapshotConfiguration();
+ java.lang.reflect.Method m =
+ newSnapshot
+ .getClass()
+ .getMethod("resolveSchemaCompatibility", TypeSerializerSnapshot.class);
+ return (TypeSerializerSchemaCompatibility<E>) m.invoke(newSnapshot, typedSnapshot);
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to resolve schema compatibility (Flink 2.x path)", e);
+ }
}
private static TypeSerializer<?>[] snapshotsToRestoreSerializers(
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NullableSerializerWrapper.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NullableSerializerWrapper.java
index c230c8c9a..33eb92742 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NullableSerializerWrapper.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/NullableSerializerWrapper.java
@@ -172,7 +172,8 @@ public class NullableSerializerWrapper<T> extends TypeSerializer<T> {
return new NullableSerializerWrapper<>(innerSerializer);
}
- @Override
+ // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
+ // compatibility
public TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
TypeSerializer<T> newSerializer) {
if (!(newSerializer instanceof NullableSerializerWrapper)) {
@@ -187,5 +188,20 @@ public class NullableSerializerWrapper<T> extends TypeSerializer<T> {
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
+
+ // Flink 2.x new abstract method - no @Override so this also compiles against Flink 1.x
+ @SuppressWarnings("unchecked")
+ public TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
+ TypeSerializerSnapshot<T> oldSerializerSnapshot) {
+ if (!(oldSerializerSnapshot instanceof NullableSerializerWrapperSnapshot)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ NullableSerializerWrapperSnapshot<T> snapshot =
+ (NullableSerializerWrapperSnapshot<T>) oldSerializerSnapshot;
+ if (!innerSerializer.equals(snapshot.innerSerializer)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ return TypeSerializerSchemaCompatibility.compatibleAsIs();
+ }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ArrayDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ArrayDataSerializer.java
index 39939375d..ecbc03d7e 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ArrayDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ArrayDataSerializer.java
@@ -265,7 +265,8 @@ public class ArrayDataSerializer extends TypeSerializer<ArrayData> {
return new ArrayDataSerializer(previousType, previousEleSer);
}
- @Override
+ // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
+ // compatibility
public TypeSerializerSchemaCompatibility<ArrayData> resolveSchemaCompatibility(
TypeSerializer<ArrayData> newSerializer) {
if (!(newSerializer instanceof ArrayDataSerializer)) {
@@ -280,5 +281,20 @@ public class ArrayDataSerializer extends TypeSerializer<ArrayData> {
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
+
+ // Flink 2.x new abstract method - no @Override so this also compiles against Flink 1.x
+ public TypeSerializerSchemaCompatibility<ArrayData> resolveSchemaCompatibility(
+ TypeSerializerSnapshot<ArrayData> oldSerializerSnapshot) {
+ if (!(oldSerializerSnapshot instanceof ArrayDataSerializerSnapshot)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ ArrayDataSerializerSnapshot snapshot =
+ (ArrayDataSerializerSnapshot) oldSerializerSnapshot;
+ if (!previousType.equals(snapshot.previousType)
+ || !previousEleSer.equals(snapshot.previousEleSer)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ return TypeSerializerSchemaCompatibility.compatibleAsIs();
+ }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/DecimalDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/DecimalDataSerializer.java
index 4872330f3..97d2e3561 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/DecimalDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/DecimalDataSerializer.java
@@ -178,7 +178,8 @@ public final class DecimalDataSerializer extends TypeSerializer<DecimalData> {
return new DecimalDataSerializer(previousPrecision, previousScale);
}
- @Override
+ // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
+ // compatibility
public TypeSerializerSchemaCompatibility<DecimalData> resolveSchemaCompatibility(
TypeSerializer<DecimalData> newSerializer) {
if (!(newSerializer instanceof DecimalDataSerializer)) {
@@ -193,5 +194,21 @@ public final class DecimalDataSerializer extends TypeSerializer<DecimalData> {
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
+
+ // Flink 2.x changed the abstract method to
+ // resolveSchemaCompatibility(TypeSerializerSnapshot).
+ // No @Override so this also compiles against Flink 1.x where the method does not exist.
+ public TypeSerializerSchemaCompatibility<DecimalData> resolveSchemaCompatibility(
+ TypeSerializerSnapshot<DecimalData> oldSerializerSnapshot) {
+ if (!(oldSerializerSnapshot instanceof DecimalSerializerSnapshot)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ DecimalSerializerSnapshot snapshot = (DecimalSerializerSnapshot) oldSerializerSnapshot;
+ if (previousPrecision != snapshot.previousPrecision
+ || previousScale != snapshot.previousScale) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ return TypeSerializerSchemaCompatibility.compatibleAsIs();
+ }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/LocalZonedTimestampDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/LocalZonedTimestampDataSerializer.java
index c3622db23..987f20823 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/LocalZonedTimestampDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/LocalZonedTimestampDataSerializer.java
@@ -177,7 +177,8 @@ public class LocalZonedTimestampDataSerializer extends TypeSerializer<LocalZoned
return new LocalZonedTimestampDataSerializer(previousPrecision);
}
- @Override
+ // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
+ // compatibility
public TypeSerializerSchemaCompatibility<LocalZonedTimestampData>
resolveSchemaCompatibility(TypeSerializer<LocalZonedTimestampData> newSerializer) {
if (!(newSerializer instanceof LocalZonedTimestampDataSerializer)) {
@@ -192,5 +193,22 @@ public class LocalZonedTimestampDataSerializer extends TypeSerializer<LocalZoned
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
+
+ // Flink 2.x changed the abstract method to
+ // resolveSchemaCompatibility(TypeSerializerSnapshot).
+ // No @Override so this also compiles against Flink 1.x where the method does not exist.
+ public TypeSerializerSchemaCompatibility<LocalZonedTimestampData>
+ resolveSchemaCompatibility(
+ TypeSerializerSnapshot<LocalZonedTimestampData> oldSerializerSnapshot) {
+ if (!(oldSerializerSnapshot instanceof LocalZonedTimestampDataSerializerSnapshot)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ LocalZonedTimestampDataSerializerSnapshot snapshot =
+ (LocalZonedTimestampDataSerializerSnapshot) oldSerializerSnapshot;
+ if (previousPrecision != snapshot.previousPrecision) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ return TypeSerializerSchemaCompatibility.compatibleAsIs();
+ }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/MapDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/MapDataSerializer.java
index d8f92ee29..8b2e19cde 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/MapDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/MapDataSerializer.java
@@ -306,7 +306,8 @@ public class MapDataSerializer extends TypeSerializer<MapData> {
return new MapDataSerializer(keyType, valueType, keySerializer, valueSerializer);
}
- @Override
+ // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
+ // compatibility
public TypeSerializerSchemaCompatibility<MapData> resolveSchemaCompatibility(
TypeSerializer<MapData> newSerializer) {
if (!(newSerializer instanceof MapDataSerializer)) {
@@ -323,5 +324,21 @@ public class MapDataSerializer extends TypeSerializer<MapData> {
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
+
+ // Flink 2.x new abstract method - no @Override so this also compiles against Flink 1.x
+ public TypeSerializerSchemaCompatibility<MapData> resolveSchemaCompatibility(
+ TypeSerializerSnapshot<MapData> oldSerializerSnapshot) {
+ if (!(oldSerializerSnapshot instanceof MapDataSerializerSnapshot)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ MapDataSerializerSnapshot snapshot = (MapDataSerializerSnapshot) oldSerializerSnapshot;
+ if (!keyType.equals(snapshot.keyType)
+ || !valueType.equals(snapshot.valueType)
+ || !keySerializer.equals(snapshot.keySerializer)
+ || !valueSerializer.equals(snapshot.valueSerializer)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ return TypeSerializerSchemaCompatibility.compatibleAsIs();
+ }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/TimestampDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/TimestampDataSerializer.java
index babc6345b..10dfc0606 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/TimestampDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/TimestampDataSerializer.java
@@ -171,7 +171,8 @@ public class TimestampDataSerializer extends TypeSerializer<TimestampData> {
return new TimestampDataSerializer(previousPrecision);
}
- @Override
+ // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
+ // compatibility
public TypeSerializerSchemaCompatibility<TimestampData> resolveSchemaCompatibility(
TypeSerializer<TimestampData> newSerializer) {
if (!(newSerializer instanceof TimestampDataSerializer)) {
@@ -186,5 +187,21 @@ public class TimestampDataSerializer extends TypeSerializer<TimestampData> {
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
+
+ // Flink 2.x changed the abstract method to
+ // resolveSchemaCompatibility(TypeSerializerSnapshot).
+ // No @Override so this also compiles against Flink 1.x where the method does not exist.
+ public TypeSerializerSchemaCompatibility<TimestampData> resolveSchemaCompatibility(
+ TypeSerializerSnapshot<TimestampData> oldSerializerSnapshot) {
+ if (!(oldSerializerSnapshot instanceof TimestampDataSerializerSnapshot)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ TimestampDataSerializerSnapshot snapshot =
+ (TimestampDataSerializerSnapshot) oldSerializerSnapshot;
+ if (previousPrecision != snapshot.previousPrecision) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ return TypeSerializerSchemaCompatibility.compatibleAsIs();
+ }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ZonedTimestampDataSerializer.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ZonedTimestampDataSerializer.java
index ca9861761..8676d903f 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ZonedTimestampDataSerializer.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/serializer/data/ZonedTimestampDataSerializer.java
@@ -181,7 +181,8 @@ public class ZonedTimestampDataSerializer extends TypeSerializer<ZonedTimestampD
return new ZonedTimestampDataSerializer(previousPrecision);
}
- @Override
+ // Flink 1.x abstract method - removed in Flink 2.x, @Override omitted for cross-version
+ // compatibility
public TypeSerializerSchemaCompatibility<ZonedTimestampData> resolveSchemaCompatibility(
TypeSerializer<ZonedTimestampData> newSerializer) {
if (!(newSerializer instanceof ZonedTimestampDataSerializer)) {
@@ -196,5 +197,19 @@ public class ZonedTimestampDataSerializer extends TypeSerializer<ZonedTimestampD
return TypeSerializerSchemaCompatibility.compatibleAsIs();
}
}
+
+ // Flink 2.x new abstract method - no @Override so this also compiles against Flink 1.x
+ public TypeSerializerSchemaCompatibility<ZonedTimestampData> resolveSchemaCompatibility(
+ TypeSerializerSnapshot<ZonedTimestampData> oldSerializerSnapshot) {
+ if (!(oldSerializerSnapshot instanceof ZonedTimestampDataSerializerSnapshot)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ ZonedTimestampDataSerializerSnapshot snapshot =
+ (ZonedTimestampDataSerializerSnapshot) oldSerializerSnapshot;
+ if (previousPrecision != snapshot.previousPrecision) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+ return TypeSerializerSchemaCompatibility.compatibleAsIs();
+ }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/typeutils/EventTypeInfo.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/typeutils/EventTypeInfo.java
index 8087746cd..c126abca6 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/typeutils/EventTypeInfo.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/typeutils/EventTypeInfo.java
@@ -18,6 +18,7 @@
package org.apache.flink.cdc.runtime.typeutils;
import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.serialization.SerializerConfig;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.cdc.common.event.Event;
@@ -58,11 +59,18 @@ public class EventTypeInfo extends TypeInformation<Event> {
return false;
}
- @Override
+ // Flink 1.x abstract method - not abstract in Flink 2.x so @Override omitted for cross-version
+ // compatibility
public TypeSerializer<Event> createSerializer(ExecutionConfig config) {
return EventSerializer.INSTANCE;
}
+ /** For Flink 2.2+ (TypeInformation.createSerializer now takes SerializerConfig). */
+ @Override
+ public TypeSerializer<Event> createSerializer(SerializerConfig config) {
+ return EventSerializer.INSTANCE;
+ }
+
@Override
public String toString() {
return "Event";
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/typeutils/PartitioningEventTypeInfo.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/typeutils/PartitioningEventTypeInfo.java
index 4a3018fb9..1de112319 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/typeutils/PartitioningEventTypeInfo.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/typeutils/PartitioningEventTypeInfo.java
@@ -18,6 +18,7 @@
package org.apache.flink.cdc.runtime.typeutils;
import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.serialization.SerializerConfig;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.cdc.common.annotation.Internal;
@@ -58,11 +59,18 @@ public class PartitioningEventTypeInfo extends TypeInformation<PartitioningEvent
return false;
}
- @Override
+ // Flink 1.x abstract method - not abstract in Flink 2.x so @Override omitted for cross-version
+ // compatibility
public TypeSerializer<PartitioningEvent> createSerializer(ExecutionConfig config) {
return PartitioningEventSerializer.INSTANCE;
}
+ /** For Flink 2.2+ (TypeInformation.createSerializer now takes SerializerConfig). */
+ @Override
+ public TypeSerializer<PartitioningEvent> createSerializer(SerializerConfig config) {
+ return PartitioningEventSerializer.INSTANCE;
+ }
+
@Override
public String toString() {
return "PartitioningEvent";
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/utils/FlinkCompatibilityUtils.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/utils/FlinkCompatibilityUtils.java
new file mode 100644
index 000000000..974ad8641
--- /dev/null
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/utils/FlinkCompatibilityUtils.java
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.runtime.utils;
+
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.cdc.common.annotation.Internal;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+
+/** Utility class for handling Flink version compatibility issues. */
+@Internal
+public class FlinkCompatibilityUtils {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkCompatibilityUtils.class);
+
+ private FlinkCompatibilityUtils() {
+ // Private constructor to prevent instantiation
+ }
+
+ /**
+ * Sets the chaining strategy for an operator via reflection.
+ *
+ * <p>This method is needed because Flink 2.x removed the {@code chainingStrategy} field from
+ * {@link AbstractStreamOperator}. This method attempts to set the field via reflection for
+ * backward compatibility with Flink 1.x, and silently ignores the failure in Flink 2.x.
+ *
+ * @param operator the operator to set the chaining strategy for
+ * @param strategy the chaining strategy to set
+ */
+ public static void setChainingStrategyIfAvailable(
+ AbstractStreamOperator<?> operator, ChainingStrategy strategy) {
+ try {
+ Field field = AbstractStreamOperator.class.getDeclaredField("chainingStrategy");
+ field.setAccessible(true);
+ field.set(operator, strategy);
+ LOG.debug("Successfully set chainingStrategy to {} via reflection", strategy);
+ } catch (NoSuchFieldException e) {
+ LOG.debug(
+ "chainingStrategy field not available (likely Flink 2.x), skipping for operator {}",
+ operator.getClass().getSimpleName());
+ } catch (Exception e) {
+ LOG.warn(
+ "Failed to set chainingStrategy via reflection for operator {}",
+ operator.getClass().getSimpleName(),
+ e);
+ }
+ }
+
+ /**
+ * Resolves schema compatibility between a serializer snapshot and a new serializer, handling
+ * API differences between Flink 1.x and 2.x.
+ *
+ * <p>In Flink 1.x, the method is {@code snapshot.resolveSchemaCompatibility(serializer)}. In
+ * Flink 2.x, the method was changed to {@code
+ * newSnapshot.resolveSchemaCompatibility(oldSnapshot)}.
+ */
+ @SuppressWarnings("unchecked")
+ public static <T> TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
+ TypeSerializerSnapshot<T> snapshot, TypeSerializer<T> serializer) {
+ // Try Flink 1.x approach: snapshot.resolveSchemaCompatibility(serializer)
+ try {
+ Method m =
+ snapshot.getClass()
+ .getMethod("resolveSchemaCompatibility", TypeSerializer.class);
+ return (TypeSerializerSchemaCompatibility<T>) m.invoke(snapshot, serializer);
+ } catch (NoSuchMethodException ignored) {
+ // fall through to Flink 2.x approach
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof RuntimeException) {
+ throw (RuntimeException) cause;
+ }
+ throw new RuntimeException(
+ "Failed to resolve schema compatibility (Flink 1.x path)", cause);
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to resolve schema compatibility (Flink 1.x path)", e);
+ }
+
+ // Flink 2.x approach: newSnapshot.resolveSchemaCompatibility(oldSnapshot)
+ try {
+ TypeSerializerSnapshot<T> newSnapshot = serializer.snapshotConfiguration();
+ Method m =
+ newSnapshot
+ .getClass()
+ .getMethod("resolveSchemaCompatibility", TypeSerializerSnapshot.class);
+ return (TypeSerializerSchemaCompatibility<T>) m.invoke(newSnapshot, snapshot);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof RuntimeException) {
+ throw (RuntimeException) cause;
+ }
+ throw new RuntimeException(
+ "Failed to resolve schema compatibility (Flink 2.x path)", cause);
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Failed to resolve schema compatibility (Flink 2.x path)", e);
+ }
+ }
+
+ /**
+ * Calls {@link AbstractStreamOperator#setup} via reflection to handle access level changes
+ * between Flink 1.x (public) and Flink 2.x (protected).
+ */
+ public static void setupOperator(
+ AbstractStreamOperator<?> operator, Object containingTask, Object config, Object output)
+ throws Exception {
+ Method setupMethod = null;
+ for (Method m : AbstractStreamOperator.class.getDeclaredMethods()) {
+ if ("setup".equals(m.getName()) && m.getParameterCount() == 3) {
+ setupMethod = m;
+ break;
+ }
+ }
+ if (setupMethod == null) {
+ throw new NoSuchMethodException("Cannot find setup method in AbstractStreamOperator");
+ }
+ setupMethod.setAccessible(true);
+ try {
+ setupMethod.invoke(operator, containingTask, config, output);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException(cause);
+ }
+ }
+}
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkOperatorAdapter.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkOperatorAdapter.java
index 8fd60f632..0a9e60b7f 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkOperatorAdapter.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkOperatorAdapter.java
@@ -59,7 +59,15 @@ public class DataSinkOperatorAdapter extends AbstractStreamOperator<Event>
public DataSinkOperatorAdapter() {
this.schemaOperatorID = new OperatorID();
this.processedTableIds = new HashSet<>();
- this.chainingStrategy = ChainingStrategy.ALWAYS;
+ // Try to set chainingStrategy via reflection for backward compatibility with Flink 1.x
+ try {
+ java.lang.reflect.Field field =
+ AbstractStreamOperator.class.getDeclaredField("chainingStrategy");
+ field.setAccessible(true);
+ field.set(this, ChainingStrategy.ALWAYS);
+ } catch (Exception e) {
+ // Ignore if chainingStrategy doesn't exist in Flink 2.x
+ }
}
@Override
@@ -79,7 +87,8 @@ public class DataSinkOperatorAdapter extends AbstractStreamOperator<Event>
@Override
public void initializeState(StateInitializationContext context) throws Exception {
- schemaEvolutionClient.registerSubtask(getRuntimeContext().getIndexOfThisSubtask());
+ schemaEvolutionClient.registerSubtask(
+ getRuntimeContext().getTaskInfo().getIndexOfThisSubtask());
}
@Override
@@ -149,7 +158,8 @@ public class DataSinkOperatorAdapter extends AbstractStreamOperator<Event>
});
}
schemaEvolutionClient.notifyFlushSuccess(
- getRuntimeContext().getIndexOfThisSubtask(), event.getSourceSubTaskId());
+ getRuntimeContext().getTaskInfo().getIndexOfThisSubtask(),
+ event.getSourceSubTaskId());
}
private void emitLatestSchema(TableId tableId) throws Exception {
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/EnumSerializerTest.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/EnumSerializerTest.java
index e101492cd..c7fe72b1b 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/EnumSerializerTest.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/EnumSerializerTest.java
@@ -22,6 +22,7 @@ import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotSerializationUtil;
import org.apache.flink.cdc.common.utils.InstantiationUtil;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.core.memory.DataInputViewStreamWrapper;
import org.apache.flink.core.memory.DataOutputViewStreamWrapper;
import org.apache.flink.util.TestLogger;
@@ -85,7 +86,7 @@ class EnumSerializerTest extends TestLogger {
EnumSerializer.EnumSerializerSnapshot serializerSnapshot =
new EnumSerializer.EnumSerializerSnapshot(PublicEnum.class, mockPreviousOrder);
TypeSerializerSchemaCompatibility compatibility =
- serializerSnapshot.resolveSchemaCompatibility(serializer);
+ FlinkCompatibilityUtils.resolveSchemaCompatibility(serializerSnapshot, serializer);
Assertions.assertThat(compatibility.isCompatibleWithReconfiguredSerializer()).isTrue();
// after reconfiguration, the order should be first the original BAR, PAULA, NATHANIEL,
@@ -131,7 +132,7 @@ class EnumSerializerTest extends TestLogger {
}
TypeSerializerSchemaCompatibility<PublicEnum> compatResult =
- restoredConfig.resolveSchemaCompatibility(serializer);
+ FlinkCompatibilityUtils.resolveSchemaCompatibility(restoredConfig, serializer);
Assertions.assertThat(compatResult.isCompatibleAsIs()).isTrue();
Assertions.assertThat(serializer.getValueToOrdinal().get(PublicEnum.FOO).intValue())
@@ -217,7 +218,7 @@ class EnumSerializerTest extends TestLogger {
EnumSerializer.EnumSerializerSnapshot serializerSnapshot =
new EnumSerializer.EnumSerializerSnapshot(PublicEnum.class, mockPreviousOrder);
TypeSerializerSchemaCompatibility compatibility =
- serializerSnapshot.resolveSchemaCompatibility(serializer);
+ FlinkCompatibilityUtils.resolveSchemaCompatibility(serializerSnapshot, serializer);
Assertions.assertThat(compatibility.isCompatibleWithReconfiguredSerializer()).isTrue();
// verify that after the serializer was read, the reconfigured constant ordering is
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/SerializerTestBase.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/SerializerTestBase.java
index 79566763b..d50bb8a39 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/SerializerTestBase.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/serializer/SerializerTestBase.java
@@ -23,6 +23,7 @@ import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
import org.apache.flink.api.common.typeutils.TypeSerializerSnapshotSerializationUtil;
import org.apache.flink.api.java.typeutils.runtime.NullableSerializer;
import org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.core.memory.DataInputDeserializer;
import org.apache.flink.core.memory.DataInputView;
import org.apache.flink.core.memory.DataInputViewStreamWrapper;
@@ -152,7 +153,7 @@ public abstract class SerializerTestBase<T> {
}
TypeSerializerSchemaCompatibility<T> strategy =
- restoredConfig.resolveSchemaCompatibility(getSerializer());
+ FlinkCompatibilityUtils.resolveSchemaCompatibility(restoredConfig, getSerializer());
final TypeSerializer<T> restoreSerializer;
if (strategy.isCompatibleAsIs()) {
restoreSerializer = restoredConfig.restoreSerializer();
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/DistributedEventOperatorTestHarness.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/DistributedEventOperatorTestHarness.java
index 2b9030bc4..ab86809e1 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/DistributedEventOperatorTestHarness.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/DistributedEventOperatorTestHarness.java
@@ -30,20 +30,16 @@ import org.apache.flink.cdc.runtime.operators.schema.distributed.SchemaCoordinat
import org.apache.flink.cdc.runtime.operators.sink.SchemaEvolutionClient;
import org.apache.flink.cdc.runtime.testutils.schema.CollectingMetadataApplier;
import org.apache.flink.cdc.runtime.testutils.schema.TestingSchemaRegistryGateway;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.runtime.operators.testutils.DummyEnvironment;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
-import org.apache.flink.streaming.api.watermark.Watermark;
-import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
-import org.apache.flink.streaming.runtime.streamrecord.RecordAttributes;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
import org.apache.flink.streaming.runtime.tasks.StreamTask;
-import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
import org.apache.flink.streaming.util.MockStreamConfig;
-import org.apache.flink.util.OutputTag;
import org.apache.flink.util.SerializedValue;
import java.io.IOException;
@@ -154,68 +150,73 @@ public class DistributedEventOperatorTestHarness<
// -------------------------------------- Helper functions -------------------------------
+ @SuppressWarnings("unchecked")
private void initializeOperator() throws Exception {
- operator.setup(
+ Output<StreamRecord<E>> output =
+ (Output<StreamRecord<E>>)
+ java.lang.reflect.Proxy.newProxyInstance(
+ Output.class.getClassLoader(),
+ new Class<?>[] {Output.class},
+ new EventCollectingOutputHandler<>(
+ outputRecords, schemaRegistryGateway));
+ FlinkCompatibilityUtils.setupOperator(
+ operator,
new MockStreamTask(schemaRegistryGateway),
new MockStreamConfig(new Configuration(), numOutputs),
- new EventCollectingOutput<>(outputRecords, schemaRegistryGateway));
+ output);
schemaRegistryGateway.sendOperatorEventToCoordinator(
SINK_OPERATOR_ID, new SerializedValue<>(new SinkWriterRegisterEvent(0)));
}
// ---------------------------------------- Helper classes ---------------------------------
- private static class EventCollectingOutput<E extends Event> implements Output<StreamRecord<E>> {
+ private static class EventCollectingOutputHandler<E extends Event>
+ implements java.lang.reflect.InvocationHandler {
private final LinkedList<StreamRecord<E>> outputRecords;
-
private final TestingSchemaRegistryGateway schemaRegistryGateway;
- public EventCollectingOutput(
+ EventCollectingOutputHandler(
LinkedList<StreamRecord<E>> outputRecords,
TestingSchemaRegistryGateway schemaRegistryGateway) {
this.outputRecords = outputRecords;
this.schemaRegistryGateway = schemaRegistryGateway;
}
+ @SuppressWarnings("unchecked")
@Override
- public void collect(StreamRecord<E> record) {
- outputRecords.add(record);
- Event event = record.getValue();
- if (event instanceof FlushEvent) {
- try {
- schemaRegistryGateway.sendOperatorEventToCoordinator(
- SINK_OPERATOR_ID, new SerializedValue<>(new FlushSuccessEvent(0, 0)));
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
+ public Object invoke(Object proxy, java.lang.reflect.Method method, Object[] args)
+ throws Throwable {
+ switch (method.getName()) {
+ case "collect":
+ if (args != null && args.length == 1 && args[0] instanceof StreamRecord) {
+ StreamRecord<E> record = (StreamRecord<E>) args[0];
+ outputRecords.add(record);
+ Event event = record.getValue();
+ if (event instanceof FlushEvent) {
+ try {
+ schemaRegistryGateway.sendOperatorEventToCoordinator(
+ SINK_OPERATOR_ID,
+ new SerializedValue<>(new FlushSuccessEvent(0, 0)));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ return null;
+ }
+ throw new UnsupportedOperationException();
+ case "emitRecordAttributes":
+ case "close":
+ return null;
+ case "toString":
+ return "EventCollectingOutput";
+ case "hashCode":
+ return System.identityHashCode(proxy);
+ case "equals":
+ return proxy == args[0];
+ default:
+ throw new UnsupportedOperationException(method.getName());
}
}
-
- @Override
- public void emitWatermark(Watermark mark) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void emitWatermarkStatus(WatermarkStatus watermarkStatus) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public <X> void collect(OutputTag<X> outputTag, StreamRecord<X> record) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void emitLatencyMarker(LatencyMarker latencyMarker) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void emitRecordAttributes(RecordAttributes recordAttributes) {}
-
- @Override
- public void close() {}
}
private static class MockStreamTask extends StreamTask<Event, AbstractStreamOperator<Event>> {
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/RegularEventOperatorTestHarness.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/RegularEventOperatorTestHarness.java
index ea4de7a14..6b4facdbb 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/RegularEventOperatorTestHarness.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/RegularEventOperatorTestHarness.java
@@ -39,20 +39,16 @@ import org.apache.flink.cdc.runtime.operators.schema.regular.event.SchemaChangeR
import org.apache.flink.cdc.runtime.operators.sink.SchemaEvolutionClient;
import org.apache.flink.cdc.runtime.testutils.schema.CollectingMetadataApplier;
import org.apache.flink.cdc.runtime.testutils.schema.TestingSchemaRegistryGateway;
+import org.apache.flink.cdc.runtime.utils.FlinkCompatibilityUtils;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.runtime.operators.testutils.DummyEnvironment;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
-import org.apache.flink.streaming.api.watermark.Watermark;
-import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
-import org.apache.flink.streaming.runtime.streamrecord.RecordAttributes;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
import org.apache.flink.streaming.runtime.tasks.StreamTask;
-import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
import org.apache.flink.streaming.util.MockStreamConfig;
-import org.apache.flink.util.OutputTag;
import org.apache.flink.util.SerializedValue;
import org.slf4j.Logger;
@@ -285,68 +281,73 @@ public class RegularEventOperatorTestHarness<OP extends AbstractStreamOperator<E
// -------------------------------------- Helper functions -------------------------------
+ @SuppressWarnings("unchecked")
private void initializeOperator() throws Exception {
- operator.setup(
+ Output<StreamRecord<E>> output =
+ (Output<StreamRecord<E>>)
+ java.lang.reflect.Proxy.newProxyInstance(
+ Output.class.getClassLoader(),
+ new Class<?>[] {Output.class},
+ new EventCollectingOutputHandler<>(
+ outputRecords, schemaRegistryGateway));
+ FlinkCompatibilityUtils.setupOperator(
+ operator,
new MockStreamTask(schemaRegistryGateway),
new MockStreamConfig(new Configuration(), numOutputs),
- new EventCollectingOutput<>(outputRecords, schemaRegistryGateway));
+ output);
schemaRegistryGateway.sendOperatorEventToCoordinator(
SINK_OPERATOR_ID, new SerializedValue<>(new SinkWriterRegisterEvent(0)));
}
// ---------------------------------------- Helper classes ---------------------------------
- private static class EventCollectingOutput<E extends Event> implements Output<StreamRecord<E>> {
+ private static class EventCollectingOutputHandler<E extends Event>
+ implements java.lang.reflect.InvocationHandler {
private final LinkedList<StreamRecord<E>> outputRecords;
-
private final TestingSchemaRegistryGateway schemaRegistryGateway;
- public EventCollectingOutput(
+ EventCollectingOutputHandler(
LinkedList<StreamRecord<E>> outputRecords,
TestingSchemaRegistryGateway schemaRegistryGateway) {
this.outputRecords = outputRecords;
this.schemaRegistryGateway = schemaRegistryGateway;
}
+ @SuppressWarnings("unchecked")
@Override
- public void collect(StreamRecord<E> record) {
- outputRecords.add(record);
- Event event = record.getValue();
- if (event instanceof FlushEvent) {
- try {
- schemaRegistryGateway.sendOperatorEventToCoordinator(
- SINK_OPERATOR_ID, new SerializedValue<>(new FlushSuccessEvent(0, 0)));
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
+ public Object invoke(Object proxy, java.lang.reflect.Method method, Object[] args)
+ throws Throwable {
+ switch (method.getName()) {
+ case "collect":
+ if (args != null && args.length == 1 && args[0] instanceof StreamRecord) {
+ StreamRecord<E> record = (StreamRecord<E>) args[0];
+ outputRecords.add(record);
+ Event event = record.getValue();
+ if (event instanceof FlushEvent) {
+ try {
+ schemaRegistryGateway.sendOperatorEventToCoordinator(
+ SINK_OPERATOR_ID,
+ new SerializedValue<>(new FlushSuccessEvent(0, 0)));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ return null;
+ }
+ throw new UnsupportedOperationException();
+ case "emitRecordAttributes":
+ case "close":
+ return null;
+ case "toString":
+ return "EventCollectingOutput";
+ case "hashCode":
+ return System.identityHashCode(proxy);
+ case "equals":
+ return proxy == args[0];
+ default:
+ throw new UnsupportedOperationException(method.getName());
}
}
-
- @Override
- public void emitWatermark(Watermark mark) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void emitWatermarkStatus(WatermarkStatus watermarkStatus) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public <X> void collect(OutputTag<X> outputTag, StreamRecord<X> record) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void emitLatencyMarker(LatencyMarker latencyMarker) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void emitRecordAttributes(RecordAttributes recordAttributes) {}
-
- @Override
- public void close() {}
}
private static class MockStreamTask extends StreamTask<Event, AbstractStreamOperator<Event>> {
diff --git a/pom.xml b/pom.xml
index 3d87b6905..b3f8f0789 100644
--- a/pom.xml
+++ b/pom.xml
@@ -67,8 +67,13 @@ limitations under the License.
<flink.reuseForks>true</flink.reuseForks>
<!-- dependencies versions -->
- <flink.version>1.20.3</flink.version>
+ <flink.1.x.version>1.20.3</flink.1.x.version>
+ <flink.2.x.version>2.2.0</flink.2.x.version>
+ <flink.version>${flink.1.x.version}</flink.version>
<flink.major.version>1.20</flink.major.version>
+ <flink.1.x.shaded.guava.version>31.1-jre-17.0</flink.1.x.shaded.guava.version>
+ <flink.2.x.shaded.guava.version>33.4.0-jre-20.0</flink.2.x.shaded.guava.version>
+ <shaded.guava.version>${flink.1.x.shaded.guava.version}</shaded.guava.version>
<flink.shaded.version>17.0</flink.shaded.version>
<debezium.version>1.9.8.Final</debezium.version>
<tikv.version>3.2.0</tikv.version>
@@ -289,7 +294,7 @@ limitations under the License.
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-guava</artifactId>
- <version>31.1-jre-${flink.shaded.version}</version>
+ <version>${shaded.guava.version}</version>
</dependency>
<dependency>
@@ -842,6 +847,47 @@ limitations under the License.
</plugins>
</build>
</profile>
+ <profile>
+ <id>flink2</id>
+ <build>
+ <pluginManagement>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>copy-flink2-guava</id>
+ <phase>process-test-resources</phase>
+ <goals>
+ <goal>copy</goal>
+ </goals>
+ <configuration>
+ <artifactItems>
+ <artifactItem>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-shaded-guava</artifactId>
+ <version>${flink.2.x.shaded.guava.version}</version>
+ <outputDirectory>${project.build.directory}/flink2-extra-libs</outputDirectory>
+ </artifactItem>
+ </artifactItems>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+ <additionalClasspathElements>
+ <additionalClasspathElement>${project.build.directory}/flink2-extra-libs/flink-shaded-guava-${flink.2.x.shaded.guava.version}.jar</additionalClasspathElement>
+ </additionalClasspathElements>
+ </configuration>
+ </plugin>
+ </plugins>
+ </pluginManagement>
+ </build>
+ </profile>
<profile>
<id>java-17-target</id>
<activation>
(flink-cdc) 02/12: Address comments.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 5b763bd0b505a4d055bec18c88c050fb9807fcc2
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Mon Mar 9 12:06:59 2026 +0800
Address comments.
---
.../flink/cdc/composer/flink/translator/DataSinkTranslator.java | 4 +++-
.../flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java | 5 +++++
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
index c53d9f275..8a6bde5d1 100644
--- a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
+++ b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
@@ -119,7 +119,9 @@ public class DataSinkTranslator {
// Get sink provider
EventSinkProvider eventSinkProvider = dataSink.getEventSinkProvider();
if (eventSinkProvider == null) {
- return;
+ throw new IllegalStateException(
+ "DataSink#getEventSinkProvider() must not return null. "
+ + "Please ensure the sink is correctly configured.");
}
String sinkName = generateSinkName(sinkDef);
if (eventSinkProvider instanceof FlinkSinkProvider) {
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
index 6038045dd..79261bc43 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
@@ -44,6 +44,9 @@ import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
import org.apache.flink.streaming.runtime.tasks.StreamTask;
import org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
import java.lang.reflect.Constructor;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
@@ -65,6 +68,8 @@ import java.util.Set;
public class DataSinkWriterOperator<CommT> extends AbstractStreamOperator<CommittableMessage<CommT>>
implements OneInputStreamOperator<Event, CommittableMessage<CommT>>, BoundedOneInput {
+ private static final Logger LOG = LoggerFactory.getLogger(DataSinkWriterOperator.class);
+
private SchemaEvolutionClient schemaEvolutionClient;
private final OperatorID schemaOperatorID;
(flink-cdc) 05/12: Add flink-cdc-flink2.2-compat module.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 0ddbfb4fa5d8fb5e8159f2753136943691f29f4c
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Mon Mar 9 20:48:38 2026 +0800
Add flink-cdc-flink2.2-compat module.
---
.github/workflows/modules.py | 3 +-
flink-cdc-cli/pom.xml | 8 +
.../java/org/apache/flink/cdc/cli/CliFrontend.java | 8 +-
.../flink/cdc/cli/utils/ConfigurationUtils.java | 6 +-
.../org/apache/flink/cdc/cli/CliFrontendTest.java | 9 +-
flink-cdc-common/pom.xml | 32 +-
.../flink/cdc/common/factories/FactoryHelper.java | 2 +-
.../flink/cdc/common/sink/EventSinkProvider.java | 1 +
.../cdc/common/source/EventSourceProvider.java | 1 +
.../cdc/common/factories/FactoryHelperTests.java | 2 +-
flink-cdc-composer/pom.xml | 15 +-
.../flink/translator/DataSinkTranslator.java | 261 ++-------
.../flink/translator/DataSourceTranslator.java | 67 +--
.../testsource/source/DistributedDataSource.java | 5 +-
.../testsource/source/DistributedSource.java | 587 ---------------------
.../source/DistributedSourceFunction.java | 276 ++++++++++
.../sink/ElasticsearchDataSinkFactory.java | 2 +-
.../sink/ElasticsearchDataSinkFactoryTest.java | 2 +-
.../fluss/factory/FlussDataSinkFactoryTest.java | 2 +-
.../hudi/sink/HudiDataSinkFactoryTest.java | 2 +-
.../iceberg/sink/IcebergDataSinkFactoryTest.java | 2 +-
.../kafka/sink/KafkaDataSinkFactoryTest.java | 2 +-
.../mysql/factory/MySqlDataSourceFactory.java | 2 +-
.../mysql/source/MySqlDataSourceFactoryTest.java | 2 +-
.../oracle/factory/OracleDataSourceFactory.java | 2 +-
.../paimon/sink/PaimonDataSinkFactoryTest.java | 2 +-
.../factory/PostgresDataSourceFactory.java | 2 +-
.../factory/PostgresDataSourceFactoryTest.java | 1 -
.../sink/StarRocksDataSinkFactoryTest.java | 2 +-
.../pom.xml | 132 -----
.../cdc/connectors/values/sink/ValuesDataSink.java | 187 -------
.../org.apache.flink.cdc.common.factories.Factory | 16 -
.../flink-cdc-pipeline-connector-values/pom.xml | 35 ++
.../cdc/connectors/values/sink/ValuesDataSink.java | 10 +-
.../values/sink/ValuesDataSinkFunction.java | 101 ++++
.../values/source/ValuesDataSourceITCase.java | 4 +-
.../flink-cdc-pipeline-connectors/pom.xml | 1 -
flink-cdc-dist/pom.xml | 5 +
.../flink-cdc-pipeline-e2e-tests-2.x/pom.xml | 4 +-
.../tests/utils/PipelineTestEnvironment.java | 2 +-
.../cdc/pipeline/tests/MaxComputeE2eITCase.java | 2 +-
.../cdc/pipeline/tests/MySqlToHudiE2eITCase.java | 2 +-
.../tests/utils/PipelineTestEnvironment.java | 2 +-
.../tests/utils/FlinkContainerTestEnvironment.java | 2 +-
.../pom.xml | 93 ++--
.../org/apache/flink/api/connector/sink2/Sink.java | 72 +++
.../connector/sink2/TwoPhaseCommittingSink.java | 15 +-
.../apache/flink/core/execution/RestoreMode.java | 54 ++
.../runtime/jobgraph/SavepointConfigOptions.java | 56 ++
.../connector/sink2/WithPostCommitTopology.java | 14 +-
.../api/connector/sink2/WithPreCommitTopology.java | 15 +-
.../api/connector/sink2/WithPreWriteTopology.java | 13 +-
.../streaming/api/functions/sink/SinkFunction.java | 15 +-
.../source/RichParallelSourceFunction.java | 40 ++
.../api/functions/source/SourceFunction.java | 15 +-
.../flink/table/api}/ValidationException.java | 11 +-
flink-cdc-runtime/pom.xml | 32 +-
pom.xml | 1 +
58 files changed, 840 insertions(+), 1419 deletions(-)
diff --git a/.github/workflows/modules.py b/.github/workflows/modules.py
index ffbcf0742..d06c36d02 100755
--- a/.github/workflows/modules.py
+++ b/.github/workflows/modules.py
@@ -22,8 +22,7 @@ MODULES_CORE_2_X = [
"flink-cdc-common",
"flink-cdc-composer",
"flink-cdc-runtime",
- "flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values",
- "flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x"
+ "flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values"
]
MODULES_PIPELINE_CONNECTORS = [
diff --git a/flink-cdc-cli/pom.xml b/flink-cdc-cli/pom.xml
index 7934cf576..6230f22a1 100644
--- a/flink-cdc-cli/pom.xml
+++ b/flink-cdc-cli/pom.xml
@@ -88,6 +88,14 @@ limitations under the License.
<properties>
<flink.version>${flink.2.x.version}</flink.version>
</properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
<build>
<plugins>
<plugin>
diff --git a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java
index fdabbee77..5f327eca2 100644
--- a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java
+++ b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontend.java
@@ -27,6 +27,7 @@ import org.apache.flink.cdc.common.utils.StringUtils;
import org.apache.flink.cdc.composer.PipelineExecution;
import org.apache.flink.configuration.DeploymentOptions;
import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.jobgraph.SavepointConfigOptions;
import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
import org.apache.flink.shaded.guava31.com.google.common.base.Joiner;
@@ -172,12 +173,7 @@ public class CliFrontend {
commandLine.getOptionValue(SAVEPOINT_CLAIM_MODE),
ConfigurationUtils.getClaimModeClass());
} else {
- try {
- restoreMode =
- ConfigurationUtils.getClaimModeClass().getField("DEFAULT").get(null);
- } catch (NoSuchFieldException | IllegalAccessException e) {
- throw new RuntimeException("Failed to get default restore mode.", e);
- }
+ restoreMode = SavepointConfigOptions.RESTORE_MODE.defaultValue();
}
// allowNonRestoredState is always false because all operators are predefined.
diff --git a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java
index 75b8c9019..7280c0679 100644
--- a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java
+++ b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/utils/ConfigurationUtils.java
@@ -72,11 +72,7 @@ public class ConfigurationUtils {
try {
return Class.forName("org.apache.flink.runtime.jobgraph.RestoreMode");
} catch (ClassNotFoundException e) {
- try {
- return Class.forName("org.apache.flink.core.execution.RecoveryClaimMode");
- } catch (ClassNotFoundException classNotFoundException) {
- throw new RuntimeException(classNotFoundException);
- }
+ throw new RuntimeException(e);
}
}
}
diff --git a/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java b/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java
index ec7084a91..b9f42cefc 100644
--- a/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java
+++ b/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java
@@ -17,10 +17,10 @@
package org.apache.flink.cdc.cli;
-import org.apache.flink.cdc.cli.utils.ConfigurationUtils;
import org.apache.flink.cdc.composer.PipelineComposer;
import org.apache.flink.cdc.composer.PipelineExecution;
import org.apache.flink.cdc.composer.definition.PipelineDef;
+import org.apache.flink.core.execution.RestoreMode;
import org.apache.flink.core.fs.Path;
import org.apache.flink.shaded.guava31.com.google.common.io.Resources;
@@ -107,8 +107,7 @@ class CliFrontendTest {
"-n");
assertThat(executor.getFlinkConfig().get(SAVEPOINT_PATH))
.isEqualTo(flinkHome() + "/savepoints/savepoint-1");
- assertThat(executor.getFlinkConfig().get(RESTORE_MODE))
- .isEqualTo(getRestoreModeEnum("NO_CLAIM"));
+ assertThat(executor.getFlinkConfig().get(RESTORE_MODE)).isEqualTo(RestoreMode.NO_CLAIM);
assertThat(executor.getFlinkConfig().get(SAVEPOINT_IGNORE_UNCLAIMED_STATE)).isTrue();
}
@@ -226,10 +225,6 @@ class CliFrontendTest {
return CliFrontend.createExecutor(parser.parse(cliOptions, args));
}
- private Object getRestoreModeEnum(String name) throws Exception {
- return Enum.valueOf((Class<Enum>) ConfigurationUtils.getClaimModeClass(), name);
- }
-
private String pipelineDef() throws Exception {
URL resource = Resources.getResource("definitions/pipeline-definition-full.yaml");
return Paths.get(resource.toURI()).toString();
diff --git a/flink-cdc-common/pom.xml b/flink-cdc-common/pom.xml
index 4e56ca8d4..8721e620e 100644
--- a/flink-cdc-common/pom.xml
+++ b/flink-cdc-common/pom.xml
@@ -50,30 +50,14 @@ limitations under the License.
<properties>
<flink.version>${flink.2.x.version}</flink.version>
</properties>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <configuration>
- <excludes>
- <!-- SinkFunction is moved to legacy package in Flink 2.x -->
- <exclude>org/apache/flink/cdc/common/sink/FlinkSinkFunctionProvider.java</exclude>
- <!-- SourceFunction is moved to legacy package in Flink 2.x -->
- <exclude>org/apache/flink/cdc/common/source/FlinkSourceFunctionProvider.java</exclude>
- </excludes>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-dependency-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- </plugin>
- </plugins>
- </build>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
</profile>
</profiles>
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/factories/FactoryHelper.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/factories/FactoryHelper.java
index f9b53c973..8e4ff0227 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/factories/FactoryHelper.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/factories/FactoryHelper.java
@@ -21,9 +21,9 @@ import org.apache.flink.cdc.common.annotation.PublicEvolving;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.configuration.FallbackKey;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.utils.Preconditions;
import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
import java.util.Arrays;
import java.util.HashMap;
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
index 4e1114971..ec1d34b67 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
@@ -19,6 +19,7 @@ package org.apache.flink.cdc.common.sink;
import org.apache.flink.api.connector.sink2.Sink;
import org.apache.flink.cdc.common.annotation.PublicEvolving;
+import org.apache.flink.streaming.api.functions.sink.SinkFunction;
/**
* A marker interface used to provide an event sink for writing change events to external systems.
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source/EventSourceProvider.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source/EventSourceProvider.java
index f94747f30..bf33f839e 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source/EventSourceProvider.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/source/EventSourceProvider.java
@@ -19,6 +19,7 @@ package org.apache.flink.cdc.common.source;
import org.apache.flink.api.connector.source.Source;
import org.apache.flink.cdc.common.annotation.PublicEvolving;
+import org.apache.flink.streaming.api.functions.source.SourceFunction;
/**
* A marker interface used to provide an event source for reading events from external systems. We
diff --git a/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/factories/FactoryHelperTests.java b/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/factories/FactoryHelperTests.java
index d04d958a9..a7be9c3f5 100644
--- a/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/factories/FactoryHelperTests.java
+++ b/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/factories/FactoryHelperTests.java
@@ -20,7 +20,7 @@ package org.apache.flink.cdc.common.factories;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.ConfigOptions;
import org.apache.flink.cdc.common.configuration.Configuration;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.Sets;
diff --git a/flink-cdc-composer/pom.xml b/flink-cdc-composer/pom.xml
index 37210f834..ef3040f49 100644
--- a/flink-cdc-composer/pom.xml
+++ b/flink-cdc-composer/pom.xml
@@ -27,10 +27,6 @@ limitations under the License.
<artifactId>flink-cdc-composer</artifactId>
- <properties>
- <values.connector.artifactId>flink-cdc-pipeline-connector-values</values.connector.artifactId>
- </properties>
-
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
@@ -44,7 +40,7 @@ limitations under the License.
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>${values.connector.artifactId}</artifactId>
+ <artifactId>flink-cdc-pipeline-connector-values</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
@@ -105,7 +101,6 @@ limitations under the License.
<id>flink2</id>
<properties>
<flink.version>${flink.2.x.version}</flink.version>
- <values.connector.artifactId>flink-cdc-pipeline-connector-values-2.x</values.connector.artifactId>
</properties>
<build>
<plugins>
@@ -129,6 +124,14 @@ limitations under the License.
</plugin>
</plugins>
</build>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
</profile>
</profiles>
diff --git a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
index c53d9f275..aa530300b 100644
--- a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
+++ b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
@@ -19,6 +19,7 @@ package org.apache.flink.cdc.composer.flink.translator;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.annotation.VisibleForTesting;
import org.apache.flink.cdc.common.configuration.Configuration;
@@ -27,59 +28,38 @@ import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.common.sink.EventSinkProvider;
+import org.apache.flink.cdc.common.sink.FlinkSinkFunctionProvider;
import org.apache.flink.cdc.common.sink.FlinkSinkProvider;
import org.apache.flink.cdc.composer.definition.SinkDef;
import org.apache.flink.cdc.composer.flink.FlinkEnvironmentUtils;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
+import org.apache.flink.cdc.runtime.operators.sink.BatchDataSinkFunctionOperator;
+import org.apache.flink.cdc.runtime.operators.sink.DataSinkFunctionOperator;
import org.apache.flink.cdc.runtime.operators.sink.DataSinkWriterOperatorFactory;
import org.apache.flink.core.io.SimpleVersionedSerializer;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
import org.apache.flink.streaming.api.connector.sink2.CommittableMessageTypeInfo;
+import org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology;
+import org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.SinkFunction;
import org.apache.flink.streaming.api.operators.OneInputStreamOperatorFactory;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.api.transformations.LegacySinkTransformation;
+import org.apache.flink.streaming.api.transformations.PhysicalTransformation;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
-import java.lang.reflect.Method;
/** Translator used to build {@link DataSink} for given {@link DataStream}. */
@Internal
public class DataSinkTranslator {
- private static final Logger LOG = LoggerFactory.getLogger(DataSinkTranslator.class);
-
private static final String SINK_WRITER_PREFIX = "Sink Writer: ";
private static final String SINK_COMMITTER_PREFIX = "Sink Committer: ";
- private static final String FLINK_SINK_FUNCTION_PROVIDER_CLASS =
- "org.apache.flink.cdc.common.sink.FlinkSinkFunctionProvider";
- private static final String SINK_FUNCTION_CLASS =
- "org.apache.flink.streaming.api.functions.sink.SinkFunction";
- private static final String STREAM_SINK_CLASS =
- "org.apache.flink.streaming.api.operators.StreamSink";
- private static final String DATA_SINK_FUNCTION_OPERATOR_CLASS =
- "org.apache.flink.cdc.runtime.operators.sink.DataSinkFunctionOperator";
- private static final String BATCH_DATA_SINK_FUNCTION_OPERATOR_CLASS =
- "org.apache.flink.cdc.runtime.operators.sink.BatchDataSinkFunctionOperator";
- private static final String LEGACY_SINK_TRANSFORMATION_CLASS =
- "org.apache.flink.streaming.api.transformations.LegacySinkTransformation";
- // Sink V2 commit topology classes (not available in Flink 2.x)
- private static final String TWO_PHASE_COMMITTING_SINK_CLASS =
- "org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink";
- private static final String SUPPORTS_COMMITTER_CLASS =
- "org.apache.flink.api.connector.sink2.SupportsCommitter";
- private static final String WITH_PRE_WRITE_TOPOLOGY_CLASS =
- "org.apache.flink.streaming.api.connector.sink2.WithPreWriteTopology";
- private static final String WITH_PRE_COMMIT_TOPOLOGY_CLASS =
- "org.apache.flink.streaming.api.connector.sink2.WithPreCommitTopology";
- private static final String WITH_POST_COMMIT_TOPOLOGY_CLASS =
- "org.apache.flink.streaming.api.connector.sink2.WithPostCommitTopology";
-
public DataSink createDataSink(
SinkDef sinkDef, Configuration pipelineConfig, StreamExecutionEnvironment env) {
// Search the data sink factory
@@ -118,20 +98,20 @@ public class DataSinkTranslator {
OperatorUidGenerator operatorUidGenerator) {
// Get sink provider
EventSinkProvider eventSinkProvider = dataSink.getEventSinkProvider();
- if (eventSinkProvider == null) {
- return;
- }
String sinkName = generateSinkName(sinkDef);
if (eventSinkProvider instanceof FlinkSinkProvider) {
- // Sink V2 (works with both Flink 1.x and 2.x)
+ // Sink V2
FlinkSinkProvider sinkProvider = (FlinkSinkProvider) eventSinkProvider;
Sink<Event> sink = sinkProvider.getSink();
sinkTo(input, sink, sinkName, isBatchMode, schemaOperatorID, operatorUidGenerator);
- } else {
- // Try legacy SinkFunction path via reflection (Flink 1.x only)
- sinkToLegacy(
+ } else if (eventSinkProvider instanceof FlinkSinkFunctionProvider) {
+ // SinkFunction
+ FlinkSinkFunctionProvider sinkFunctionProvider =
+ (FlinkSinkFunctionProvider) eventSinkProvider;
+ SinkFunction<Event> sinkFunction = sinkFunctionProvider.getSinkFunction();
+ sinkTo(
input,
- eventSinkProvider,
+ sinkFunction,
sinkName,
isBatchMode,
schemaOperatorID,
@@ -148,10 +128,12 @@ public class DataSinkTranslator {
OperatorID schemaOperatorID,
OperatorUidGenerator operatorUidGenerator) {
DataStream<Event> stream = input;
- // Pre-write topology (use reflection since WithPreWriteTopology is removed in Flink 2.x)
- stream = tryAddPreWriteTopology(sink, stream);
+ // Pre-write topology
+ if (sink instanceof WithPreWriteTopology) {
+ stream = ((WithPreWriteTopology<Event>) sink).addPreWriteTopology(stream);
+ }
- if (isTwoPhaseCommittingSink(sink)) {
+ if (sink instanceof TwoPhaseCommittingSink) {
addCommittingTopology(
sink, stream, sinkName, isBatchMode, schemaOperatorID, operatorUidGenerator);
} else {
@@ -164,138 +146,29 @@ public class DataSinkTranslator {
}
}
- /**
- * Checks if the sink supports two-phase committing. In Flink 1.x, this is indicated by
- * implementing {@code TwoPhaseCommittingSink}. In Flink 2.x, the equivalent interface is {@code
- * SupportsCommitter}.
- */
- private static boolean isTwoPhaseCommittingSink(Sink<Event> sink) {
- // Try Flink 1.x: TwoPhaseCommittingSink
- try {
- Class<?> tpcClass = Class.forName(TWO_PHASE_COMMITTING_SINK_CLASS);
- if (tpcClass.isInstance(sink)) {
- return true;
- }
- } catch (ClassNotFoundException ignored) {
- // Not available in Flink 2.x
- }
- // Try Flink 2.x: SupportsCommitter
- try {
- Class<?> scClass = Class.forName(SUPPORTS_COMMITTER_CLASS);
- if (scClass.isInstance(sink)) {
- return true;
- }
- } catch (ClassNotFoundException ignored) {
- // Not available in Flink 1.x
- }
- return false;
- }
-
- /**
- * Tries to add a pre-write topology via reflection. In Flink 1.x, this checks for {@code
- * WithPreWriteTopology}. In Flink 2.x, this interface does not exist.
- */
- @SuppressWarnings("unchecked")
- private static DataStream<Event> tryAddPreWriteTopology(
- Sink<Event> sink, DataStream<Event> stream) {
- try {
- Class<?> clazz = Class.forName(WITH_PRE_WRITE_TOPOLOGY_CLASS);
- if (clazz.isInstance(sink)) {
- Method method = clazz.getMethod("addPreWriteTopology", DataStream.class);
- return (DataStream<Event>) method.invoke(sink, stream);
- }
- } catch (ClassNotFoundException ignored) {
- // WithPreWriteTopology not available (Flink 2.x)
- } catch (Exception e) {
- throw new RuntimeException("Failed to add pre-write topology via reflection", e);
- }
- return stream;
- }
-
- /**
- * Sink to a legacy SinkFunction via reflection. This method handles the case where
- * FlinkSinkFunctionProvider, SinkFunction, StreamSink, DataSinkFunctionOperator,
- * BatchDataSinkFunctionOperator, and LegacySinkTransformation are not available in Flink 2.x.
- */
- private void sinkToLegacy(
+ private void sinkTo(
DataStream<Event> input,
- EventSinkProvider eventSinkProvider,
+ SinkFunction<Event> sinkFunction,
String sinkName,
boolean isBatchMode,
OperatorID schemaOperatorID,
OperatorUidGenerator operatorUidGenerator) {
- try {
- // Check if FlinkSinkFunctionProvider is available
- Class<?> sinkFuncProviderClass = Class.forName(FLINK_SINK_FUNCTION_PROVIDER_CLASS);
- if (!sinkFuncProviderClass.isInstance(eventSinkProvider)) {
- throw new IllegalStateException(
- String.format(
- "Unsupported EventSinkProvider type \"%s\"",
- eventSinkProvider.getClass().getCanonicalName()));
- }
-
- // Get SinkFunction via reflection
- Object sinkFunction =
- sinkFuncProviderClass.getMethod("getSinkFunction").invoke(eventSinkProvider);
-
- // Create the operator (StreamSink subclass)
- Class<?> sinkFunctionClass = Class.forName(SINK_FUNCTION_CLASS);
- Object sinkOperator;
- if (isBatchMode) {
- Class<?> batchOpClass = Class.forName(BATCH_DATA_SINK_FUNCTION_OPERATOR_CLASS);
- sinkOperator =
- batchOpClass.getConstructor(sinkFunctionClass).newInstance(sinkFunction);
- } else {
- Class<?> opClass = Class.forName(DATA_SINK_FUNCTION_OPERATOR_CLASS);
- sinkOperator =
- opClass.getConstructor(sinkFunctionClass, OperatorID.class)
- .newInstance(sinkFunction, schemaOperatorID);
- }
-
- // Create LegacySinkTransformation via reflection
- Class<?> streamSinkClass = Class.forName(STREAM_SINK_CLASS);
- Class<?> legacySinkTransClass = Class.forName(LEGACY_SINK_TRANSFORMATION_CLASS);
- StreamExecutionEnvironment executionEnvironment = input.getExecutionEnvironment();
-
- // Find the constructor: LegacySinkTransformation(Transformation, String, StreamSink,
- // int, boolean)
- Constructor<?> constructor = null;
- for (Constructor<?> c : legacySinkTransClass.getDeclaredConstructors()) {
- Class<?>[] paramTypes = c.getParameterTypes();
- if (paramTypes.length == 5 && streamSinkClass.isAssignableFrom(paramTypes[2])) {
- constructor = c;
- break;
- }
- }
- if (constructor == null) {
- throw new NoSuchMethodException(
- "Cannot find suitable constructor for LegacySinkTransformation");
- }
-
- Object transformation =
- constructor.newInstance(
- input.getTransformation(),
- SINK_WRITER_PREFIX + sinkName,
- sinkOperator,
- executionEnvironment.getParallelism(),
- false);
-
- // Set UID
- Method setUidMethod = transformation.getClass().getMethod("setUid", String.class);
- setUidMethod.invoke(transformation, operatorUidGenerator.generateUid("sink-writer"));
-
- // Add operator to environment
- executionEnvironment.addOperator(
- (org.apache.flink.api.dag.Transformation<?>) transformation);
-
- } catch (ClassNotFoundException e) {
- throw new IllegalStateException(
- "Legacy SinkFunction API is not available. This is expected in Flink 2.x. "
- + "Please use FlinkSinkProvider (Sink V2 API) instead.",
- e);
- } catch (Exception e) {
- throw new RuntimeException("Failed to create legacy sink via reflection", e);
+ StreamSink<Event> sinkOperator;
+ if (isBatchMode) {
+ sinkOperator = new BatchDataSinkFunctionOperator(sinkFunction);
+ } else {
+ sinkOperator = new DataSinkFunctionOperator(sinkFunction, schemaOperatorID);
}
+ final StreamExecutionEnvironment executionEnvironment = input.getExecutionEnvironment();
+ PhysicalTransformation<Event> transformation =
+ new LegacySinkTransformation<>(
+ input.getTransformation(),
+ SINK_WRITER_PREFIX + sinkName,
+ sinkOperator,
+ executionEnvironment.getParallelism(),
+ false);
+ transformation.setUid(operatorUidGenerator.generateUid("sink-writer"));
+ executionEnvironment.addOperator(transformation);
}
private <CommT> void addCommittingTopology(
@@ -317,8 +190,10 @@ public class DataSinkTranslator {
.uid(operatorUidGenerator.generateUid("sink-writer"));
DataStream<CommittableMessage<CommT>> preCommitted = written;
- // Pre-commit topology (use reflection since WithPreCommitTopology is removed in Flink 2.x)
- preCommitted = tryAddPreCommitTopology(sink, written, preCommitted);
+ if (sink instanceof WithPreCommitTopology) {
+ preCommitted =
+ ((WithPreCommitTopology<Event, CommT>) sink).addPreCommitTopology(written);
+ }
// TODO: Hard coding checkpoint
boolean isCheckpointingEnabled = true;
@@ -331,50 +206,8 @@ public class DataSinkTranslator {
sink, isBatchMode, isCheckpointingEnabled))
.uid(operatorUidGenerator.generateUid("sink-committer"));
- // Post-commit topology (use reflection since WithPostCommitTopology is removed in Flink
- // 2.x)
- tryAddPostCommitTopology(sink, committed);
- }
-
- /**
- * Tries to add a pre-commit topology via reflection. In Flink 1.x, this checks for {@code
- * WithPreCommitTopology}. In Flink 2.x, this interface does not exist.
- */
- @SuppressWarnings("unchecked")
- private static <CommT> DataStream<CommittableMessage<CommT>> tryAddPreCommitTopology(
- Sink<Event> sink,
- DataStream<CommittableMessage<CommT>> written,
- DataStream<CommittableMessage<CommT>> fallback) {
- try {
- Class<?> clazz = Class.forName(WITH_PRE_COMMIT_TOPOLOGY_CLASS);
- if (clazz.isInstance(sink)) {
- Method method = clazz.getMethod("addPreCommitTopology", DataStream.class);
- return (DataStream<CommittableMessage<CommT>>) method.invoke(sink, written);
- }
- } catch (ClassNotFoundException ignored) {
- // WithPreCommitTopology not available (Flink 2.x)
- } catch (Exception e) {
- throw new RuntimeException("Failed to add pre-commit topology via reflection", e);
- }
- return fallback;
- }
-
- /**
- * Tries to add a post-commit topology via reflection. In Flink 1.x, this checks for {@code
- * WithPostCommitTopology}. In Flink 2.x, this interface does not exist.
- */
- private static <CommT> void tryAddPostCommitTopology(
- Sink<Event> sink, DataStream<CommittableMessage<CommT>> committed) {
- try {
- Class<?> clazz = Class.forName(WITH_POST_COMMIT_TOPOLOGY_CLASS);
- if (clazz.isInstance(sink)) {
- Method method = clazz.getMethod("addPostCommitTopology", DataStream.class);
- method.invoke(sink, committed);
- }
- } catch (ClassNotFoundException ignored) {
- // WithPostCommitTopology not available (Flink 2.x)
- } catch (Exception e) {
- throw new RuntimeException("Failed to add post-commit topology via reflection", e);
+ if (sink instanceof WithPostCommitTopology) {
+ ((WithPostCommitTopology<Event, CommT>) sink).addPostCommitTopology(committed);
}
}
diff --git a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java
index 280f7f5ad..e4f684f8a 100644
--- a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java
+++ b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java
@@ -18,7 +18,6 @@
package org.apache.flink.cdc.composer.flink.translator;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
-import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.Event;
@@ -26,6 +25,7 @@ import org.apache.flink.cdc.common.factories.DataSourceFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.source.DataSource;
import org.apache.flink.cdc.common.source.EventSourceProvider;
+import org.apache.flink.cdc.common.source.FlinkSourceFunctionProvider;
import org.apache.flink.cdc.common.source.FlinkSourceProvider;
import org.apache.flink.cdc.composer.definition.SourceDef;
import org.apache.flink.cdc.composer.flink.FlinkEnvironmentUtils;
@@ -35,22 +35,10 @@ import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.lang.reflect.Method;
-
/** Translator used to build {@link DataSource} which will generate a {@link DataStream}. */
@Internal
public class DataSourceTranslator {
- private static final Logger LOG = LoggerFactory.getLogger(DataSourceTranslator.class);
-
- private static final String FLINK_SOURCE_FUNCTION_PROVIDER_CLASS =
- "org.apache.flink.cdc.common.source.FlinkSourceFunctionProvider";
- private static final String SOURCE_FUNCTION_CLASS =
- "org.apache.flink.streaming.api.functions.source.SourceFunction";
-
public DataStream<Event> translate(
SourceDef sourceDef,
DataSource dataSource,
@@ -69,7 +57,7 @@ public class DataSourceTranslator {
StreamExecutionEnvironment env,
EventSourceProvider eventSourceProvider,
SourceDef sourceDef) {
- // Source (New Source API, works with both Flink 1.x and 2.x)
+ // Source
if (eventSourceProvider instanceof FlinkSourceProvider) {
FlinkSourceProvider sourceProvider = (FlinkSourceProvider) eventSourceProvider;
return env.fromSource(
@@ -79,13 +67,14 @@ public class DataSourceTranslator {
new EventTypeInfo());
}
- // SourceFunction (Legacy API, Flink 1.x only)
- // Use reflection since FlinkSourceFunctionProvider and SourceFunction are not available
- // in Flink 2.x builds.
- DataStreamSource<Event> legacyResult =
- tryCreateLegacyDataStreamSource(env, eventSourceProvider, sourceDef);
- if (legacyResult != null) {
- return legacyResult;
+ // SourceFunction
+ if (eventSourceProvider instanceof FlinkSourceFunctionProvider) {
+ FlinkSourceFunctionProvider sourceFunctionProvider =
+ (FlinkSourceFunctionProvider) eventSourceProvider;
+ DataStreamSource<Event> stream =
+ env.addSource(sourceFunctionProvider.getSourceFunction(), new EventTypeInfo());
+ sourceDef.getName().ifPresent(stream::name);
+ return stream;
}
// Unknown provider type
@@ -95,42 +84,6 @@ public class DataSourceTranslator {
eventSourceProvider.getClass().getCanonicalName()));
}
- @SuppressWarnings("unchecked")
- private DataStreamSource<Event> tryCreateLegacyDataStreamSource(
- StreamExecutionEnvironment env,
- EventSourceProvider eventSourceProvider,
- SourceDef sourceDef) {
- try {
- Class<?> sourceFuncProviderClass = Class.forName(FLINK_SOURCE_FUNCTION_PROVIDER_CLASS);
- if (!sourceFuncProviderClass.isInstance(eventSourceProvider)) {
- return null;
- }
- // Get the SourceFunction via reflection
- Object sourceFunction =
- sourceFuncProviderClass
- .getMethod("getSourceFunction")
- .invoke(eventSourceProvider);
- // Call env.addSource(sourceFunction, typeInfo) via reflection
- Class<?> sourceFunctionClass = Class.forName(SOURCE_FUNCTION_CLASS);
- Method addSourceMethod =
- StreamExecutionEnvironment.class.getMethod(
- "addSource", sourceFunctionClass, TypeInformation.class);
- DataStreamSource<Event> stream =
- (DataStreamSource<Event>)
- addSourceMethod.invoke(env, sourceFunction, new EventTypeInfo());
- sourceDef.getName().ifPresent(stream::name);
- return stream;
- } catch (ClassNotFoundException e) {
- LOG.debug(
- "FlinkSourceFunctionProvider or SourceFunction not available (likely Flink 2.x), "
- + "skipping legacy source function path.");
- return null;
- } catch (Exception e) {
- throw new RuntimeException(
- "Failed to create legacy data stream source via reflection", e);
- }
- }
-
public DataSource createDataSource(
SourceDef sourceDef, Configuration pipelineConfig, StreamExecutionEnvironment env) {
// Search the data source factory
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedDataSource.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedDataSource.java
index 910239423..dd3952039 100644
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedDataSource.java
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedDataSource.java
@@ -20,7 +20,7 @@ package org.apache.flink.cdc.composer.testsource.source;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.source.DataSource;
import org.apache.flink.cdc.common.source.EventSourceProvider;
-import org.apache.flink.cdc.common.source.FlinkSourceProvider;
+import org.apache.flink.cdc.common.source.FlinkSourceFunctionProvider;
import org.apache.flink.cdc.common.source.MetadataAccessor;
/** A {@link DataSource} that emits single-table-in-multi-partition data records. */
@@ -37,7 +37,8 @@ public class DistributedDataSource implements DataSource {
@Override
public EventSourceProvider getEventSourceProvider() {
- return FlinkSourceProvider.of(new DistributedSource(tableCount, distributedTables));
+ return FlinkSourceFunctionProvider.of(
+ new DistributedSourceFunction(tableCount, distributedTables));
}
@Override
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSource.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSource.java
deleted file mode 100644
index 9e012b110..000000000
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSource.java
+++ /dev/null
@@ -1,587 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.composer.testsource.source;
-
-import org.apache.flink.api.connector.source.Boundedness;
-import org.apache.flink.api.connector.source.ReaderOutput;
-import org.apache.flink.api.connector.source.Source;
-import org.apache.flink.api.connector.source.SourceReader;
-import org.apache.flink.api.connector.source.SourceReaderContext;
-import org.apache.flink.api.connector.source.SourceSplit;
-import org.apache.flink.api.connector.source.SplitEnumerator;
-import org.apache.flink.api.connector.source.SplitEnumeratorContext;
-import org.apache.flink.cdc.common.data.DecimalData;
-import org.apache.flink.cdc.common.data.LocalZonedTimestampData;
-import org.apache.flink.cdc.common.data.TimeData;
-import org.apache.flink.cdc.common.data.TimestampData;
-import org.apache.flink.cdc.common.data.ZonedTimestampData;
-import org.apache.flink.cdc.common.data.binary.BinaryRecordData;
-import org.apache.flink.cdc.common.data.binary.BinaryStringData;
-import org.apache.flink.cdc.common.event.AddColumnEvent;
-import org.apache.flink.cdc.common.event.CreateTableEvent;
-import org.apache.flink.cdc.common.event.DataChangeEvent;
-import org.apache.flink.cdc.common.event.Event;
-import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.schema.Column;
-import org.apache.flink.cdc.common.schema.Schema;
-import org.apache.flink.cdc.common.types.DataType;
-import org.apache.flink.cdc.common.types.DataTypes;
-import org.apache.flink.cdc.common.utils.SchemaUtils;
-import org.apache.flink.cdc.runtime.typeutils.BinaryRecordDataGenerator;
-import org.apache.flink.core.io.InputStatus;
-import org.apache.flink.core.io.SimpleVersionedSerializer;
-import org.apache.flink.core.memory.DataInputViewStreamWrapper;
-import org.apache.flink.core.memory.DataOutputViewStreamWrapper;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.ByteArrayInputStream;
-import java.io.ByteArrayOutputStream;
-import java.io.IOException;
-import java.math.BigDecimal;
-import java.sql.Timestamp;
-import java.time.Instant;
-import java.time.ZoneId;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.concurrent.CompletableFuture;
-import java.util.function.Consumer;
-import java.util.stream.Collectors;
-import java.util.stream.IntStream;
-
-/** Distributed source using FLIP-27 Source API for testing purposes only. */
-public class DistributedSource
- implements Source<
- Event,
- DistributedSource.DistributedSourceSplit,
- Collection<DistributedSource.DistributedSourceSplit>> {
-
- private static final long serialVersionUID = 1L;
-
- private final int numOfTables;
- private final boolean distributedTables;
-
- public DistributedSource(int numOfTables, boolean distributedTables) {
- this.numOfTables = numOfTables;
- this.distributedTables = distributedTables;
- }
-
- @Override
- public Boundedness getBoundedness() {
- return Boundedness.BOUNDED;
- }
-
- @Override
- public SplitEnumerator<DistributedSourceSplit, Collection<DistributedSourceSplit>>
- createEnumerator(SplitEnumeratorContext<DistributedSourceSplit> enumContext) {
- return new DistributedSourceEnumerator(enumContext, new ArrayList<>());
- }
-
- @Override
- public SplitEnumerator<DistributedSourceSplit, Collection<DistributedSourceSplit>>
- restoreEnumerator(
- SplitEnumeratorContext<DistributedSourceSplit> enumContext,
- Collection<DistributedSourceSplit> checkpoint) {
- return new DistributedSourceEnumerator(enumContext, new ArrayList<>(checkpoint));
- }
-
- @Override
- public SimpleVersionedSerializer<DistributedSourceSplit> getSplitSerializer() {
- return new DistributedSourceSplitSerializer();
- }
-
- @Override
- public SimpleVersionedSerializer<Collection<DistributedSourceSplit>>
- getEnumeratorCheckpointSerializer() {
- return new DistributedSourceEnumeratorSerializer();
- }
-
- @Override
- public SourceReader<Event, DistributedSourceSplit> createReader(
- SourceReaderContext readerContext) {
- return new DistributedSourceReader(readerContext, numOfTables, distributedTables);
- }
-
- // ------------------------------ Split ------------------------------
-
- /** Split for the distributed source, one per subtask. */
- static class DistributedSourceSplit implements SourceSplit {
-
- final int splitIndex;
- final int parallelism;
- final int eventOffset;
-
- DistributedSourceSplit(int splitIndex, int parallelism, int eventOffset) {
- this.splitIndex = splitIndex;
- this.parallelism = parallelism;
- this.eventOffset = eventOffset;
- }
-
- @Override
- public String splitId() {
- return "distributed-split-" + splitIndex;
- }
- }
-
- // ------------------------------ Enumerator ------------------------------
-
- /** Enumerator that creates one split per subtask and assigns them by subtask ID. */
- private static class DistributedSourceEnumerator
- implements SplitEnumerator<DistributedSourceSplit, Collection<DistributedSourceSplit>> {
-
- private final SplitEnumeratorContext<DistributedSourceSplit> context;
- private final List<DistributedSourceSplit> pendingSplits;
-
- DistributedSourceEnumerator(
- SplitEnumeratorContext<DistributedSourceSplit> context,
- List<DistributedSourceSplit> pendingSplits) {
- this.context = context;
- this.pendingSplits = pendingSplits;
- }
-
- @Override
- public void start() {
- if (pendingSplits.isEmpty()) {
- int parallelism = context.currentParallelism();
- for (int i = 0; i < parallelism; i++) {
- pendingSplits.add(new DistributedSourceSplit(i, parallelism, 0));
- }
- }
- }
-
- @Override
- public void handleSplitRequest(int subtaskId, String requesterHostname) {
- // No-op: splits are assigned in addReader via push model
- }
-
- @Override
- public void addSplitsBack(List<DistributedSourceSplit> splits, int subtaskId) {
- pendingSplits.addAll(splits);
- }
-
- @Override
- public void addReader(int subtaskId) {
- DistributedSourceSplit assigned = null;
- for (int i = 0; i < pendingSplits.size(); i++) {
- if (pendingSplits.get(i).splitIndex == subtaskId) {
- assigned = pendingSplits.remove(i);
- break;
- }
- }
- if (assigned != null) {
- context.assignSplit(assigned, subtaskId);
- }
- context.signalNoMoreSplits(subtaskId);
- }
-
- @Override
- public Collection<DistributedSourceSplit> snapshotState(long checkpointId) {
- return new ArrayList<>(pendingSplits);
- }
-
- @Override
- public void close() {
- // No-op
- }
- }
-
- // ------------------------------ Reader ------------------------------
-
- /** Reader that generates events using the same logic as the old DistributedSourceFunction. */
- private static class DistributedSourceReader
- implements SourceReader<Event, DistributedSourceSplit> {
-
- private static final Logger LOG = LoggerFactory.getLogger(DistributedSourceReader.class);
-
- private final SourceReaderContext context;
- private final int numOfTables;
- private final boolean distributedTables;
- private final int subTaskId;
-
- private int parallelism;
- private List<Event> events;
- private int currentIndex;
- private boolean splitReceived;
- private boolean finished;
- private CompletableFuture<Void> availability;
-
- DistributedSourceReader(
- SourceReaderContext context, int numOfTables, boolean distributedTables) {
- this.context = context;
- this.numOfTables = numOfTables;
- this.distributedTables = distributedTables;
- this.subTaskId = context.getIndexOfSubtask();
- this.splitReceived = false;
- this.finished = false;
- this.availability = new CompletableFuture<>();
- }
-
- @Override
- public void start() {
- // Wait for splits to be assigned
- }
-
- @Override
- public InputStatus pollNext(ReaderOutput<Event> output) throws InterruptedException {
- if (!splitReceived) {
- return InputStatus.NOTHING_AVAILABLE;
- }
- if (currentIndex < events.size()) {
- Event event = events.get(currentIndex++);
- LOG.info("{}> Emitting event {}", subTaskId, event);
- output.collect(event);
- return currentIndex < events.size()
- ? InputStatus.MORE_AVAILABLE
- : InputStatus.END_OF_INPUT;
- }
- if (parallelism > 1 && !finished) {
- // To allow test running correctly, we need to wait for downstream schema
- // evolutions to finish before closing any subTask.
- Thread.sleep(10000);
- finished = true;
- }
- return InputStatus.END_OF_INPUT;
- }
-
- @Override
- public List<DistributedSourceSplit> snapshotState(long checkpointId) {
- if (!splitReceived || finished) {
- return Collections.emptyList();
- }
- return Collections.singletonList(
- new DistributedSourceSplit(subTaskId, parallelism, currentIndex));
- }
-
- @Override
- public CompletableFuture<Void> isAvailable() {
- return availability;
- }
-
- @Override
- public void addSplits(List<DistributedSourceSplit> splits) {
- for (DistributedSourceSplit split : splits) {
- this.parallelism = split.parallelism;
- this.events = generateAllEvents(split);
- this.currentIndex = split.eventOffset;
- }
- this.splitReceived = true;
- this.availability.complete(null);
- }
-
- @Override
- public void notifyNoMoreSplits() {
- // If no splits were assigned (shouldn't happen), mark as finished
- if (!splitReceived) {
- finished = true;
- availability.complete(null);
- }
- }
-
- @Override
- public void close() {
- // No-op
- }
-
- // ----- Event generation logic (migrated from DistributedSourceFunction) -----
-
- private List<Event> generateAllEvents(DistributedSourceSplit split) {
- int parallelism = split.parallelism;
- int iotaCounter = 0;
-
- // Initialize dummy data types (same as original open())
- Map<DataType, Object> dummyDataTypes = new LinkedHashMap<>();
- dummyDataTypes.put(DataTypes.BOOLEAN(), true);
- dummyDataTypes.put(DataTypes.TINYINT(), (byte) 17);
- dummyDataTypes.put(DataTypes.SMALLINT(), (short) 34);
- dummyDataTypes.put(DataTypes.INT(), (int) 68);
- dummyDataTypes.put(DataTypes.BIGINT(), (long) 136);
- dummyDataTypes.put(DataTypes.FLOAT(), (float) 272.0);
- dummyDataTypes.put(DataTypes.DOUBLE(), (double) 544.0);
- dummyDataTypes.put(
- DataTypes.DECIMAL(17, 11),
- DecimalData.fromBigDecimal(new BigDecimal("1088.000"), 17, 11));
- dummyDataTypes.put(DataTypes.CHAR(17), BinaryStringData.fromString("Alice"));
- dummyDataTypes.put(DataTypes.VARCHAR(17), BinaryStringData.fromString("Bob"));
- dummyDataTypes.put(DataTypes.BINARY(17), "Cicada".getBytes());
- dummyDataTypes.put(DataTypes.VARBINARY(17), "Derrida".getBytes());
- dummyDataTypes.put(DataTypes.TIME(9), TimeData.fromMillisOfDay(64801000));
- dummyDataTypes.put(
- DataTypes.TIMESTAMP(9),
- TimestampData.fromTimestamp(Timestamp.valueOf("2020-07-17 18:00:00")));
- dummyDataTypes.put(
- DataTypes.TIMESTAMP_TZ(9),
- ZonedTimestampData.of(364800000, 123456, "Asia/Shanghai"));
- dummyDataTypes.put(
- DataTypes.TIMESTAMP_LTZ(9),
- LocalZonedTimestampData.fromInstant(toInstant("2019-12-31 18:00:00")));
-
- // Build tables list
- List<TableId> tables;
- if (distributedTables) {
- tables =
- IntStream.range(0, numOfTables)
- .mapToObj(
- idx ->
- TableId.tableId(
- "default_namespace",
- "default_database",
- "table_" + idx))
- .collect(Collectors.toList());
- } else {
- tables =
- IntStream.range(0, numOfTables)
- .mapToObj(
- idx ->
- TableId.tableId(
- "default_namespace_subtask_" + subTaskId,
- "default_database",
- "table_" + idx))
- .collect(Collectors.toList());
- }
-
- // Collect all events
- List<Event> allEvents = new ArrayList<>();
- Map<TableId, Schema> headSchemaMap = new HashMap<>();
-
- // Use a mutable counter wrapper for iota in lambdas
- int[] iotaRef = {iotaCounter};
-
- // Phase 1: Create tables and initial data
- Consumer<TableId> createTableConsumer =
- tableId -> {
- Schema initialSchema =
- Schema.newBuilder()
- .physicalColumn("id", DataTypes.STRING())
- .primaryKey("id")
- .partitionKey("id")
- .build();
- CreateTableEvent createTableEvent =
- new CreateTableEvent(tableId, initialSchema);
- headSchemaMap.compute(
- tableId,
- (tbl, schema) ->
- SchemaUtils.applySchemaChangeEvent(
- schema, createTableEvent));
- allEvents.add(createTableEvent);
- for (int i = 0; i < 10; i++) {
- allEvents.add(
- DataChangeEvent.insertEvent(
- tableId,
- generateBinRec(
- headSchemaMap.get(tableId),
- dummyDataTypes,
- subTaskId,
- iotaRef)));
- }
- };
-
- if (parallelism > 1) {
- Collections.shuffle(tables);
- }
- tables.forEach(createTableConsumer);
-
- // Phase 2: Add columns for each data type
- List<DataType> fullTypes = new ArrayList<>(dummyDataTypes.keySet());
- if (parallelism > 1) {
- Collections.shuffle(fullTypes);
- }
-
- for (DataType colType : fullTypes) {
- // Global column
- if (parallelism > 1) {
- Collections.shuffle(tables);
- }
- for (TableId tableId : tables) {
- AddColumnEvent addColumnEvent =
- new AddColumnEvent(
- tableId,
- Collections.singletonList(
- new AddColumnEvent.ColumnWithPosition(
- Column.physicalColumn(
- "col_"
- + colType.getClass()
- .getSimpleName()
- .toLowerCase(),
- colType))));
- headSchemaMap.compute(
- tableId,
- (tbl, schema) ->
- SchemaUtils.applySchemaChangeEvent(schema, addColumnEvent));
- allEvents.add(addColumnEvent);
- allEvents.add(
- DataChangeEvent.insertEvent(
- tableId,
- generateBinRec(
- headSchemaMap.get(tableId),
- dummyDataTypes,
- subTaskId,
- iotaRef)));
- }
-
- // Subtask-specific column
- if (parallelism > 1) {
- Collections.shuffle(tables);
- }
- for (TableId tableId : tables) {
- AddColumnEvent addColumnEvent =
- new AddColumnEvent(
- tableId,
- Collections.singletonList(
- new AddColumnEvent.ColumnWithPosition(
- Column.physicalColumn(
- "subtask_"
- + subTaskId
- + "_col_"
- + colType.getClass()
- .getSimpleName()
- .toLowerCase(),
- colType))));
- headSchemaMap.compute(
- tableId,
- (tbl, schema) ->
- SchemaUtils.applySchemaChangeEvent(schema, addColumnEvent));
- allEvents.add(addColumnEvent);
- allEvents.add(
- DataChangeEvent.insertEvent(
- tableId,
- generateBinRec(
- headSchemaMap.get(tableId),
- dummyDataTypes,
- subTaskId,
- iotaRef)));
- }
- }
-
- return allEvents;
- }
-
- private static BinaryRecordData generateBinRec(
- Schema schema, Map<DataType, Object> dummyDataTypes, int subTaskId, int[] iotaRef) {
- BinaryRecordDataGenerator generator =
- new BinaryRecordDataGenerator(
- schema.getColumnDataTypes().toArray(new DataType[0]));
-
- int arity = schema.getColumnDataTypes().size();
- List<DataType> rowTypes = schema.getColumnDataTypes();
- Object[] rowObjects = new Object[arity];
-
- for (int i = 0; i < arity; i++) {
- DataType type = rowTypes.get(i);
- if (Objects.equals(type, DataTypes.STRING())) {
- rowObjects[i] =
- BinaryStringData.fromString(
- String.format("__$%d$%d$__", subTaskId, iotaRef[0]++));
- } else {
- rowObjects[i] = dummyDataTypes.get(type);
- }
- }
- return generator.generate(rowObjects);
- }
-
- private static Instant toInstant(String ts) {
- return Timestamp.valueOf(ts).toLocalDateTime().atZone(ZoneId.of("UTC")).toInstant();
- }
- }
-
- // ------------------------------ Serializers ------------------------------
-
- /** Serializer for {@link DistributedSourceSplit}. */
- private static class DistributedSourceSplitSerializer
- implements SimpleVersionedSerializer<DistributedSourceSplit> {
-
- private static final int VERSION = 1;
-
- @Override
- public int getVersion() {
- return VERSION;
- }
-
- @Override
- public byte[] serialize(DistributedSourceSplit split) throws IOException {
- try (ByteArrayOutputStream bao = new ByteArrayOutputStream(12)) {
- DataOutputViewStreamWrapper view = new DataOutputViewStreamWrapper(bao);
- view.writeInt(split.splitIndex);
- view.writeInt(split.parallelism);
- view.writeInt(split.eventOffset);
- return bao.toByteArray();
- }
- }
-
- @Override
- public DistributedSourceSplit deserialize(int version, byte[] serialized)
- throws IOException {
- try (ByteArrayInputStream bis = new ByteArrayInputStream(serialized)) {
- DataInputViewStreamWrapper view = new DataInputViewStreamWrapper(bis);
- int splitIndex = view.readInt();
- int parallelism = view.readInt();
- int eventOffset = view.readInt();
- return new DistributedSourceSplit(splitIndex, parallelism, eventOffset);
- }
- }
- }
-
- /** Serializer for the enumerator checkpoint (collection of splits). */
- private static class DistributedSourceEnumeratorSerializer
- implements SimpleVersionedSerializer<Collection<DistributedSourceSplit>> {
-
- private static final int VERSION = 1;
-
- @Override
- public int getVersion() {
- return VERSION;
- }
-
- @Override
- public byte[] serialize(Collection<DistributedSourceSplit> splits) throws IOException {
- try (ByteArrayOutputStream bao = new ByteArrayOutputStream(256);
- DataOutputViewStreamWrapper view = new DataOutputViewStreamWrapper(bao)) {
- view.writeInt(splits.size());
- for (DistributedSourceSplit split : splits) {
- view.writeInt(split.splitIndex);
- view.writeInt(split.parallelism);
- view.writeInt(split.eventOffset);
- }
- return bao.toByteArray();
- }
- }
-
- @Override
- public Collection<DistributedSourceSplit> deserialize(int version, byte[] serialized)
- throws IOException {
- try (ByteArrayInputStream bis = new ByteArrayInputStream(serialized);
- DataInputViewStreamWrapper view = new DataInputViewStreamWrapper(bis)) {
- int size = view.readInt();
- List<DistributedSourceSplit> splits = new ArrayList<>(size);
- for (int i = 0; i < size; i++) {
- int splitIndex = view.readInt();
- int parallelism = view.readInt();
- int eventOffset = view.readInt();
- splits.add(new DistributedSourceSplit(splitIndex, parallelism, eventOffset));
- }
- return splits;
- }
- }
- }
-}
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSourceFunction.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSourceFunction.java
new file mode 100644
index 000000000..d95034e0b
--- /dev/null
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/testsource/source/DistributedSourceFunction.java
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.composer.testsource.source;
+
+import org.apache.flink.cdc.common.data.DecimalData;
+import org.apache.flink.cdc.common.data.LocalZonedTimestampData;
+import org.apache.flink.cdc.common.data.TimeData;
+import org.apache.flink.cdc.common.data.TimestampData;
+import org.apache.flink.cdc.common.data.ZonedTimestampData;
+import org.apache.flink.cdc.common.data.binary.BinaryRecordData;
+import org.apache.flink.cdc.common.data.binary.BinaryStringData;
+import org.apache.flink.cdc.common.event.AddColumnEvent;
+import org.apache.flink.cdc.common.event.CreateTableEvent;
+import org.apache.flink.cdc.common.event.DataChangeEvent;
+import org.apache.flink.cdc.common.event.Event;
+import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.schema.Column;
+import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.types.DataType;
+import org.apache.flink.cdc.common.types.DataTypes;
+import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.runtime.typeutils.BinaryRecordDataGenerator;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.math.BigDecimal;
+import java.sql.Timestamp;
+import java.time.Instant;
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/** Distributed source function for testing purposes only. */
+public class DistributedSourceFunction extends RichParallelSourceFunction<Event> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DistributedSourceFunction.class);
+
+ private int subTaskId;
+ private int parallelism;
+
+ private final int numOfTables;
+ private final boolean distributedTables;
+ private transient Map<DataType, Object> dummyDataTypes;
+ private List<TableId> tables;
+ private transient int iotaCounter;
+
+ public DistributedSourceFunction(int numOfTables, boolean distributedTables) {
+ this.numOfTables = numOfTables;
+ this.distributedTables = distributedTables;
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ super.open(parameters);
+ iotaCounter = 0;
+ subTaskId = getRuntimeContext().getTaskInfo().getIndexOfThisSubtask();
+ parallelism = getRuntimeContext().getTaskInfo().getNumberOfParallelSubtasks();
+ if (distributedTables) {
+ tables =
+ IntStream.range(0, numOfTables)
+ .mapToObj(
+ idx ->
+ TableId.tableId(
+ "default_namespace",
+ "default_database",
+ "table_" + idx))
+ .collect(Collectors.toList());
+ } else {
+ tables =
+ IntStream.range(0, numOfTables)
+ .mapToObj(
+ idx ->
+ TableId.tableId(
+ "default_namespace_subtask_" + subTaskId,
+ "default_database",
+ "table_" + idx))
+ .collect(Collectors.toList());
+ }
+ dummyDataTypes = new LinkedHashMap<>();
+ dummyDataTypes.put(DataTypes.BOOLEAN(), true);
+ dummyDataTypes.put(DataTypes.TINYINT(), (byte) 17);
+ dummyDataTypes.put(DataTypes.SMALLINT(), (short) 34);
+ dummyDataTypes.put(DataTypes.INT(), (int) 68);
+ dummyDataTypes.put(DataTypes.BIGINT(), (long) 136);
+ dummyDataTypes.put(DataTypes.FLOAT(), (float) 272.0);
+ dummyDataTypes.put(DataTypes.DOUBLE(), (double) 544.0);
+ dummyDataTypes.put(
+ DataTypes.DECIMAL(17, 11),
+ DecimalData.fromBigDecimal(new BigDecimal("1088.000"), 17, 11));
+ dummyDataTypes.put(DataTypes.CHAR(17), BinaryStringData.fromString("Alice"));
+ dummyDataTypes.put(DataTypes.VARCHAR(17), BinaryStringData.fromString("Bob"));
+ dummyDataTypes.put(DataTypes.BINARY(17), "Cicada".getBytes());
+ dummyDataTypes.put(DataTypes.VARBINARY(17), "Derrida".getBytes());
+ dummyDataTypes.put(DataTypes.TIME(9), TimeData.fromMillisOfDay(64801000));
+ dummyDataTypes.put(
+ DataTypes.TIMESTAMP(9),
+ TimestampData.fromTimestamp(Timestamp.valueOf("2020-07-17 18:00:00")));
+ dummyDataTypes.put(
+ DataTypes.TIMESTAMP_TZ(9),
+ ZonedTimestampData.of(364800000, 123456, "Asia/Shanghai"));
+ dummyDataTypes.put(
+ DataTypes.TIMESTAMP_LTZ(9),
+ LocalZonedTimestampData.fromInstant(toInstant("2019-12-31 18:00:00")));
+ }
+
+ // Generates statically incrementing data, could be used for data integrity verification.
+ private BinaryStringData iota() {
+ return BinaryStringData.fromString(String.format("__$%d$%d$__", subTaskId, iotaCounter++));
+ }
+
+ private void sendFromTables(Consumer<TableId> tableIdConsumer) {
+ if (parallelism > 1) {
+ // Inject a little randomness in multi-parallelism mode
+ Collections.shuffle(tables);
+ }
+ tables.forEach(tableIdConsumer);
+ }
+
+ @Override
+ public void run(SourceContext<Event> context) throws InterruptedException {
+ Schema initialSchema =
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.STRING())
+ .primaryKey("id")
+ .partitionKey("id")
+ .build();
+
+ Map<TableId, Schema> headSchemaMap = new HashMap<>();
+
+ sendFromTables(
+ tableId -> {
+ CreateTableEvent createTableEvent =
+ new CreateTableEvent(tableId, initialSchema);
+ headSchemaMap.compute(
+ tableId,
+ (tbl, schema) ->
+ SchemaUtils.applySchemaChangeEvent(schema, createTableEvent));
+ collect(context, createTableEvent);
+ for (int i = 0; i < 10; i++) {
+ collect(
+ context,
+ DataChangeEvent.insertEvent(
+ tableId, generateBinRec(headSchemaMap.get(tableId))));
+ }
+ });
+
+ List<DataType> fullTypes = new ArrayList<>(dummyDataTypes.keySet());
+ if (parallelism > 1) {
+ // Inject randomness in multi-partition mode
+ Collections.shuffle(fullTypes);
+ }
+ fullTypes.forEach(
+ colType -> {
+ sendFromTables(
+ tableId -> {
+ AddColumnEvent addColumnEvent =
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "col_"
+ + colType.getClass()
+ .getSimpleName()
+ .toLowerCase(),
+ colType))));
+
+ headSchemaMap.compute(
+ tableId,
+ (tbl, schema) ->
+ SchemaUtils.applySchemaChangeEvent(
+ schema, addColumnEvent));
+ collect(context, addColumnEvent);
+ collect(
+ context,
+ DataChangeEvent.insertEvent(
+ tableId,
+ generateBinRec(headSchemaMap.get(tableId))));
+ });
+
+ sendFromTables(
+ tableId -> {
+ AddColumnEvent addColumnEvent =
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "subtask_"
+ + subTaskId
+ + "_col_"
+ + colType.getClass()
+ .getSimpleName()
+ .toLowerCase(),
+ colType))));
+
+ headSchemaMap.compute(
+ tableId,
+ (tbl, schema) ->
+ SchemaUtils.applySchemaChangeEvent(
+ schema, addColumnEvent));
+ collect(context, addColumnEvent);
+ collect(
+ context,
+ DataChangeEvent.insertEvent(
+ tableId,
+ generateBinRec(headSchemaMap.get(tableId))));
+ });
+ });
+
+ if (parallelism > 1) {
+ // To allow test running correctly, we need to wait for downstream schema evolutions
+ // to finish before closing any subTask.
+ Thread.sleep(10000);
+ }
+ }
+
+ @Override
+ public void cancel() {
+ // Do nothing
+ }
+
+ private BinaryRecordData generateBinRec(Schema schema) {
+ BinaryRecordDataGenerator generator =
+ new BinaryRecordDataGenerator(schema.getColumnDataTypes().toArray(new DataType[0]));
+
+ int arity = schema.getColumnDataTypes().size();
+ List<DataType> rowTypes = schema.getColumnDataTypes();
+ Object[] rowObjects = new Object[arity];
+
+ for (int i = 0; i < arity; i++) {
+ DataType type = rowTypes.get(i);
+ if (Objects.equals(type, DataTypes.STRING())) {
+ rowObjects[i] = iota();
+ } else {
+ rowObjects[i] = dummyDataTypes.get(type);
+ }
+ }
+ return generator.generate(rowObjects);
+ }
+
+ private void collect(SourceContext<Event> sourceContext, Event event) {
+ LOG.info("{}> Emitting event {}", subTaskId, event);
+ sourceContext.collect(event);
+ }
+
+ private Instant toInstant(String ts) {
+ return Timestamp.valueOf(ts).toLocalDateTime().atZone(ZoneId.of("UTC")).toInstant();
+ }
+}
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/main/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactory.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/main/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactory.java
index c7e34183a..14229dcc3 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/main/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactory.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/main/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactory.java
@@ -20,13 +20,13 @@ package org.apache.flink.cdc.connectors.elasticsearch.sink;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.pipeline.PipelineOptions;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.connectors.elasticsearch.config.ElasticsearchSinkOptions;
import org.apache.flink.cdc.connectors.elasticsearch.v2.NetworkConfig;
+import org.apache.flink.table.api.ValidationException;
import org.apache.http.HttpHost;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/test/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/test/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactoryTest.java
index a5443d5c8..ef3691d31 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/test/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-elasticsearch/src/test/java/org/apache/flink/cdc/connectors/elasticsearch/sink/ElasticsearchDataSinkFactoryTest.java
@@ -21,11 +21,11 @@ package org.apache.flink.cdc.connectors.elasticsearch.sink;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/test/java/org/apache/flink/cdc/connectors/fluss/factory/FlussDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/test/java/org/apache/flink/cdc/connectors/fluss/factory/FlussDataSinkFactoryTest.java
index f84714ac7..49ec01d30 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/test/java/org/apache/flink/cdc/connectors/fluss/factory/FlussDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/test/java/org/apache/flink/cdc/connectors/fluss/factory/FlussDataSinkFactoryTest.java
@@ -18,13 +18,13 @@
package org.apache.flink.cdc.connectors.fluss.factory;
import org.apache.flink.cdc.common.configuration.Configuration;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
import org.apache.flink.cdc.connectors.fluss.sink.FlussDataSink;
import org.apache.flink.cdc.connectors.fluss.sink.FlussDataSinkOptions;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-hudi/src/test/java/org/apache/flink/cdc/connectors/hudi/sink/HudiDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-hudi/src/test/java/org/apache/flink/cdc/connectors/hudi/sink/HudiDataSinkFactoryTest.java
index 89d77209d..d6c141626 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-hudi/src/test/java/org/apache/flink/cdc/connectors/hudi/sink/HudiDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-hudi/src/test/java/org/apache/flink/cdc/connectors/hudi/sink/HudiDataSinkFactoryTest.java
@@ -19,11 +19,11 @@ package org.apache.flink.cdc.connectors.hudi.sink;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkFactoryTest.java
index 1752fed14..10f89e16d 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-iceberg/src/test/java/org/apache/flink/cdc/connectors/iceberg/sink/IcebergDataSinkFactoryTest.java
@@ -20,13 +20,13 @@ package org.apache.flink.cdc.connectors.iceberg.sink;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.CreateTableEvent;
import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.common.types.DataTypes;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/test/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/test/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSinkFactoryTest.java
index 3342166c8..f7da80dd2 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/test/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/test/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSinkFactoryTest.java
@@ -18,11 +18,11 @@
package org.apache.flink.cdc.connectors.kafka.sink;
import org.apache.flink.cdc.common.configuration.Configuration;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/factory/MySqlDataSourceFactory.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/factory/MySqlDataSourceFactory.java
index 0ab054379..1b3540da0 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/factory/MySqlDataSourceFactory.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/factory/MySqlDataSourceFactory.java
@@ -21,7 +21,6 @@ import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSourceFactory;
import org.apache.flink.cdc.common.factories.Factory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
@@ -40,6 +39,7 @@ import org.apache.flink.cdc.connectors.mysql.table.StartupOptions;
import org.apache.flink.cdc.connectors.mysql.utils.MySqlSchemaUtils;
import org.apache.flink.cdc.connectors.mysql.utils.OptionUtils;
import org.apache.flink.cdc.debezium.table.DebeziumOptions;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.table.catalog.ObjectPath;
import com.mysql.cj.conf.PropertyKey;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlDataSourceFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlDataSourceFactoryTest.java
index 03f677ec0..75fb6590d 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlDataSourceFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlDataSourceFactoryTest.java
@@ -19,10 +19,10 @@ package org.apache.flink.cdc.connectors.mysql.source;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.Factory;
import org.apache.flink.cdc.connectors.mysql.factory.MySqlDataSourceFactory;
import org.apache.flink.cdc.connectors.mysql.testutils.UniqueDatabase;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.table.catalog.ObjectPath;
import org.junit.jupiter.api.Test;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-oracle/src/main/java/org/apache/flink/cdc/connectors/oracle/factory/OracleDataSourceFactory.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-oracle/src/main/java/org/apache/flink/cdc/connectors/oracle/factory/OracleDataSourceFactory.java
index 7771dcc1f..a2b4647fc 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-oracle/src/main/java/org/apache/flink/cdc/connectors/oracle/factory/OracleDataSourceFactory.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-oracle/src/main/java/org/apache/flink/cdc/connectors/oracle/factory/OracleDataSourceFactory.java
@@ -21,7 +21,6 @@ import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSourceFactory;
import org.apache.flink.cdc.common.factories.Factory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
@@ -36,6 +35,7 @@ import org.apache.flink.cdc.connectors.oracle.source.config.OracleSourceConfig;
import org.apache.flink.cdc.connectors.oracle.source.config.OracleSourceConfigFactory;
import org.apache.flink.cdc.connectors.oracle.table.OracleReadableMetaData;
import org.apache.flink.cdc.connectors.oracle.utils.OracleSchemaUtils;
+import org.apache.flink.table.api.ValidationException;
import java.time.Duration;
import java.util.ArrayList;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonDataSinkFactoryTest.java
index 5f3f929ca..dcbbdd682 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonDataSinkFactoryTest.java
@@ -19,13 +19,13 @@ package org.apache.flink.cdc.connectors.paimon.sink;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.common.sink.FlinkSinkProvider;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
import org.apache.flink.cdc.connectors.paimon.sink.v2.PaimonEventSink;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactory.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactory.java
index 569395ea1..578b6e0f8 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactory.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/main/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactory.java
@@ -21,7 +21,6 @@ import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSourceFactory;
import org.apache.flink.cdc.common.factories.Factory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
@@ -35,6 +34,7 @@ import org.apache.flink.cdc.connectors.postgres.source.PostgresSourceBuilder;
import org.apache.flink.cdc.connectors.postgres.source.config.PostgresSourceConfigFactory;
import org.apache.flink.cdc.connectors.postgres.table.PostgreSQLReadableMetadata;
import org.apache.flink.cdc.connectors.postgres.utils.PostgresSchemaUtils;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.table.data.RowData;
import org.slf4j.Logger;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/test/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/test/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactoryTest.java
index f3780e5fe..c3cce43bc 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/test/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-postgres/src/test/java/org/apache/flink/cdc/connectors/postgres/factory/PostgresDataSourceFactoryTest.java
@@ -20,7 +20,6 @@ package org.apache.flink.cdc.connectors.postgres.factory;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.Factory;
import org.apache.flink.cdc.common.source.SupportedMetadataColumn;
import org.apache.flink.cdc.connectors.postgres.PostgresTestBase;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/test/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksDataSinkFactoryTest.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/test/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksDataSinkFactoryTest.java
index f00b346d8..4e7278254 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/test/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksDataSinkFactoryTest.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/test/java/org/apache/flink/cdc/connectors/starrocks/sink/StarRocksDataSinkFactoryTest.java
@@ -19,11 +19,11 @@ package org.apache.flink.cdc.connectors.starrocks.sink;
import org.apache.flink.cdc.common.configuration.ConfigOption;
import org.apache.flink.cdc.common.configuration.Configuration;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.factories.DataSinkFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/pom.xml b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/pom.xml
deleted file mode 100644
index ebfee3e24..000000000
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/pom.xml
+++ /dev/null
@@ -1,132 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <parent>
- <artifactId>flink-cdc-pipeline-connectors</artifactId>
- <groupId>org.apache.flink</groupId>
- <version>${revision}</version>
- </parent>
- <modelVersion>4.0.0</modelVersion>
-
- <artifactId>flink-cdc-pipeline-connector-values-2.x</artifactId>
-
- <properties>
- <flink.version>${flink.2.x.version}</flink.version>
- </properties>
-
- <dependencies>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-pipeline-connector-values</artifactId>
- <version>${project.version}</version>
- </dependency>
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-jar-plugin</artifactId>
- <executions>
- <execution>
- <id>test-jar</id>
- <goals>
- <goal>test-jar</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-shade-plugin</artifactId>
- <version>${maven.shade.plugin.version}</version>
- <executions>
- <execution>
- <id>shade-flink</id>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- <configuration>
- <shadeTestJar>false</shadeTestJar>
- <artifactSet>
- <includes>
- <include>*:*</include>
- </includes>
- </artifactSet>
- <filters>
- <filter>
- <artifact>org.apache.kafka:*</artifact>
- <excludes>
- <exclude>kafka/kafka-version.properties</exclude>
- <exclude>LICENSE</exclude>
- <!-- Does not contain anything relevant.
- Cites a binary dependency on jersey, but this is neither reflected in the
- dependency graph, nor are any jersey files bundled. -->
- <exclude>NOTICE</exclude>
- <exclude>common/**</exclude>
- </excludes>
- </filter>
- </filters>
- <relocations>
- <relocation>
- <pattern>org.apache.kafka</pattern>
- <shadedPattern>
- org.apache.flink.cdc.connectors.shaded.org.apache.kafka
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.google</pattern>
- <shadedPattern>
- org.apache.flink.cdc.connectors.shaded.com.google
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.fasterxml</pattern>
- <shadedPattern>
- org.apache.flink.cdc.connectors.shaded.com.fasterxml
- </shadedPattern>
- </relocation>
- </relocations>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
-
- <profiles>
- <profile>
- <id>flink2</id>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-dependency-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- </plugin>
- </plugins>
- </build>
- </profile>
- </profiles>
-</project>
\ No newline at end of file
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
deleted file mode 100644
index 72b5b1fa3..000000000
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
+++ /dev/null
@@ -1,187 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.connectors.values.sink;
-
-import org.apache.flink.api.connector.sink2.Sink;
-import org.apache.flink.api.connector.sink2.SinkWriter;
-import org.apache.flink.api.connector.sink2.WriterInitContext;
-import org.apache.flink.cdc.common.annotation.Internal;
-import org.apache.flink.cdc.common.data.RecordData;
-import org.apache.flink.cdc.common.event.ChangeEvent;
-import org.apache.flink.cdc.common.event.CreateTableEvent;
-import org.apache.flink.cdc.common.event.DataChangeEvent;
-import org.apache.flink.cdc.common.event.Event;
-import org.apache.flink.cdc.common.event.SchemaChangeEvent;
-import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.schema.Schema;
-import org.apache.flink.cdc.common.sink.DataSink;
-import org.apache.flink.cdc.common.sink.EventSinkProvider;
-import org.apache.flink.cdc.common.sink.FlinkSinkProvider;
-import org.apache.flink.cdc.common.sink.MetadataApplier;
-import org.apache.flink.cdc.common.utils.SchemaUtils;
-import org.apache.flink.cdc.connectors.values.ValuesDatabase;
-
-import java.io.IOException;
-import java.io.Serializable;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-/** A {@link DataSink} for "values" connector that supports schema evolution. */
-@Internal
-public class ValuesDataSink implements DataSink, Serializable {
-
- /** {@link ValuesDataSinkOptions#MATERIALIZED_IN_MEMORY}. */
- private final boolean materializedInMemory;
-
- private final boolean print;
-
- private final SinkApi sinkApi;
-
- private final boolean errorOnSchemaChange;
-
- public ValuesDataSink(
- boolean materializedInMemory,
- boolean print,
- SinkApi sinkApi,
- boolean errorOnSchemaChange) {
- this.materializedInMemory = materializedInMemory;
- this.print = print;
- this.sinkApi = sinkApi;
- this.errorOnSchemaChange = errorOnSchemaChange;
- }
-
- @Override
- public EventSinkProvider getEventSinkProvider() {
- return FlinkSinkProvider.of(new ValuesSink(materializedInMemory, print));
- }
-
- @Override
- public MetadataApplier getMetadataApplier() {
- if (errorOnSchemaChange) {
- return new ValuesDatabase.ErrorOnChangeMetadataApplier();
- } else {
- return new ValuesDatabase.ValuesMetadataApplier(materializedInMemory);
- }
- }
-
- /** an e2e {@link Sink} implementation that print all {@link DataChangeEvent} out. */
- private static class ValuesSink implements Sink<Event> {
-
- private final boolean materializedInMemory;
-
- private final boolean print;
-
- public ValuesSink(boolean materializedInMemory, boolean print) {
- this.materializedInMemory = materializedInMemory;
- this.print = print;
- }
-
- @Override
- public SinkWriter<Event> createWriter(WriterInitContext context) throws IOException {
- return new ValuesSinkWriter(
- materializedInMemory,
- print,
- context.getTaskInfo().getIndexOfThisSubtask(),
- context.getTaskInfo().getNumberOfParallelSubtasks());
- }
- }
-
- /**
- * Print {@link DataChangeEvent} to console, and update table records in {@link ValuesDatabase}.
- */
- private static class ValuesSinkWriter implements SinkWriter<Event> {
-
- private final boolean materializedInMemory;
-
- private final boolean print;
-
- private final int subtaskIndex;
-
- private final int numSubtasks;
-
- /**
- * keep the relationship of TableId and Schema as write method may rely on the schema
- * information of DataChangeEvent.
- */
- private final Map<TableId, Schema> schemaMaps;
-
- private final Map<TableId, List<RecordData.FieldGetter>> fieldGetterMaps;
-
- public ValuesSinkWriter(
- boolean materializedInMemory, boolean print, int subtaskIndex, int numSubtasks) {
- super();
- this.materializedInMemory = materializedInMemory;
- this.print = print;
- this.subtaskIndex = subtaskIndex;
- this.numSubtasks = numSubtasks;
- schemaMaps = new HashMap<>();
- fieldGetterMaps = new HashMap<>();
- }
-
- @Override
- public void write(Event event, Context context) {
- if (event instanceof SchemaChangeEvent) {
- SchemaChangeEvent schemaChangeEvent = (SchemaChangeEvent) event;
- TableId tableId = schemaChangeEvent.tableId();
- if (event instanceof CreateTableEvent) {
- Schema schema = ((CreateTableEvent) event).getSchema();
- schemaMaps.put(tableId, schema);
- fieldGetterMaps.put(tableId, SchemaUtils.createFieldGetters(schema));
- } else {
- if (!schemaMaps.containsKey(tableId)) {
- throw new RuntimeException("schema of " + tableId + " is not existed.");
- }
- Schema schema =
- SchemaUtils.applySchemaChangeEvent(
- schemaMaps.get(tableId), schemaChangeEvent);
- schemaMaps.put(tableId, schema);
- fieldGetterMaps.put(tableId, SchemaUtils.createFieldGetters(schema));
- }
- } else if (materializedInMemory && event instanceof DataChangeEvent) {
- ValuesDatabase.applyDataChangeEvent((DataChangeEvent) event);
- }
- if (print) {
- TableId tableId = ((ChangeEvent) event).tableId();
- String prefix = numSubtasks > 1 ? subtaskIndex + "> " : "";
- // print the detail message to console for verification.
- System.out.println(
- prefix
- + ValuesDataSinkHelper.convertEventToStr(
- event,
- schemaMaps.get(tableId).getColumnDataTypes(),
- fieldGetterMaps.get(tableId)));
- }
- }
-
- @Override
- public void flush(boolean endOfInput) {}
-
- @Override
- public void close() {}
- }
-
- /** SinkApi which sink based on. */
- public enum SinkApi {
- /** Sink based on SinkFunction. */
- SINK_FUNCTION,
-
- /** Sink based on SinkV2. */
- SINK_V2;
- }
-}
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/resources/META-INF/services/org.apache.flink.cdc.common.factories.Factory b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/resources/META-INF/services/org.apache.flink.cdc.common.factories.Factory
deleted file mode 100644
index 12658f8d1..000000000
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values-2.x/src/main/resources/META-INF/services/org.apache.flink.cdc.common.factories.Factory
+++ /dev/null
@@ -1,16 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-org.apache.flink.cdc.connectors.values.factory.ValuesDataFactory
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/pom.xml b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/pom.xml
index 3eda26dad..96052ee64 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/pom.xml
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/pom.xml
@@ -107,4 +107,39 @@ limitations under the License.
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>flink2</id>
+ <properties>
+ <flink.version>${flink.2.x.version}</flink.version>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java</artifactId>
+ <version>${flink.2.x.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+
</project>
\ No newline at end of file
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
index 3cc498cfa..c15594622 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSink.java
@@ -31,6 +31,7 @@ import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.common.sink.EventSinkProvider;
+import org.apache.flink.cdc.common.sink.FlinkSinkFunctionProvider;
import org.apache.flink.cdc.common.sink.FlinkSinkProvider;
import org.apache.flink.cdc.common.sink.MetadataApplier;
import org.apache.flink.cdc.common.utils.SchemaUtils;
@@ -68,7 +69,12 @@ public class ValuesDataSink implements DataSink, Serializable {
@Override
public EventSinkProvider getEventSinkProvider() {
- return FlinkSinkProvider.of(new ValuesSink(materializedInMemory, print));
+ if (SinkApi.SINK_V2.equals(sinkApi)) {
+ return FlinkSinkProvider.of(new ValuesSink(materializedInMemory, print));
+ } else {
+ return FlinkSinkFunctionProvider.of(
+ new ValuesDataSinkFunction(materializedInMemory, print));
+ }
}
@Override
@@ -92,6 +98,7 @@ public class ValuesDataSink implements DataSink, Serializable {
this.print = print;
}
+ @Override
public SinkWriter<Event> createWriter(InitContext context) {
return new ValuesSinkWriter(
materializedInMemory,
@@ -100,6 +107,7 @@ public class ValuesDataSink implements DataSink, Serializable {
context.getTaskInfo().getNumberOfParallelSubtasks());
}
+ @Override
public SinkWriter<Event> createWriter(WriterInitContext context) throws IOException {
return new ValuesSinkWriter(
materializedInMemory,
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSinkFunction.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSinkFunction.java
new file mode 100644
index 000000000..b33a2d0e9
--- /dev/null
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/sink/ValuesDataSinkFunction.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.values.sink;
+
+import org.apache.flink.api.common.eventtime.Watermark;
+import org.apache.flink.cdc.common.data.RecordData;
+import org.apache.flink.cdc.common.event.ChangeEvent;
+import org.apache.flink.cdc.common.event.CreateTableEvent;
+import org.apache.flink.cdc.common.event.DataChangeEvent;
+import org.apache.flink.cdc.common.event.Event;
+import org.apache.flink.cdc.common.event.SchemaChangeEvent;
+import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.connectors.values.ValuesDatabase;
+import org.apache.flink.streaming.api.functions.sink.SinkFunction;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/** An e2e {@link SinkFunction} implementation that print all {@link DataChangeEvent} out. */
+public class ValuesDataSinkFunction implements SinkFunction<Event> {
+ private final boolean materializedInMemory;
+
+ private final boolean print;
+
+ /**
+ * keep the relationship of TableId and Schema as write method may rely on the schema
+ * information of DataChangeEvent.
+ */
+ private final Map<TableId, Schema> schemaMaps;
+
+ private final Map<TableId, List<RecordData.FieldGetter>> fieldGetterMaps;
+
+ public ValuesDataSinkFunction(boolean materializedInMemory, boolean print) {
+ this.materializedInMemory = materializedInMemory;
+ this.print = print;
+ schemaMaps = new HashMap<>();
+ fieldGetterMaps = new HashMap<>();
+ }
+
+ @Override
+ public void invoke(Event event, Context context) throws Exception {
+ if (event instanceof SchemaChangeEvent) {
+ SchemaChangeEvent schemaChangeEvent = (SchemaChangeEvent) event;
+ TableId tableId = schemaChangeEvent.tableId();
+ if (event instanceof CreateTableEvent) {
+ Schema schema = ((CreateTableEvent) event).getSchema();
+ schemaMaps.put(tableId, schema);
+ fieldGetterMaps.put(tableId, SchemaUtils.createFieldGetters(schema));
+ } else {
+ if (!schemaMaps.containsKey(tableId)) {
+ throw new RuntimeException("schema of " + tableId + " is not existed.");
+ }
+ Schema schema =
+ SchemaUtils.applySchemaChangeEvent(
+ schemaMaps.get(tableId), schemaChangeEvent);
+ schemaMaps.put(tableId, schema);
+ fieldGetterMaps.put(tableId, SchemaUtils.createFieldGetters(schema));
+ }
+ } else if (materializedInMemory && event instanceof DataChangeEvent) {
+ ValuesDatabase.applyDataChangeEvent((DataChangeEvent) event);
+ }
+
+ if (print) {
+ TableId tableId = ((ChangeEvent) event).tableId();
+ // print the detail message to console for verification.
+ System.out.println(
+ ValuesDataSinkHelper.convertEventToStr(
+ event,
+ schemaMaps.get(tableId).getColumnDataTypes(),
+ fieldGetterMaps.get(tableId)));
+ }
+ }
+
+ @Override
+ public void writeWatermark(Watermark watermark) throws Exception {
+ SinkFunction.super.writeWatermark(watermark);
+ }
+
+ @Override
+ public void finish() throws Exception {
+ SinkFunction.super.finish();
+ }
+}
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/test/java/org/apache/flink/cdc/connectors/values/source/ValuesDataSourceITCase.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/test/java/org/apache/flink/cdc/connectors/values/source/ValuesDataSourceITCase.java
index 7bea12a18..640fa9cc3 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/test/java/org/apache/flink/cdc/connectors/values/source/ValuesDataSourceITCase.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/test/java/org/apache/flink/cdc/connectors/values/source/ValuesDataSourceITCase.java
@@ -18,7 +18,6 @@
package org.apache.flink.cdc.connectors.values.source;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
-import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.cdc.common.data.binary.BinaryStringData;
import org.apache.flink.cdc.common.event.CreateTableEvent;
import org.apache.flink.cdc.common.event.DataChangeEvent;
@@ -40,6 +39,7 @@ import org.assertj.core.api.Assertions;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
import java.util.ArrayList;
import java.util.List;
@@ -49,6 +49,7 @@ import java.util.List;
* different enumeration situations of {@link
* org.apache.flink.cdc.connectors.values.source.ValuesDataSourceHelper}.
*/
+@Timeout(value = 60, unit = java.util.concurrent.TimeUnit.SECONDS)
class ValuesDataSourceITCase {
@BeforeEach
@@ -68,7 +69,6 @@ class ValuesDataSourceITCase {
private void executeDataStreamJob(ValuesDataSourceHelper.EventSetId type) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(3000);
- env.setRestartStrategy(RestartStrategies.noRestart());
FlinkSourceProvider sourceProvider =
(FlinkSourceProvider) new ValuesDataSource(type).getEventSourceProvider();
CloseableIterator<Event> events =
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/pom.xml b/flink-cdc-connect/flink-cdc-pipeline-connectors/pom.xml
index d1bb4db33..f7afcf980 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/pom.xml
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/pom.xml
@@ -42,7 +42,6 @@ limitations under the License.
<module>flink-cdc-pipeline-connector-iceberg</module>
<module>flink-cdc-pipeline-connector-fluss</module>
<module>flink-cdc-pipeline-connector-hudi</module>
- <module>flink-cdc-pipeline-connector-values-2.x</module>
</modules>
<dependencies>
diff --git a/flink-cdc-dist/pom.xml b/flink-cdc-dist/pom.xml
index 703e3138c..5e9890bfe 100644
--- a/flink-cdc-dist/pom.xml
+++ b/flink-cdc-dist/pom.xml
@@ -58,6 +58,11 @@ limitations under the License.
<profile>
<id>flink2</id>
<dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <version>${project.version}</version>
+ </dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-guava</artifactId>
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/pom.xml b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/pom.xml
index 2d5a92045..2d2a11259 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/pom.xml
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/pom.xml
@@ -77,7 +77,7 @@ limitations under the License.
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-pipeline-connector-values-2.x</artifactId>
+ <artifactId>flink-cdc-pipeline-connector-values</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
@@ -409,7 +409,7 @@ limitations under the License.
<artifactItem>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-pipeline-connector-values-2.x</artifactId>
+ <artifactId>flink-cdc-pipeline-connector-values</artifactId>
<version>${project.version}</version>
<destFileName>values-cdc-pipeline-connector.jar</destFileName>
<type>jar</type>
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
index 948ae3b8f..b1d57c778 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
@@ -20,7 +20,6 @@ package org.apache.flink.cdc.pipeline.tests.utils;
import org.apache.flink.api.common.JobID;
import org.apache.flink.api.common.JobStatus;
import org.apache.flink.api.common.time.Deadline;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.test.utils.TestUtils;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlContainer;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlVersion;
@@ -29,6 +28,7 @@ import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.RestOptions;
import org.apache.flink.runtime.client.JobStatusMessage;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.util.TestLogger;
import com.github.dockerjava.api.DockerClient;
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java
index 24c72bd2f..59d966bc2 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MaxComputeE2eITCase.java
@@ -19,13 +19,13 @@ package org.apache.flink.cdc.pipeline.tests;
import org.apache.flink.api.common.JobStatus;
import org.apache.flink.api.common.time.Deadline;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.test.utils.TestUtils;
import org.apache.flink.cdc.connectors.maxcompute.options.MaxComputeOptions;
import org.apache.flink.cdc.connectors.maxcompute.utils.MaxComputeUtils;
import org.apache.flink.cdc.pipeline.tests.utils.PipelineTestEnvironment;
import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.runtime.client.JobStatusMessage;
+import org.apache.flink.table.api.ValidationException;
import com.aliyun.odps.Instance;
import com.aliyun.odps.Odps;
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java
index 3de85c4d1..76936b698 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToHudiE2eITCase.java
@@ -20,7 +20,6 @@ package org.apache.flink.cdc.pipeline.tests;
import org.apache.flink.api.common.JobID;
import org.apache.flink.api.common.JobStatus;
import org.apache.flink.api.common.time.Deadline;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.test.utils.TestUtils;
import org.apache.flink.cdc.connectors.mysql.testutils.UniqueDatabase;
import org.apache.flink.cdc.pipeline.tests.utils.PipelineTestEnvironment;
@@ -28,6 +27,7 @@ import org.apache.flink.cdc.pipeline.tests.utils.TarballFetcher;
import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.core.execution.SavepointFormatType;
import org.apache.flink.runtime.client.JobStatusMessage;
+import org.apache.flink.table.api.ValidationException;
import org.apache.hudi.common.model.HoodieTableType;
import org.assertj.core.api.Assertions;
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
index fa3ecec7d..641180580 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
@@ -20,7 +20,6 @@ package org.apache.flink.cdc.pipeline.tests.utils;
import org.apache.flink.api.common.JobID;
import org.apache.flink.api.common.JobStatus;
import org.apache.flink.api.common.time.Deadline;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.test.utils.TestUtils;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlContainer;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlVersion;
@@ -29,6 +28,7 @@ import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.RestOptions;
import org.apache.flink.runtime.client.JobStatusMessage;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.util.TestLogger;
import com.github.dockerjava.api.DockerClient;
diff --git a/flink-cdc-e2e-tests/flink-cdc-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/utils/FlinkContainerTestEnvironment.java b/flink-cdc-e2e-tests/flink-cdc-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/utils/FlinkContainerTestEnvironment.java
index 0d1ae70f6..96173989c 100644
--- a/flink-cdc-e2e-tests/flink-cdc-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/utils/FlinkContainerTestEnvironment.java
+++ b/flink-cdc-e2e-tests/flink-cdc-source-e2e-tests/src/test/java/org/apache/flink/cdc/connectors/tests/utils/FlinkContainerTestEnvironment.java
@@ -19,7 +19,6 @@ package org.apache.flink.cdc.connectors.tests.utils;
import org.apache.flink.api.common.JobStatus;
import org.apache.flink.api.common.time.Deadline;
-import org.apache.flink.cdc.common.exceptions.ValidationException;
import org.apache.flink.cdc.common.test.utils.TestUtils;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlContainer;
import org.apache.flink.cdc.connectors.mysql.testutils.MySqlVersion;
@@ -29,6 +28,7 @@ import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.RestOptions;
import org.apache.flink.runtime.client.JobStatusMessage;
+import org.apache.flink.table.api.ValidationException;
import org.apache.flink.util.TestLogger;
import com.fasterxml.jackson.core.Version;
diff --git a/flink-cdc-cli/pom.xml b/flink-cdc-flink2.2-compat/pom.xml
similarity index 50%
copy from flink-cdc-cli/pom.xml
copy to flink-cdc-flink2.2-compat/pom.xml
index 7934cf576..ffed8f2dc 100644
--- a/flink-cdc-cli/pom.xml
+++ b/flink-cdc-flink2.2-compat/pom.xml
@@ -25,82 +25,71 @@ limitations under the License.
</parent>
<modelVersion>4.0.0</modelVersion>
- <artifactId>flink-cdc-cli</artifactId>
+ <artifactId>flink-cdc-flink2.2-compat</artifactId>
<properties>
- <commons-cli.version>1.7.0</commons-cli.version>
- <snakeyaml.version>2.6</snakeyaml.version>
+ <flink.version>${flink.2.x.version}</flink.version>
+ <shaded.guava.version>${flink.2.x.shaded.guava.version}</shaded.guava.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-common</artifactId>
- <version>${project.version}</version>
- </dependency>
- <!-- YAML parser utilities -->
- <dependency>
- <groupId>org.snakeyaml</groupId>
- <artifactId>snakeyaml-engine</artifactId>
- <version>${snakeyaml.version}</version>
+ <artifactId>flink-table-common</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-composer</artifactId>
- <version>${project.version}</version>
+ <artifactId>flink-connector-base</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
</dependency>
-
<dependency>
- <groupId>commons-cli</groupId>
- <artifactId>commons-cli</artifactId>
- <version>${commons-cli.version}</version>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-api-java-bridge</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
</dependency>
-
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-clients</artifactId>
+ <artifactId>flink-core</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
-
<dependency>
<groupId>org.apache.flink</groupId>
- <artifactId>flink-yarn</artifactId>
+ <artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
- <exclusions>
- <exclusion>
- <groupId>log4j</groupId>
- <artifactId>log4j</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-log4j12</artifactId>
- </exclusion>
- </exclusions>
<scope>provided</scope>
</dependency>
-
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <version>${slf4j.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-shaded-guava</artifactId>
+ <version>${shaded.guava.version}</version>
+ </dependency>
</dependencies>
- <profiles>
- <profile>
- <id>flink2</id>
- <properties>
- <flink.version>${flink.2.x.version}</flink.version>
- </properties>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-dependency-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- </plugin>
- </plugins>
- </build>
- </profile>
- </profiles>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>test-jar</id>
+ <goals>
+ <goal>test-jar</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
</project>
\ No newline at end of file
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/Sink.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/Sink.java
new file mode 100644
index 000000000..f4b5d6d01
--- /dev/null
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/Sink.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.connector.sink2;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.Public;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.operators.MailboxExecutor;
+import org.apache.flink.api.common.operators.ProcessingTimeService;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.util.UserCodeClassLoader;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Optional;
+import java.util.function.Consumer;
+
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+@Public
+public interface Sink<InputT> extends Serializable {
+
+ SinkWriter<InputT> createWriter(WriterInitContext var1) throws IOException;
+
+ /**
+ * @deprecated
+ */
+ @Deprecated
+ SinkWriter<InputT> createWriter(InitContext var1) throws IOException;
+
+ /**
+ * @deprecated
+ */
+ @Deprecated
+ @PublicEvolving
+ public interface InitContext extends org.apache.flink.api.connector.sink2.InitContext {
+ UserCodeClassLoader getUserCodeClassLoader();
+
+ MailboxExecutor getMailboxExecutor();
+
+ ProcessingTimeService getProcessingTimeService();
+
+ SinkWriterMetricGroup metricGroup();
+
+ SerializationSchema.InitializationContext asSerializationSchemaInitializationContext();
+
+ boolean isObjectReuseEnabled();
+
+ <IN> TypeSerializer<IN> createInputSerializer();
+
+ @Experimental
+ default <MetaT> Optional<Consumer<MetaT>> metadataConsumer() {
+ return Optional.empty();
+ }
+ }
+}
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java
similarity index 62%
copy from flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
copy to flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java
index 4e1114971..86d0636a5 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/api/connector/sink2/TwoPhaseCommittingSink.java
@@ -15,15 +15,8 @@
* limitations under the License.
*/
-package org.apache.flink.cdc.common.sink;
+package org.apache.flink.api.connector.sink2;
-import org.apache.flink.api.connector.sink2.Sink;
-import org.apache.flink.cdc.common.annotation.PublicEvolving;
-
-/**
- * A marker interface used to provide an event sink for writing change events to external systems.
- * We can reuse exiting Flink {@link Sink} and Flink {@link SinkFunction} implementation, and we can
- * support our own {@code EventSink} Implementation in the future.
- */
-@PublicEvolving
-public interface EventSinkProvider {}
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public interface TwoPhaseCommittingSink<InputT, CommT>
+ extends Sink<InputT>, SupportsCommitter<CommT> {}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/core/execution/RestoreMode.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/core/execution/RestoreMode.java
new file mode 100644
index 000000000..80227ea9b
--- /dev/null
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/core/execution/RestoreMode.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.execution;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.configuration.DescribedEnum;
+import org.apache.flink.configuration.description.InlineElement;
+import org.apache.flink.configuration.description.TextElement;
+
+/**
+ * Copy from <a
+ * href="https://github.com/apache/flink/blob/release-1.20.3/flink-core/src/main/java/org/apache/flink/core/execution/RestoreMode.java">...</a>.
+ */
+@PublicEvolving
+public enum RestoreMode implements DescribedEnum {
+ CLAIM(
+ "Flink will take ownership of the given snapshot. It will clean the snapshot once it is subsumed by newer ones."),
+ NO_CLAIM(
+ "Flink will not claim ownership of the snapshot files. However it will make sure it does not depend on any artefacts from the restored snapshot. In order to do that, Flink will take the first checkpoint as a full one, which means it might reupload/duplicate files that are part of the restored checkpoint."),
+ /**
+ * @deprecated
+ */
+ @Deprecated
+ LEGACY(
+ "This is the mode in which Flink worked until 1.15. It will not claim ownership of the snapshot and will not delete the files. However, it can directly depend on the existence of the files of the restored checkpoint. It might not be safe to delete checkpoints that were restored in legacy mode. This mode is deprecated, please use CLAIM or NO_CLAIM mode to get a clear state file ownership.");
+
+ private final String description;
+ public static final RestoreMode DEFAULT = NO_CLAIM;
+
+ private RestoreMode(String description) {
+ this.description = description;
+ }
+
+ @Internal
+ public InlineElement getDescription() {
+ return TextElement.text(this.description);
+ }
+}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java
new file mode 100644
index 000000000..f281c0ee4
--- /dev/null
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobgraph;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.docs.Documentation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.core.execution.RestoreMode;
+
+/**
+ * Copy from <a
+ * href="https://github.com/apache/flink/blob/release-1.20.3/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/SavepointConfigOptions.java">...</a>.
+ */
+@Deprecated
+@Documentation.ExcludeFromDocumentation("Hidden for deprecated.")
+@PublicEvolving
+public class SavepointConfigOptions {
+ public static final ConfigOption<String> SAVEPOINT_PATH =
+ ConfigOptions.key("execution.savepoint.path")
+ .stringType()
+ .noDefaultValue()
+ .withDescription(
+ "Path to a savepoint to restore the job from (for example hdfs:///flink/savepoint-1537).");
+ public static final ConfigOption<Boolean> SAVEPOINT_IGNORE_UNCLAIMED_STATE =
+ ConfigOptions.key("execution.savepoint.ignore-unclaimed-state")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ "Allow to skip savepoint state that cannot be restored. Allow this if you removed an operator from your pipeline after the savepoint was triggered.");
+ public static final ConfigOption<RestoreMode> RESTORE_MODE;
+
+ static {
+ RESTORE_MODE =
+ ConfigOptions.key("execution.savepoint-restore-mode")
+ .enumType(RestoreMode.class)
+ .defaultValue(RestoreMode.DEFAULT)
+ .withDescription(
+ "Describes the mode how Flink should restore from the given savepoint or retained checkpoint.");
+ }
+}
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPostCommitTopology.java
similarity index 62%
copy from flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
copy to flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPostCommitTopology.java
index 4e1114971..1f0cb5c86 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPostCommitTopology.java
@@ -15,15 +15,7 @@
* limitations under the License.
*/
-package org.apache.flink.cdc.common.sink;
+package org.apache.flink.streaming.api.connector.sink2;
-import org.apache.flink.api.connector.sink2.Sink;
-import org.apache.flink.cdc.common.annotation.PublicEvolving;
-
-/**
- * A marker interface used to provide an event sink for writing change events to external systems.
- * We can reuse exiting Flink {@link Sink} and Flink {@link SinkFunction} implementation, and we can
- * support our own {@code EventSink} Implementation in the future.
- */
-@PublicEvolving
-public interface EventSinkProvider {}
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public interface WithPostCommitTopology<InputT, CommT> extends SupportsPostCommitTopology<CommT> {}
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreCommitTopology.java
similarity index 62%
copy from flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
copy to flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreCommitTopology.java
index 4e1114971..eda582e47 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreCommitTopology.java
@@ -15,15 +15,8 @@
* limitations under the License.
*/
-package org.apache.flink.cdc.common.sink;
+package org.apache.flink.streaming.api.connector.sink2;
-import org.apache.flink.api.connector.sink2.Sink;
-import org.apache.flink.cdc.common.annotation.PublicEvolving;
-
-/**
- * A marker interface used to provide an event sink for writing change events to external systems.
- * We can reuse exiting Flink {@link Sink} and Flink {@link SinkFunction} implementation, and we can
- * support our own {@code EventSink} Implementation in the future.
- */
-@PublicEvolving
-public interface EventSinkProvider {}
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public interface WithPreCommitTopology<InputT, CommT>
+ extends SupportsPreCommitTopology<CommT, CommT> {}
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreWriteTopology.java
similarity index 66%
copy from flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
copy to flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreWriteTopology.java
index 4e1114971..719c47459 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/connector/sink2/WithPreWriteTopology.java
@@ -15,15 +15,10 @@
* limitations under the License.
*/
-package org.apache.flink.cdc.common.sink;
+package org.apache.flink.streaming.api.connector.sink2;
import org.apache.flink.api.connector.sink2.Sink;
-import org.apache.flink.cdc.common.annotation.PublicEvolving;
-/**
- * A marker interface used to provide an event sink for writing change events to external systems.
- * We can reuse exiting Flink {@link Sink} and Flink {@link SinkFunction} implementation, and we can
- * support our own {@code EventSink} Implementation in the future.
- */
-@PublicEvolving
-public interface EventSinkProvider {}
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public interface WithPreWriteTopology<InputT>
+ extends Sink<InputT>, SupportsPreWriteTopology<InputT> {}
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java
similarity index 62%
copy from flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
copy to flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java
index 4e1114971..e4f457374 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java
@@ -15,15 +15,8 @@
* limitations under the License.
*/
-package org.apache.flink.cdc.common.sink;
+package org.apache.flink.streaming.api.functions.sink;
-import org.apache.flink.api.connector.sink2.Sink;
-import org.apache.flink.cdc.common.annotation.PublicEvolving;
-
-/**
- * A marker interface used to provide an event sink for writing change events to external systems.
- * We can reuse exiting Flink {@link Sink} and Flink {@link SinkFunction} implementation, and we can
- * support our own {@code EventSink} Implementation in the future.
- */
-@PublicEvolving
-public interface EventSinkProvider {}
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public interface SinkFunction<IN>
+ extends org.apache.flink.streaming.api.functions.sink.legacy.SinkFunction<IN> {}
diff --git a/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/RichParallelSourceFunction.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/RichParallelSourceFunction.java
new file mode 100644
index 000000000..127ab3cc9
--- /dev/null
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/RichParallelSourceFunction.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.functions.source;
+
+import org.apache.flink.annotation.Public;
+import org.apache.flink.api.common.functions.AbstractRichFunction;
+import org.apache.flink.api.common.functions.OpenContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.functions.source.legacy.ParallelSourceFunction;
+
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+@Deprecated
+@Public
+public abstract class RichParallelSourceFunction<OUT> extends AbstractRichFunction
+ implements ParallelSourceFunction<OUT>, SourceFunction<OUT> {
+ private static final long serialVersionUID = 1L;
+
+ @Override
+ public void open(OpenContext openContext) throws Exception {
+ super.open(openContext);
+ open(new Configuration());
+ }
+
+ public void open(Configuration parameters) throws Exception {}
+}
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java
similarity index 62%
copy from flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
copy to flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java
index 4e1114971..d61e8a991 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/EventSinkProvider.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/streaming/api/functions/source/SourceFunction.java
@@ -15,15 +15,8 @@
* limitations under the License.
*/
-package org.apache.flink.cdc.common.sink;
+package org.apache.flink.streaming.api.functions.source;
-import org.apache.flink.api.connector.sink2.Sink;
-import org.apache.flink.cdc.common.annotation.PublicEvolving;
-
-/**
- * A marker interface used to provide an event sink for writing change events to external systems.
- * We can reuse exiting Flink {@link Sink} and Flink {@link SinkFunction} implementation, and we can
- * support our own {@code EventSink} Implementation in the future.
- */
-@PublicEvolving
-public interface EventSinkProvider {}
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
+public interface SourceFunction<T>
+ extends org.apache.flink.streaming.api.functions.source.legacy.SourceFunction<T> {}
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/ValidationException.java b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/table/api/ValidationException.java
similarity index 72%
rename from flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/ValidationException.java
rename to flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/table/api/ValidationException.java
index adfbbfa16..fc6a8e37c 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/ValidationException.java
+++ b/flink-cdc-flink2.2-compat/src/main/java/org/apache/flink/table/api/ValidationException.java
@@ -15,18 +15,13 @@
* limitations under the License.
*/
-package org.apache.flink.cdc.common.exceptions;
+package org.apache.flink.table.api;
-import org.apache.flink.cdc.common.annotation.PublicEvolving;
+import org.apache.flink.annotation.PublicEvolving;
import javax.annotation.Nullable;
-/**
- * Exception for validation errors (e.g. invalid options, unsupported options). Defined in
- * flink-cdc-common to avoid runtime dependency on Flink table API (e.g. {@code
- * org.apache.flink.table.api.ValidationException}), which may not be on the classpath when running
- * with different Flink versions.
- */
+/** Compatibility adapter to add class that was present in Flink 1.20 but removed in Flink 2.2. */
@PublicEvolving
public class ValidationException extends RuntimeException {
diff --git a/flink-cdc-runtime/pom.xml b/flink-cdc-runtime/pom.xml
index cb38487c3..baac0b6db 100644
--- a/flink-cdc-runtime/pom.xml
+++ b/flink-cdc-runtime/pom.xml
@@ -109,30 +109,14 @@ limitations under the License.
<properties>
<flink.version>${flink.2.x.version}</flink.version>
</properties>
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <configuration>
- <excludes>
- <!-- SinkFunction is moved to legacy package in Flink 2.x -->
- <exclude>org/apache/flink/cdc/runtime/operators/sink/DataSinkFunctionOperator.java</exclude>
- <exclude>org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkFunctionOperator.java</exclude>
- </excludes>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-dependency-plugin</artifactId>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- </plugin>
- </plugins>
- </build>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-cdc-flink2.2-compat</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
</profile>
</profiles>
-
</project>
\ No newline at end of file
diff --git a/pom.xml b/pom.xml
index b3f8f0789..295538df5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -33,6 +33,7 @@ limitations under the License.
<packaging>pom</packaging>
<modules>
+ <module>flink-cdc-flink2.2-compat</module>
<module>flink-cdc-cli</module>
<module>flink-cdc-common</module>
<module>flink-cdc-composer</module>
(flink-cdc) 03/12: Fix CI failure.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 557cd42c556bd0f3a9b255cf2973b3ab624c50af
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Mon Mar 9 12:47:15 2026 +0800
Fix CI failure.
---
.../flink/cdc/composer/flink/translator/DataSinkTranslator.java | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
index 8a6bde5d1..c53d9f275 100644
--- a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
+++ b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslator.java
@@ -119,9 +119,7 @@ public class DataSinkTranslator {
// Get sink provider
EventSinkProvider eventSinkProvider = dataSink.getEventSinkProvider();
if (eventSinkProvider == null) {
- throw new IllegalStateException(
- "DataSink#getEventSinkProvider() must not return null. "
- + "Please ensure the sink is correctly configured.");
+ return;
}
String sinkName = generateSinkName(sinkDef);
if (eventSinkProvider instanceof FlinkSinkProvider) {
(flink-cdc) 07/12: Fix compile.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 0faf780792e8994b98d3ddd3cce54faedb907025
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Tue Mar 10 08:55:13 2026 +0800
Fix compile.
---
.../flink/cdc/composer/flink/translator/DataSinkTranslatorTest.java | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslatorTest.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslatorTest.java
index 8ca8eb5cf..001f0ba64 100644
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslatorTest.java
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/translator/DataSinkTranslatorTest.java
@@ -18,6 +18,7 @@
package org.apache.flink.cdc.composer.flink.translator;
import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.api.connector.sink2.WriterInitContext;
import org.apache.flink.api.dag.Transformation;
import org.apache.flink.cdc.common.event.Event;
import org.apache.flink.runtime.jobgraph.OperatorID;
@@ -87,5 +88,10 @@ class DataSinkTranslatorTest {
public SinkWriter<Event> createWriter(InitContext context) throws IOException {
return null;
}
+
+ @Override
+ public SinkWriter<Event> createWriter(WriterInitContext context) throws IOException {
+ return null;
+ }
}
}
(flink-cdc) 11/12: Remove flink-cdc-pipeline-e2e-tests-2.x module.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 45948829ed3048c893a581a76529082e16024649
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Wed Mar 11 16:51:57 2026 +0800
Remove flink-cdc-pipeline-e2e-tests-2.x module.
---
.github/workflows/flink_cdc_ci.yml | 2 +-
.github/workflows/modules.py | 11 +-
.../flink-cdc-pipeline-e2e-tests-2.x/pom.xml | 454 ------------------
.../tests/utils/PipelineTestEnvironment.java | 505 ---------------------
.../cdc/pipeline/tests/utils/TarballFetcher.java | 151 ------
.../src/test/resources/docker/mysql/my.cnf | 64 ---
.../src/test/resources/docker/mysql/setup.sql | 30 --
.../src/test/resources/log4j2-test.properties | 25 -
.../flink-cdc-pipeline-e2e-tests/pom.xml | 39 ++
.../flink/cdc/pipeline/tests/ValuesE2eITCase.java | 14 +-
.../cdc/pipeline/tests/utils/TarballFetcher.java | 30 +-
flink-cdc-e2e-tests/pom.xml | 1 -
.../sink/BatchDataSinkWriterOperator.java | 23 +-
.../operators/sink/DataSinkWriterOperator.java | 23 +-
14 files changed, 104 insertions(+), 1268 deletions(-)
diff --git a/.github/workflows/flink_cdc_ci.yml b/.github/workflows/flink_cdc_ci.yml
index 66d2f71be..f7514f7e0 100644
--- a/.github/workflows/flink_cdc_ci.yml
+++ b/.github/workflows/flink_cdc_ci.yml
@@ -108,7 +108,7 @@ jobs:
java-versions: "[11]"
flink-versions: "['2.2.0']"
custom-maven-parameter: "-Pflink2"
- modules: "['pipeline_e2e_2.x']"
+ modules: "['pipeline_e2e']"
parallelism: ${{ matrix.parallelism }}
source_e2e:
name: Source E2E Tests
diff --git a/.github/workflows/modules.py b/.github/workflows/modules.py
index d06c36d02..86e94164f 100755
--- a/.github/workflows/modules.py
+++ b/.github/workflows/modules.py
@@ -144,10 +144,6 @@ MODULES_PIPELINE_E2E = [
"flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests"
]
-MODULES_PIPELINE_E2E_2_X = [
- "flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x"
-]
-
MODULES_SOURCE_E2E = [
"flink-cdc-e2e-tests/flink-cdc-source-e2e-tests"
]
@@ -182,18 +178,13 @@ ALL_MODULES = set(
MODULES_SOURCE_E2E
)
-# Modules that require the flink2 Maven profile to be activated
-ALL_MODULES_FLINK2 = ALL_MODULES | set(MODULES_PIPELINE_E2E_2_X)
-
test_modules = set()
compile_modules = set()
for module in INPUT_MODULES.split(', '):
module_list = set(globals()['MODULES_' + module.upper().replace('-', '_').replace('.', '_')])
test_modules |= module_list
- if module == 'pipeline_e2e_2.x':
- compile_modules |= ALL_MODULES_FLINK2
- elif module == 'source_e2e' or module == 'pipeline_e2e':
+ if module == 'source_e2e' or module == 'pipeline_e2e':
compile_modules |= ALL_MODULES
else:
compile_modules |= module_list
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/pom.xml b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/pom.xml
deleted file mode 100644
index 2d2a11259..000000000
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/pom.xml
+++ /dev/null
@@ -1,454 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <parent>
- <artifactId>flink-cdc-e2e-tests</artifactId>
- <groupId>org.apache.flink</groupId>
- <version>${revision}</version>
- </parent>
- <modelVersion>4.0.0</modelVersion>
-
- <artifactId>flink-cdc-pipeline-e2e-tests-2.x</artifactId>
-
- <properties>
- <flink.version>${flink.2.x.version}</flink.version>
- <mysql.driver.version>8.0.27</mysql.driver.version>
- <flink.release.download.skip>true</flink.release.download.skip>
- <skipITs>true</skipITs>
- <flink.release.name>flink-${flink.version}-bin-scala_${scala.binary.version}.tgz</flink.release.name>
- <flink.release.mirror>https://dlcdn.apache.org/flink/flink-${flink.version}</flink.release.mirror>
- <maven.plugin.download.version>1.6.8</maven.plugin.download.version>
- <jmh.version>1.37</jmh.version>
- </properties>
-
- <dependencies>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-e2e-utils</artifactId>
- <version>${project.version}</version>
- <type>test-jar</type>
- <scope>test</scope>
- </dependency>
-
- <!-- Drivers -->
- <dependency>
- <groupId>mysql</groupId>
- <artifactId>mysql-connector-java</artifactId>
- <exclusions>
- <exclusion>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- </exclusion>
- </exclusions>
- <version>${mysql.driver.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- CDC connectors test utils - 2.x versions -->
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-dist</artifactId>
- <version>${project.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-mysql-cdc</artifactId>
- <version>${project.version}</version>
- <type>test-jar</type>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-pipeline-connector-values</artifactId>
- <version>${project.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-test-util</artifactId>
- <version>${project.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-pipeline-udf-examples</artifactId>
- <version>${project.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- testcontainers -->
- <dependency>
- <groupId>org.testcontainers</groupId>
- <artifactId>testcontainers</artifactId>
- <version>${testcontainers.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.testcontainers</groupId>
- <artifactId>junit-jupiter</artifactId>
- <version>${testcontainers.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.testcontainers</groupId>
- <artifactId>mysql</artifactId>
- <version>${testcontainers.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- benchmark -->
- <dependency>
- <groupId>org.openjdk.jmh</groupId>
- <artifactId>jmh-core</artifactId>
- <version>${jmh.version}</version>
- <scope>test</scope>
- </dependency>
-
- <dependency>
- <groupId>org.openjdk.jmh</groupId>
- <artifactId>jmh-generator-annprocess</artifactId>
- <version>${jmh.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- test dependencies on TestContainers -->
- <dependency>
- <groupId>org.testcontainers</groupId>
- <artifactId>postgresql</artifactId>
- <version>${testcontainers.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- This is for testing Scala UDF.-->
- <dependency>
- <groupId>org.scala-lang</groupId>
- <artifactId>scala-library</artifactId>
- <version>${scala.version}</version>
- <scope>test</scope>
- </dependency>
-
- <dependency>
- <groupId>org.testcontainers</groupId>
- <artifactId>kafka</artifactId>
- <version>${testcontainers.version}</version>
- <scope>test</scope>
- </dependency>
-
- <!-- mini yarn -->
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-yarn-api</artifactId>
- <version>${hadoop.version}</version>
- <scope>test</scope>
- <exclusions>
- <exclusion>
- <!-- This dependency is no longer shipped with the JDK since Java 9.-->
- <groupId>jdk.tools</groupId>
- <artifactId>jdk.tools</artifactId>
- </exclusion>
- <exclusion>
- <groupId>ch.qos.reload4j</groupId>
- <artifactId>reload4j</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-reload4j</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.version}</version>
- <scope>test</scope>
- <exclusions>
- <exclusion>
- <groupId>jdk.tools</groupId>
- <artifactId>jdk.tools</artifactId>
- </exclusion>
- <exclusion>
- <groupId>log4j</groupId>
- <artifactId>log4j</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-log4j12</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-yarn-client</artifactId>
- <version>${hadoop.version}</version>
- <scope>test</scope>
- <exclusions>
- <exclusion>
- <artifactId>jdk.tools</artifactId>
- <groupId>jdk.tools</groupId>
- </exclusion>
- <exclusion>
- <artifactId>log4j</artifactId>
- <groupId>log4j</groupId>
- </exclusion>
- <exclusion>
- <artifactId>slf4j-log4j12</artifactId>
- <groupId>org.slf4j</groupId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-yarn-server-tests</artifactId>
- <version>${hadoop.version}</version>
- <scope>test</scope>
- <type>test-jar</type>
- <exclusions>
- <exclusion>
- <artifactId>jdk.tools</artifactId>
- <groupId>jdk.tools</groupId>
- </exclusion>
- <exclusion>
- <artifactId>log4j</artifactId>
- <groupId>log4j</groupId>
- </exclusion>
- <exclusion>
- <artifactId>slf4j-log4j12</artifactId>
- <groupId>org.slf4j</groupId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.version}</version>
- <scope>test</scope>
- <type>test-jar</type>
- <exclusions>
- <exclusion>
- <artifactId>jdk.tools</artifactId>
- <groupId>jdk.tools</groupId>
- </exclusion>
- <exclusion>
- <artifactId>log4j</artifactId>
- <groupId>log4j</groupId>
- </exclusion>
- <exclusion>
- <artifactId>slf4j-log4j12</artifactId>
- <groupId>org.slf4j</groupId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-yarn</artifactId>
- <version>${flink.version}</version>
- <scope>test</scope>
- </dependency>
- </dependencies>
-
- <build>
- <testResources>
- <testResource>
- <directory>src/test/resources</directory>
- <excludes>
- <exclude>**/flink-cdc.sh</exclude>
- <exclude>**/flink-cdc.yaml</exclude>
- </excludes>
- </testResource>
- </testResources>
- <plugins>
- <!--
- Skip tests in the test phase.
- Tests will run in the integration-test phase after shading is complete.
- -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <configuration>
- <skip>true</skip>
- </configuration>
- </plugin>
-
- <!--
- Configure failsafe to run integration tests AFTER the package phase.
- This ensures tests use the shaded artifacts where Guava and Flink APIs
- have been relocated.
- -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-failsafe-plugin</artifactId>
- <version>3.0.0-M5</version>
- <executions>
- <execution>
- <id>end-to-end-tests</id>
- <goals>
- <goal>integration-test</goal>
- <goal>verify</goal>
- </goals>
- <configuration>
- <includes>
- <include>**/*.*</include>
- </includes>
- <excludes>
- <exclude>**/MysqlE2eWithYarnApplicationITCase.java</exclude>
- </excludes>
- <forkCount>1</forkCount>
- <systemPropertyVariables>
- <moduleDir>${project.basedir}</moduleDir>
- </systemPropertyVariables>
- </configuration>
- </execution>
- <execution>
- <id>run-last-test</id>
- <goals>
- <goal>integration-test</goal>
- <goal>verify</goal>
- </goals>
- <configuration>
- <includes>
- <include>**/MysqlE2eWithYarnApplicationITCase.java</include>
- </includes>
- </configuration>
- </execution>
- </executions>
- </plugin>
- <plugin>
- <groupId>com.googlecode.maven-download-plugin</groupId>
- <artifactId>download-maven-plugin</artifactId>
- <version>1.6.8</version>
- <configuration>
- <cacheDirectory>${maven.plugin.download.cache.path}</cacheDirectory>
- <outputDirectory>${project.build.directory}</outputDirectory>
- <readTimeOut>60000</readTimeOut>
- <retries>3</retries>
- <unpack>true</unpack>
- </configuration>
- <executions>
- <execution>
- <id>download-flink-release</id>
- <goals>
- <goal>wget</goal>
- </goals>
- <phase>test</phase>
- <configuration>
- <skip>${flink.release.download.skip}</skip>
- <url>${flink.release.mirror}/${flink.release.name}</url>
- </configuration>
- </execution>
- </executions>
- </plugin>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-dependency-plugin</artifactId>
- <executions>
- <execution>
- <id>copy-jars</id>
- <phase>test</phase>
- <goals>
- <goal>copy</goal>
- </goals>
- </execution>
- <execution>
- <id>store-classpath-in-target-for-tests</id>
- <phase>process-test-resources</phase>
- <goals>
- <goal>build-classpath</goal>
- </goals>
- <configuration>
- <outputFile>${project.build.directory}/yarn.classpath</outputFile>
- <excludeGroupIds>org.apache.flink</excludeGroupIds>
- </configuration>
- </execution>
- </executions>
- <configuration>
- <artifactItems>
-
- <artifactItem>
- <groupId>mysql</groupId>
- <artifactId>mysql-connector-java</artifactId>
- <version>${mysql.driver.version}</version>
- <destFileName>mysql-driver.jar</destFileName>
- <type>jar</type>
- <outputDirectory>${project.build.directory}/dependencies
- </outputDirectory>
- </artifactItem>
-
- <artifactItem>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-dist</artifactId>
- <version>${project.version}</version>
- <destFileName>flink-cdc-dist.jar</destFileName>
- <type>jar</type>
- <outputDirectory>${project.build.directory}/dependencies
- </outputDirectory>
- </artifactItem>
-
- <artifactItem>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-pipeline-connector-values</artifactId>
- <version>${project.version}</version>
- <destFileName>values-cdc-pipeline-connector.jar</destFileName>
- <type>jar</type>
- <outputDirectory>${project.build.directory}/dependencies
- </outputDirectory>
- </artifactItem>
-
- <artifactItem>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-pipeline-udf-examples</artifactId>
- <version>${project.version}</version>
- <destFileName>udf-examples.jar</destFileName>
- <type>jar</type>
- <outputDirectory>${project.build.directory}/dependencies
- </outputDirectory>
- </artifactItem>
-
- <artifactItem>
- <groupId>org.scala-lang</groupId>
- <artifactId>scala-library</artifactId>
- <version>${scala.version}</version>
- <destFileName>scala-library.jar</destFileName>
- <type>jar</type>
- <outputDirectory>${project.build.directory}/dependencies
- </outputDirectory>
- </artifactItem>
- </artifactItems>
- </configuration>
- </plugin>
- </plugins>
- </build>
-
- <profiles>
- <profile>
- <id>flink2</id>
- <properties>
- <flink.release.download.skip>false</flink.release.download.skip>
- <skipITs>false</skipITs>
- </properties>
- </profile>
- </profiles>
-</project>
\ No newline at end of file
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
deleted file mode 100644
index b1d57c778..000000000
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
+++ /dev/null
@@ -1,505 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.pipeline.tests.utils;
-
-import org.apache.flink.api.common.JobID;
-import org.apache.flink.api.common.JobStatus;
-import org.apache.flink.api.common.time.Deadline;
-import org.apache.flink.cdc.common.test.utils.TestUtils;
-import org.apache.flink.cdc.connectors.mysql.testutils.MySqlContainer;
-import org.apache.flink.cdc.connectors.mysql.testutils.MySqlVersion;
-import org.apache.flink.client.deployment.StandaloneClusterId;
-import org.apache.flink.client.program.rest.RestClusterClient;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.configuration.RestOptions;
-import org.apache.flink.runtime.client.JobStatusMessage;
-import org.apache.flink.table.api.ValidationException;
-import org.apache.flink.util.TestLogger;
-
-import com.github.dockerjava.api.DockerClient;
-import com.github.dockerjava.api.command.ExecCreateCmdResponse;
-import com.github.dockerjava.api.model.Volume;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.testcontainers.DockerClientFactory;
-import org.testcontainers.containers.BindMode;
-import org.testcontainers.containers.Container.ExecResult;
-import org.testcontainers.containers.GenericContainer;
-import org.testcontainers.containers.Network;
-import org.testcontainers.containers.output.FrameConsumerResultCallback;
-import org.testcontainers.containers.output.OutputFrame;
-import org.testcontainers.containers.output.Slf4jLogConsumer;
-import org.testcontainers.containers.output.ToStringConsumer;
-import org.testcontainers.images.builder.Transferable;
-import org.testcontainers.junit.jupiter.Container;
-import org.testcontainers.junit.jupiter.Testcontainers;
-import org.testcontainers.lifecycle.Startables;
-import org.testcontainers.utility.MountableFile;
-
-import javax.annotation.Nullable;
-
-import java.io.File;
-import java.io.IOException;
-import java.net.URL;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.time.Duration;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.List;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
-import java.util.function.Function;
-import java.util.stream.Stream;
-
-import static org.apache.flink.util.Preconditions.checkState;
-
-/** Test environment running pipeline job on Flink containers. */
-@Testcontainers
-public abstract class PipelineTestEnvironment extends TestLogger {
-
- private static final Logger LOG = LoggerFactory.getLogger(PipelineTestEnvironment.class);
-
- protected Integer parallelism = getParallelism();
-
- private int getParallelism() {
- try {
- return Integer.parseInt(System.getProperty("specifiedParallelism"));
- } catch (NumberFormatException ex) {
- LOG.warn(
- "Unable to parse specified parallelism configuration ({} provided). Use 4 by default.",
- System.getProperty("specifiedParallelism"));
- return 4;
- }
- }
-
- // ------------------------------------------------------------------------------------------
- // MySQL Variables (we always use MySQL as the data source for easier verifying)
- // ------------------------------------------------------------------------------------------
- protected static final String MYSQL_TEST_USER = "mysqluser";
- protected static final String MYSQL_TEST_PASSWORD = "mysqlpw";
- protected static final String INTER_CONTAINER_MYSQL_ALIAS = "mysql";
- protected static final Duration EVENT_WAITING_TIMEOUT = Duration.ofMinutes(3);
- protected static final Duration STARTUP_WAITING_TIMEOUT = Duration.ofMinutes(5);
-
- public static final Network NETWORK = Network.newNetwork();
-
- @Container
- protected static final MySqlContainer MYSQL =
- (MySqlContainer)
- new MySqlContainer(MySqlVersion.V8_0)
- .withConfigurationOverride("docker/mysql/my.cnf")
- .withSetupSQL("docker/mysql/setup.sql")
- .withDatabaseName("flink-test")
- .withUsername("flinkuser")
- .withPassword("flinkpw")
- .withNetwork(NETWORK)
- .withNetworkAliases(INTER_CONTAINER_MYSQL_ALIAS)
- .withLogConsumer(new Slf4jLogConsumer(LOG));
-
- // ------------------------------------------------------------------------------------------
- // Flink Variables
- // ------------------------------------------------------------------------------------------
- protected static final int JOB_MANAGER_REST_PORT = 8081;
- protected static final String INTER_CONTAINER_JM_ALIAS = "jobmanager";
- protected static final String INTER_CONTAINER_TM_ALIAS = "taskmanager";
- protected static final List<String> EXTERNAL_PROPS =
- Arrays.asList(
- String.format("jobmanager.rpc.address: %s", INTER_CONTAINER_JM_ALIAS),
- "jobmanager.bind-host: 0.0.0.0",
- "taskmanager.bind-host: 0.0.0.0",
- "rest.bind-address: 0.0.0.0",
- "rest.address: 0.0.0.0",
- "jobmanager.memory.process.size: 1GB",
- "query.server.port: 6125",
- "blob.server.port: 6124",
- "taskmanager.numberOfTaskSlots: 10",
- "parallelism.default: 4",
- "execution.checkpointing.interval: 300",
- "state.backend.type: hashmap",
- "env.java.default-opts.all: --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.sec [...]
- "execution.checkpointing.savepoint-dir: file:///opt/flink",
- "restart-strategy.type: off",
- "pekko.ask.timeout: 60s",
- // Set off-heap memory explicitly to avoid "java.lang.OutOfMemoryError: Direct
- // buffer memory" error.
- "taskmanager.memory.task.off-heap.size: 128mb",
- // Fix `java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error
- // has occurred` error.
- "taskmanager.memory.jvm-metaspace.size: 512mb");
- public static final String FLINK_PROPERTIES = String.join("\n", EXTERNAL_PROPS);
-
- @Nullable protected RestClusterClient<StandaloneClusterId> restClusterClient;
-
- protected GenericContainer<?> jobManager;
- protected GenericContainer<?> taskManager;
- protected Volume sharedVolume = new Volume("/tmp/shared");
-
- protected ToStringConsumer jobManagerConsumer;
-
- protected ToStringConsumer taskManagerConsumer;
-
- protected String flinkVersion = getFlinkVersion();
-
- public static String getFlinkVersion() {
- return "2.2.0";
- }
-
- protected List<String> copyJarToFlinkLib() {
- return Collections.emptyList();
- }
-
- @BeforeEach
- public void before() throws Exception {
- LOG.info("Starting containers...");
- jobManagerConsumer = new ToStringConsumer();
- jobManager =
- new GenericContainer<>(getFlinkDockerImageTag())
- .withCommand("jobmanager")
- .withNetwork(NETWORK)
- .withNetworkAliases(INTER_CONTAINER_JM_ALIAS)
- .withExposedPorts(JOB_MANAGER_REST_PORT)
- .withEnv("FLINK_PROPERTIES", FLINK_PROPERTIES)
- .withCreateContainerCmdModifier(cmd -> cmd.withVolumes(sharedVolume))
- .withLogConsumer(jobManagerConsumer);
-
- List<String> jarToCopy = copyJarToFlinkLib();
- if (!jarToCopy.isEmpty()) {
- for (String jar : jarToCopy) {
- jobManager.withCopyFileToContainer(
- MountableFile.forHostPath(TestUtils.getResource(jar)), "/opt/flink/lib/");
- }
- }
-
- Startables.deepStart(Stream.of(jobManager)).join();
- runInContainerAsRoot(jobManager, "chmod", "0777", "-R", sharedVolume.toString());
- LOG.info("JobManager is started.");
-
- taskManagerConsumer = new ToStringConsumer();
- taskManager =
- new GenericContainer<>(getFlinkDockerImageTag())
- .withCommand("taskmanager")
- .withNetwork(NETWORK)
- .withNetworkAliases(INTER_CONTAINER_TM_ALIAS)
- .withEnv("FLINK_PROPERTIES", FLINK_PROPERTIES)
- .dependsOn(jobManager)
- .withVolumesFrom(jobManager, BindMode.READ_WRITE)
- .withLogConsumer(taskManagerConsumer);
- Startables.deepStart(Stream.of(taskManager)).join();
- runInContainerAsRoot(taskManager, "chmod", "0777", "-R", sharedVolume.toString());
- LOG.info("TaskManager is started.");
-
- TarballFetcher.fetchLatest(jobManager);
- LOG.info("CDC executables deployed.");
- }
-
- @AfterEach
- public void after() {
- if (restClusterClient != null) {
- restClusterClient.close();
- }
- if (jobManager != null) {
- jobManager.stop();
- }
- if (taskManager != null) {
- taskManager.stop();
- }
- }
-
- /**
- * Submits a YAML job to the running cluster with latest CDC version, without from previous
- * savepoint states.
- */
- public JobID submitPipelineJob(String pipelineJob, Path... jars) throws Exception {
- return submitPipelineJob(
- TarballFetcher.CdcVersion.SNAPSHOT, pipelineJob, null, false, jars);
- }
-
- /**
- * Submits a YAML job to the running cluster with specific CDC version, without from previous
- * savepoint states.
- */
- public JobID submitPipelineJob(
- TarballFetcher.CdcVersion version, String pipelineJob, Path... jars) throws Exception {
- return submitPipelineJob(version, pipelineJob, null, false, jars);
- }
-
- /** Submits a YAML job to the running cluster with latest CDC version. */
- public JobID submitPipelineJob(
- String pipelineJob,
- @Nullable String savepointPath,
- boolean allowNonRestoredState,
- Path... jars)
- throws Exception {
- return submitPipelineJob(
- TarballFetcher.CdcVersion.SNAPSHOT,
- pipelineJob,
- savepointPath,
- allowNonRestoredState,
- jars);
- }
-
- public JobID submitPipelineJob(
- TarballFetcher.CdcVersion version,
- String pipelineJob,
- @Nullable String savepointPath,
- boolean allowNonRestoredState,
- Path... jars)
- throws Exception {
-
- // Prepare external JAR dependencies
- List<Path> paths = new ArrayList<>(Arrays.asList(jars));
- List<String> containerPaths = new ArrayList<>();
- paths.add(TestUtils.getResource("mysql-driver.jar"));
-
- for (Path jar : paths) {
- String containerPath = version.workDir() + "/lib/" + jar.getFileName();
- jobManager.copyFileToContainer(MountableFile.forHostPath(jar), containerPath);
- containerPaths.add(containerPath);
- }
-
- containerPaths.add(version.workDir() + "/lib/values-cdc-pipeline-connector.jar");
-
- StringBuilder sb = new StringBuilder();
- for (String containerPath : containerPaths) {
- sb.append(" --jar ").append(containerPath);
- }
-
- jobManager.copyFileToContainer(
- Transferable.of(pipelineJob), version.workDir() + "/conf/pipeline.yaml");
-
- String commands =
- version.workDir()
- + "/bin/flink-cdc.sh "
- + version.workDir()
- + "/conf/pipeline.yaml --flink-home /opt/flink"
- + sb;
-
- if (savepointPath != null) {
- commands += " --from-savepoint " + savepointPath;
- if (allowNonRestoredState) {
- commands += " --allow-nonRestored-state";
- }
- }
- LOG.info("Execute command: {}", commands);
- ExecResult execResult = executeAndCheck(jobManager, commands);
- return Arrays.stream(execResult.getStdout().split("\n"))
- .filter(line -> line.startsWith("Job ID: "))
- .findFirst()
- .map(line -> line.split(": ")[1])
- .map(JobID::fromHexString)
- .orElse(null);
- }
-
- public String stopJobWithSavepoint(JobID jobID) {
- String savepointPath = "/opt/flink/";
- ExecResult result =
- executeAndCheck(
- jobManager,
- "flink",
- "stop",
- jobID.toHexString(),
- "--savepointPath",
- savepointPath);
-
- return Arrays.stream(result.getStdout().split("\n"))
- .filter(line -> line.startsWith("Savepoint completed."))
- .findFirst()
- .map(line -> line.split("Path: file:")[1])
- .orElseThrow(
- () -> new RuntimeException("Failed to parse savepoint path from stdout."));
- }
-
- public void cancelJob(JobID jobID) {
- executeAndCheck(jobManager, "flink", "cancel", jobID.toHexString());
- }
-
- /**
- * Get {@link RestClusterClient} connected to this FlinkContainer.
- *
- * <p>This method lazily initializes the REST client on-demand.
- */
- public RestClusterClient<StandaloneClusterId> getRestClusterClient() {
- if (restClusterClient != null) {
- return restClusterClient;
- }
- checkState(
- jobManager.isRunning(),
- "Cluster client should only be retrieved for a running cluster");
- try {
- final Configuration clientConfiguration = new Configuration();
- clientConfiguration.set(RestOptions.ADDRESS, jobManager.getHost());
- clientConfiguration.set(
- RestOptions.PORT, jobManager.getMappedPort(JOB_MANAGER_REST_PORT));
- this.restClusterClient =
- new RestClusterClient<>(clientConfiguration, StandaloneClusterId.getInstance());
- } catch (Exception e) {
- throw new IllegalStateException(
- "Failed to create client for Flink container cluster", e);
- }
- return restClusterClient;
- }
-
- public void waitUntilJobRunning(Duration timeout) {
- waitUntilJobState(timeout, JobStatus.RUNNING);
- }
-
- public void waitUntilJobFinished(Duration timeout) {
- waitUntilJobState(timeout, JobStatus.FINISHED);
- }
-
- public void waitUntilJobState(Duration timeout, JobStatus expectedStatus) {
- RestClusterClient<?> clusterClient = getRestClusterClient();
- Deadline deadline = Deadline.fromNow(timeout);
- while (deadline.hasTimeLeft()) {
- Collection<JobStatusMessage> jobStatusMessages;
- try {
- jobStatusMessages = clusterClient.listJobs().get(10, TimeUnit.SECONDS);
- } catch (Exception e) {
- LOG.warn("Error when fetching job status.", e);
- continue;
- }
- if (jobStatusMessages != null && !jobStatusMessages.isEmpty()) {
- JobStatusMessage message = jobStatusMessages.iterator().next();
- JobStatus jobStatus = message.getJobState();
- if (!expectedStatus.isTerminalState() && jobStatus.isTerminalState()) {
- throw new ValidationException(
- String.format(
- "Job has been terminated! JobName: %s, JobID: %s, Status: %s",
- message.getJobName(),
- message.getJobId(),
- message.getJobState()));
- } else if (jobStatus == expectedStatus) {
- return;
- }
- }
- }
- }
-
- protected String getFlinkDockerImageTag() {
- if (System.getProperty("java.specification.version").equals("17")) {
- return String.format("flink:%s-scala_2.12-java17", flinkVersion);
- }
- return String.format("flink:%s-scala_2.12-java11", flinkVersion);
- }
-
- private ExecResult executeAndCheck(GenericContainer<?> container, String... command) {
- String joinedCommand = String.join(" ", command);
- try {
- LOG.info("Executing command {}", joinedCommand);
- ExecResult execResult =
- container.execInContainer("bash", "-c", String.join(" ", command));
- LOG.info(execResult.getStdout());
- if (execResult.getExitCode() == 0) {
- LOG.info("Command executed successfully.");
- return execResult;
- } else {
- LOG.error(execResult.getStderr());
- throw new AssertionError(
- "Failed when submitting the pipeline job.\n"
- + "Exit code: "
- + execResult.getExitCode()
- + "\n"
- + "StdOut: "
- + execResult.getStdout()
- + "\n"
- + "StdErr: "
- + execResult.getStderr());
- }
- } catch (Exception e) {
- throw new RuntimeException(
- "Failed to execute command " + joinedCommand + " in container " + container);
- }
- }
-
- public void runInContainerAsRoot(GenericContainer<?> container, String... command)
- throws InterruptedException {
- ToStringConsumer stdoutConsumer = new ToStringConsumer();
- ToStringConsumer stderrConsumer = new ToStringConsumer();
- DockerClient dockerClient = DockerClientFactory.instance().client();
- ExecCreateCmdResponse execCreateCmdResponse =
- dockerClient
- .execCreateCmd(container.getContainerId())
- .withUser("root")
- .withCmd(command)
- .exec();
- FrameConsumerResultCallback callback = new FrameConsumerResultCallback();
- callback.addConsumer(OutputFrame.OutputType.STDOUT, stdoutConsumer);
- callback.addConsumer(OutputFrame.OutputType.STDERR, stderrConsumer);
- dockerClient.execStartCmd(execCreateCmdResponse.getId()).exec(callback).awaitCompletion();
- }
-
- protected List<String> readLines(String resource) throws IOException {
- final URL url = PipelineTestEnvironment.class.getClassLoader().getResource(resource);
- assert url != null;
- Path path = new File(url.getFile()).toPath();
- return Files.readAllLines(path);
- }
-
- protected void validateResult(String... expectedEvents) throws Exception {
- validateResult(Function.identity(), expectedEvents);
- }
-
- protected void validateResult(Function<String, String> mapper, String... expectedEvents)
- throws Exception {
- validateResult(
- taskManagerConsumer, Stream.of(expectedEvents).map(mapper).toArray(String[]::new));
- }
-
- protected void validateResult(ToStringConsumer consumer, String... expectedEvents)
- throws Exception {
- for (String event : expectedEvents) {
- waitUntilSpecificEvent(consumer, event);
- }
- }
-
- protected void validateResult(
- ToStringConsumer consumer, Function<String, String> mapper, String... expectedEvents)
- throws Exception {
- validateResult(consumer, Stream.of(expectedEvents).map(mapper).toArray(String[]::new));
- }
-
- protected void waitUntilSpecificEvent(String event) throws Exception {
- waitUntilSpecificEvent(taskManagerConsumer, event);
- }
-
- protected void waitUntilSpecificEvent(ToStringConsumer consumer, String event)
- throws Exception {
- boolean result = false;
- long endTimeout = System.currentTimeMillis() + EVENT_WAITING_TIMEOUT.toMillis();
- while (System.currentTimeMillis() < endTimeout) {
- String stdout = consumer.toUtf8String();
- if (stdout.contains(event + "\n")) {
- result = true;
- break;
- }
- Thread.sleep(1000);
- }
- if (!result) {
- throw new TimeoutException(
- "failed to get specific event: "
- + event
- + " from stdout: "
- + consumer.toUtf8String());
- }
- }
-}
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/TarballFetcher.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/TarballFetcher.java
deleted file mode 100644
index 8bd16d805..000000000
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/TarballFetcher.java
+++ /dev/null
@@ -1,151 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.pipeline.tests.utils;
-
-import org.apache.flink.cdc.common.test.utils.TestUtils;
-
-import org.apache.commons.io.FileUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.testcontainers.containers.GenericContainer;
-import org.testcontainers.utility.MountableFile;
-
-import java.net.URL;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.time.Duration;
-import java.util.Arrays;
-import java.util.List;
-
-/** Obtain and downloads corresponding Flink CDC tarball files. */
-public abstract class TarballFetcher {
-
- private static final Logger LOG = LoggerFactory.getLogger(TarballFetcher.class);
-
- public static void fetchAll(GenericContainer<?> container) throws Exception {
- fetch(container, CdcVersion.values());
- }
-
- public static void fetchLatest(GenericContainer<?> container) throws Exception {
- fetch(container, CdcVersion.SNAPSHOT);
- }
-
- public static void fetch(GenericContainer<?> container, CdcVersion... versions)
- throws Exception {
- for (CdcVersion version : versions) {
- TarballFetcher.fetchInternal(container, version);
- }
- }
-
- private static void fetchInternal(GenericContainer<?> container, CdcVersion version)
- throws Exception {
- LOG.info("Trying to download CDC tarball @ {}...", version);
- if (CdcVersion.SNAPSHOT.equals(version)) {
- LOG.info("CDC {} is a snapshot version, we should fetch it locally...", version);
-
- container.copyFileToContainer(
- MountableFile.forHostPath(
- TestUtils.getResource("flink-cdc.sh", "flink-cdc-dist", "src"), 0755),
- version.workDir() + "/bin/flink-cdc.sh");
- container.copyFileToContainer(
- MountableFile.forHostPath(
- TestUtils.getResource("flink-cdc.yaml", "flink-cdc-dist", "src"), 0755),
- version.workDir() + "/conf/flink-cdc.yaml");
- container.copyFileToContainer(
- MountableFile.forHostPath(TestUtils.getResource("flink-cdc-dist.jar")),
- version.workDir() + "/lib/flink-cdc-dist.jar");
- container.copyFileToContainer(
- MountableFile.forHostPath(
- TestUtils.getResource("values-cdc-pipeline-connector.jar")),
- version.workDir() + "/lib/values-cdc-pipeline-connector.jar");
-
- } else {
- LOG.info("CDC {} is a released version, download it from the Internet...", version);
-
- String containerPath = "/tmp/tarball/" + version.getVersion() + ".tar.gz";
- downloadAndCopyToContainer(container, version.tarballUrl(), containerPath);
- container.execInContainer("mkdir", "-p", version.workDir());
- container.execInContainer(
- "tar", "-xzvf", containerPath, "-C", version.workDir(), "--strip-components=1");
-
- downloadAndCopyToContainer(
- container,
- version.connectorJarUrl("values"),
- version.workDir() + "/lib/values-cdc-pipeline-connector.jar");
- }
- }
-
- private static void downloadAndCopyToContainer(
- GenericContainer<?> container, String url, String containerPath) throws Exception {
- Path tempFile = Files.createTempFile("download-", ".tmp");
- FileUtils.copyURLToFile(
- new URL(url),
- tempFile.toFile(),
- (int) Duration.ofMinutes(1).toMillis(),
- (int) Duration.ofMinutes(5).toMillis());
- container.copyFileToContainer(MountableFile.forHostPath(tempFile), containerPath);
- }
-
- /** Enum for all released Flink CDC version tags. */
- public enum CdcVersion {
- V3_1_1("3.1.1"),
- V3_2_0("3.2.0"),
- V3_2_1("3.2.1"),
- V3_3_0("3.3.0"),
- V3_4_0("3.4.0"),
- V3_5_0("3.5.0"),
- SNAPSHOT("SNAPSHOT");
-
- private final String version;
-
- CdcVersion(String version) {
- this.version = version;
- }
-
- public String getVersion() {
- return version;
- }
-
- public static List<CdcVersion> getAllVersions() {
- return Arrays.asList(CdcVersion.values());
- }
-
- public static List<CdcVersion> getVersionsSince(CdcVersion version) {
- return getAllVersions()
- .subList(getAllVersions().indexOf(version), getAllVersions().size());
- }
-
- public String tarballUrl() {
- return "https://dlcdn.apache.org/flink/flink-cdc-"
- + version
- + "/flink-cdc-"
- + version
- + "-bin.tar.gz";
- }
-
- public String workDir() {
- return "/tmp/cdc/" + version;
- }
-
- public String connectorJarUrl(String name) {
- return String.format(
- "https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-%s/%s/flink-cdc-pipeline-connector-%s-%s.jar",
- name, version, name, version);
- }
- }
-}
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/my.cnf b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/my.cnf
deleted file mode 100644
index 26d1abad6..000000000
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/my.cnf
+++ /dev/null
@@ -1,64 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# For advice on how to change settings please see
-# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
-
-[mysqld]
-#
-# Remove leading # and set to the amount of RAM for the most important data
-# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
-# innodb_buffer_pool_size = 128M
-#
-# Remove leading # to turn on a very important data integrity option: logging
-# changes to the binary log between backups.
-# log_bin
-#
-# Remove leading # to set options mainly useful for reporting servers.
-# The server defaults are faster for transactions and fast SELECTs.
-# Adjust sizes as needed, experiment to find the optimal values.
-# join_buffer_size = 128M
-# sort_buffer_size = 2M
-# read_rnd_buffer_size = 2M
-skip-host-cache
-skip-name-resolve
-#datadir=/var/lib/mysql
-#socket=/var/lib/mysql/mysql.sock
-#secure-file-priv=/var/lib/mysql-files
-secure-file-priv=/var/lib/mysql
-user=mysql
-
-# Disabling symbolic-links is recommended to prevent assorted security risks
-symbolic-links=0
-
-#log-error=/var/log/mysqld.log
-#pid-file=/var/run/mysqld/mysqld.pid
-
-# ----------------------------------------------
-# Enable the binlog for replication & CDC
-# ----------------------------------------------
-
-# Enable binary replication log and set the prefix, expiration, and log format.
-# The prefix is arbitrary, expiration can be short for integration tests but would
-# be longer on a production system. Row-level info is required for ingest to work.
-# Server ID is required, but this will vary on production systems
-server-id = 223344
-log_bin = mysql-bin
-expire_logs_days = 1
-binlog_format = row
-
-# enable gtid mode
-gtid_mode = on
-enforce_gtid_consistency = on
\ No newline at end of file
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/setup.sql b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/setup.sql
deleted file mode 100644
index 3c0cccb4b..000000000
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/docker/mysql/setup.sql
+++ /dev/null
@@ -1,30 +0,0 @@
--- Licensed to the Apache Software Foundation (ASF) under one or more
--- contributor license agreements. See the NOTICE file distributed with
--- this work for additional information regarding copyright ownership.
--- The ASF licenses this file to You under the Apache License, Version 2.0
--- (the "License"); you may not use this file except in compliance with
--- the License. You may obtain a copy of the License at
---
--- http://www.apache.org/licenses/LICENSE-2.0
---
--- Unless required by applicable law or agreed to in writing, software
--- distributed under the License is distributed on an "AS IS" BASIS,
--- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
--- See the License for the specific language governing permissions and
--- limitations under the License.
-
--- In production you would almost certainly limit the replication user must be on the follower (slave) machine,
--- to prevent other clients accessing the log from other machines. For example, 'replicator'@'follower.acme.com'.
--- However, in this database we'll grant 2 users different privileges:
---
--- 1) 'flinkuser' - all privileges required by the snapshot reader AND binlog reader (used for testing)
--- 2) 'mysqluser' - all privileges
---
-GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO 'flinkuser'@'%';
-CREATE USER 'mysqluser' IDENTIFIED BY 'mysqlpw';
-GRANT ALL PRIVILEGES ON *.* TO 'mysqluser'@'%';
-
--- ----------------------------------------------------------------------------------------------------------------
--- DATABASE: emptydb
--- ----------------------------------------------------------------------------------------------------------------
-CREATE DATABASE emptydb;
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/log4j2-test.properties b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/log4j2-test.properties
deleted file mode 100644
index 32df1c025..000000000
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/resources/log4j2-test.properties
+++ /dev/null
@@ -1,25 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-# http://www.apache.org/licenses/LICENSE-2.0
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-# Set root logger level to ERROR to not flood build logs
-# set manually to INFO for debugging purposes
-rootLogger.level=ERROR
-rootLogger.appenderRef.test.ref = TestLogger
-
-appender.testlogger.name = TestLogger
-appender.testlogger.type = CONSOLE
-appender.testlogger.target = SYSTEM_ERR
-appender.testlogger.layout.type = PatternLayout
-appender.testlogger.layout.pattern = %-4r [%t] %-5p %c - %m%n
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/pom.xml b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/pom.xml
index 8c263f8de..897d1acc3 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/pom.xml
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/pom.xml
@@ -805,4 +805,43 @@ limitations under the License.
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>flink2</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>default-test</id>
+ <phase>none</phase>
+ </execution>
+ <execution>
+ <id>integration-tests</id>
+ <phase>none</phase>
+ </execution>
+ <execution>
+ <id>end-to-end-tests</id>
+ <configuration>
+ <includes>
+ <include>**/ValuesE2eITCase.java</include>
+ </includes>
+ <excludes>
+ <exclude>**/MysqlE2eWithYarnApplicationITCase.java</exclude>
+ </excludes>
+ </configuration>
+ </execution>
+ <execution>
+ <id>run-last-test</id>
+ <phase>none</phase>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/ValuesE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/ValuesE2eITCase.java
similarity index 97%
rename from flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/ValuesE2eITCase.java
rename to flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/ValuesE2eITCase.java
index 7d863333d..0f7bc28cb 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests-2.x/src/test/java/org/apache/flink/cdc/pipeline/tests/ValuesE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/ValuesE2eITCase.java
@@ -47,7 +47,7 @@ class ValuesE2eITCase extends PipelineTestEnvironment {
submitPipelineJob(pipelineJob);
waitUntilJobFinished(Duration.ofSeconds(30));
- LOG.info("Pipeline job is running");
+ LOG.info("Pipeline job has finished");
validateResult(
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
@@ -79,7 +79,7 @@ class ValuesE2eITCase extends PipelineTestEnvironment {
submitPipelineJob(pipelineJob);
waitUntilJobFinished(Duration.ofSeconds(30));
- LOG.info("Pipeline job is running");
+ LOG.info("Pipeline job has finished");
validateResult(
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
@@ -115,7 +115,7 @@ class ValuesE2eITCase extends PipelineTestEnvironment {
submitPipelineJob(pipelineJob);
waitUntilJobFinished(Duration.ofSeconds(30));
- LOG.info("Pipeline job is running");
+ LOG.info("Pipeline job has finished");
validateResult(
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
@@ -151,7 +151,7 @@ class ValuesE2eITCase extends PipelineTestEnvironment {
submitPipelineJob(pipelineJob);
waitUntilJobFinished(Duration.ofSeconds(30));
- LOG.info("Pipeline job is running");
+ LOG.info("Pipeline job has finished");
validateResult(
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
@@ -186,7 +186,7 @@ class ValuesE2eITCase extends PipelineTestEnvironment {
submitPipelineJob(pipelineJob);
waitUntilJobFinished(Duration.ofSeconds(30));
- LOG.info("Pipeline job is running");
+ LOG.info("Pipeline job has finished");
validateResult(
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
@@ -214,7 +214,7 @@ class ValuesE2eITCase extends PipelineTestEnvironment {
submitPipelineJob(pipelineJob);
waitUntilJobFinished(Duration.ofSeconds(30));
- LOG.info("Pipeline job is running");
+ LOG.info("Pipeline job has finished");
validateResult(
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
@@ -246,7 +246,7 @@ class ValuesE2eITCase extends PipelineTestEnvironment {
submitPipelineJob(pipelineJob);
waitUntilJobFinished(Duration.ofSeconds(30));
- LOG.info("Pipeline job is running");
+ LOG.info("Pipeline job has finished");
validateResult(
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING}, primaryKeys=col1, options=()}",
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/TarballFetcher.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/TarballFetcher.java
index 1d5788918..450f683d6 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/TarballFetcher.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/TarballFetcher.java
@@ -19,10 +19,16 @@ package org.apache.flink.cdc.pipeline.tests.utils;
import org.apache.flink.cdc.common.test.utils.TestUtils;
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.command.ExecCreateCmdResponse;
import org.apache.commons.io.FileUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.testcontainers.DockerClientFactory;
import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.output.FrameConsumerResultCallback;
+import org.testcontainers.containers.output.OutputFrame;
+import org.testcontainers.containers.output.ToStringConsumer;
import org.testcontainers.utility.MountableFile;
import java.net.URL;
@@ -60,11 +66,11 @@ public abstract class TarballFetcher {
container.copyFileToContainer(
MountableFile.forHostPath(
- TestUtils.getResource("flink-cdc.sh", "flink-cdc-dist", "src"), 755),
+ TestUtils.getResource("flink-cdc.sh", "flink-cdc-dist", "src")),
version.workDir() + "/bin/flink-cdc.sh");
container.copyFileToContainer(
MountableFile.forHostPath(
- TestUtils.getResource("flink-cdc.yaml", "flink-cdc-dist", "src"), 755),
+ TestUtils.getResource("flink-cdc.yaml", "flink-cdc-dist", "src")),
version.workDir() + "/conf/flink-cdc.yaml");
container.copyFileToContainer(
MountableFile.forHostPath(TestUtils.getResource("flink-cdc-dist.jar")),
@@ -78,6 +84,9 @@ public abstract class TarballFetcher {
TestUtils.getResource("values-cdc-pipeline-connector.jar")),
version.workDir() + "/lib/values-cdc-pipeline-connector.jar");
+ // Ensure the script has execute permission
+ runInContainerAsRoot(container, "chmod", "+x", version.workDir() + "/bin/flink-cdc.sh");
+
} else {
LOG.info("CDC {} is a released version, download it from the Internet...", version);
@@ -109,6 +118,23 @@ public abstract class TarballFetcher {
container.copyFileToContainer(MountableFile.forHostPath(tempFile), containerPath);
}
+ private static void runInContainerAsRoot(GenericContainer<?> container, String... command)
+ throws InterruptedException {
+ ToStringConsumer stdoutConsumer = new ToStringConsumer();
+ ToStringConsumer stderrConsumer = new ToStringConsumer();
+ DockerClient dockerClient = DockerClientFactory.instance().client();
+ ExecCreateCmdResponse execCreateCmdResponse =
+ dockerClient
+ .execCreateCmd(container.getContainerId())
+ .withUser("root")
+ .withCmd(command)
+ .exec();
+ FrameConsumerResultCallback callback = new FrameConsumerResultCallback();
+ callback.addConsumer(OutputFrame.OutputType.STDOUT, stdoutConsumer);
+ callback.addConsumer(OutputFrame.OutputType.STDERR, stderrConsumer);
+ dockerClient.execStartCmd(execCreateCmdResponse.getId()).exec(callback).awaitCompletion();
+ }
+
/** Enum for all released Flink CDC version tags. */
public enum CdcVersion {
V3_1_1("3.1.1"),
diff --git a/flink-cdc-e2e-tests/pom.xml b/flink-cdc-e2e-tests/pom.xml
index 155c6b735..28d16beee 100644
--- a/flink-cdc-e2e-tests/pom.xml
+++ b/flink-cdc-e2e-tests/pom.xml
@@ -33,7 +33,6 @@ limitations under the License.
<module>flink-cdc-e2e-utils</module>
<module>flink-cdc-source-e2e-tests</module>
<module>flink-cdc-pipeline-e2e-tests</module>
- <module>flink-cdc-pipeline-e2e-tests-2.x</module>
</modules>
<build>
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java
index 6ea1ceaee..e89f3f814 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/BatchDataSinkWriterOperator.java
@@ -180,24 +180,26 @@ public class BatchDataSinkWriterOperator<CommT>
}
private Object createFlinkWriterOperator() {
+ Class<?> flinkWriterClass;
try {
- Class<?> flinkWriterClass =
+ flinkWriterClass =
getRuntimeContext()
.getUserCodeClassLoader()
.loadClass(
"org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator");
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException("Failed to load SinkWriterOperator class", e);
+ }
+
+ try {
Constructor<?> constructor =
flinkWriterClass.getDeclaredConstructor(
Sink.class, ProcessingTimeService.class, MailboxExecutor.class);
constructor.setAccessible(true);
return constructor.newInstance(sink, processingTimeService, mailboxExecutor);
- } catch (Exception ignore) {
+ } catch (NoSuchMethodException e) {
+ // Constructor with 3 parameters not found, try the 4-parameter version
try {
- Class<?> flinkWriterClass =
- getRuntimeContext()
- .getUserCodeClassLoader()
- .loadClass(
- "org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator");
Constructor<?> constructor =
flinkWriterClass.getDeclaredConstructor(
StreamOperatorParameters.class,
@@ -206,9 +208,12 @@ public class BatchDataSinkWriterOperator<CommT>
MailboxExecutor.class);
constructor.setAccessible(true);
return constructor.newInstance(null, sink, processingTimeService, mailboxExecutor);
- } catch (Exception e) {
- throw new RuntimeException("Failed to create SinkWriterOperator in Flink", e);
+ } catch (Exception ex) {
+ throw new RuntimeException("Failed to create SinkWriterOperator in Flink", ex);
}
+ } catch (Exception e) {
+ // Other exceptions (e.g., InvocationTargetException) indicate real failures
+ throw new RuntimeException("Failed to create SinkWriterOperator in Flink", e);
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
index 0c45b57c2..bac2929ce 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java
@@ -265,24 +265,26 @@ public class DataSinkWriterOperator<CommT>
}
private Object createFlinkWriterOperator() {
+ Class<?> flinkWriterClass;
try {
- Class<?> flinkWriterClass =
+ flinkWriterClass =
getRuntimeContext()
.getUserCodeClassLoader()
.loadClass(
"org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator");
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException("Failed to load SinkWriterOperator class", e);
+ }
+
+ try {
Constructor<?> constructor =
flinkWriterClass.getDeclaredConstructor(
Sink.class, ProcessingTimeService.class, MailboxExecutor.class);
constructor.setAccessible(true);
return constructor.newInstance(sink, processingTimeService, mailboxExecutor);
- } catch (Exception ignore) {
+ } catch (NoSuchMethodException e) {
+ // Constructor with 3 parameters not found, try the 4-parameter version
try {
- Class<?> flinkWriterClass =
- getRuntimeContext()
- .getUserCodeClassLoader()
- .loadClass(
- "org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator");
Constructor<?> constructor =
flinkWriterClass.getDeclaredConstructor(
StreamOperatorParameters.class,
@@ -291,9 +293,12 @@ public class DataSinkWriterOperator<CommT>
MailboxExecutor.class);
constructor.setAccessible(true);
return constructor.newInstance(null, sink, processingTimeService, mailboxExecutor);
- } catch (Exception e) {
- throw new RuntimeException("Failed to create SinkWriterOperator in Flink", e);
+ } catch (Exception ex) {
+ throw new RuntimeException("Failed to create SinkWriterOperator in Flink", ex);
}
+ } catch (Exception e) {
+ // Other exceptions (e.g., InvocationTargetException) indicate real failures
+ throw new RuntimeException("Failed to create SinkWriterOperator in Flink", e);
}
}
(flink-cdc) 06/12: Add flink-cdc-flink2.2-compat module.
Posted by ku...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kunni pushed a commit to branch FLINK-38729-2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 60acff8c086b04e1b1f28fde2395537b79aa52cb
Author: lvyanquan <lv...@alibaba-inc.com>
AuthorDate: Tue Mar 10 02:01:28 2026 +0800
Add flink-cdc-flink2.2-compat module.
---
flink-cdc-common/pom.xml | 12 ++++++++++++
flink-cdc-composer/pom.xml | 10 ----------
flink-cdc-runtime/pom.xml | 12 ++++++++++++
3 files changed, 24 insertions(+), 10 deletions(-)
diff --git a/flink-cdc-common/pom.xml b/flink-cdc-common/pom.xml
index 8721e620e..4101ad51b 100644
--- a/flink-cdc-common/pom.xml
+++ b/flink-cdc-common/pom.xml
@@ -50,6 +50,18 @@ limitations under the License.
<properties>
<flink.version>${flink.2.x.version}</flink.version>
</properties>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
diff --git a/flink-cdc-composer/pom.xml b/flink-cdc-composer/pom.xml
index ef3040f49..7d0244e6a 100644
--- a/flink-cdc-composer/pom.xml
+++ b/flink-cdc-composer/pom.xml
@@ -104,16 +104,6 @@ limitations under the License.
</properties>
<build>
<plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <configuration>
- <testExcludes>
- <!-- Sink V2 topology interfaces (WithPreWriteTopology etc.) are removed in Flink 2.x -->
- <testExclude>org/apache/flink/cdc/composer/flink/translator/DataSinkTranslatorTest.java</testExclude>
- </testExcludes>
- </configuration>
- </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
diff --git a/flink-cdc-runtime/pom.xml b/flink-cdc-runtime/pom.xml
index baac0b6db..2a2072f0b 100644
--- a/flink-cdc-runtime/pom.xml
+++ b/flink-cdc-runtime/pom.xml
@@ -109,6 +109,18 @@ limitations under the License.
<properties>
<flink.version>${flink.2.x.version}</flink.version>
</properties>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>