You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by le...@apache.org on 2024/08/27 16:53:39 UTC
(flink-cdc) branch release-3.2 updated (ab64eb3b5 -> 1ebefd6d2)
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a change to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
from ab64eb3b5 [FLINK-36056][cdc][connector/elasticsearch] Change flink.connector.elasticsearch.version to a released version (#3543)
new 74f58cd31 [FLINK-36082][pipeline-connector][kafka] Fix lamda NotSerializableException in KafkaDataSink
new 568888af7 [FLINK-36088][pipeline-connector][paimon] Fix NPE in BucketAssignOperator when job restoration
new 9f358ab1d [FLINK-36076][minor][cdc-runtime] Set isSchemaChangeApplying as volatile for thread safe consideration
new 39afc4fc8 [FLINK-36081][cdc-connector][mysql] Remove the schemas of outdated tables in the BinlogSplit when restart (#3548)
new 5a8075744 [FLINK-36094][cdc-runtime] Improve the Exception that SchemaRegistryRequestHandler thrown
new 6dd1c11fe [FLINK-36114][cdc-runtime] Make SchemaRegistryRequestHandler thread safe by blocking subsequent schemaChangeEvent
new dacbe7c34 [FLINK-36092][cdc-runtime] Fix schema evolution failure with wildcarded transform rule
new 803d43865 [hotfix][cdc-runtime] Fix schema registry hanging in multiple parallelism
new daf28ddc9 [FLINK-35056][cdc-connector/sqlserver] Fix scale mapping from SQL Server TIMESTAMP to Flink SQL timestamp
new debd43cdd [FLINK-36128][cdc-runtime] Fix potential unrecoverable in-flight data exception by promoting LENIENT as the default schema change behavior
new 9816848c4 [hotfix][cdc-runtime] Keep upstream pending requests in order to avoid checkpoint hanging
new 1ebefd6d2 [hotfix] Run schema coordinator logic asynchronously to avoid blocking the main thread
The 12 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../cli/parser/YamlPipelineDefinitionParser.java | 38 +-
.../parser/YamlPipelineDefinitionParserTest.java | 36 +-
.../cdc/common/event/SchemaChangeEventType.java | 20 +-
.../common/exceptions/SchemaEvolveException.java | 6 +-
.../UnsupportedSchemaChangeEventException.java | 28 +-
.../flink/cdc/common/pipeline/PipelineOptions.java | 2 +-
.../flink/cdc/common/utils/ChangeEventUtils.java | 8 +-
.../apache/flink/cdc/common/utils/SchemaUtils.java | 71 ++
.../cdc/common/utils/ChangeEventUtilsTest.java | 67 +-
.../cdc/composer/flink/FlinkPipelineComposer.java | 3 +-
.../flink/translator/DataSourceTranslator.java | 7 +-
.../flink/FlinkPipelineComposerITCase.java | 29 +-
.../flink/FlinkPipelineTransformITCase.java | 635 +++++++++++++++++
.../cdc/composer/flink/FlinkPipelineUdfITCase.java | 41 +-
.../cdc/connectors/kafka/sink/KafkaDataSink.java | 35 +-
.../kafka/sink/KafkaMetaDataApplier.java | 36 +-
.../source/reader/MySqlPipelineRecordEmitter.java | 62 +-
.../mysql/source/MySqlFullTypesITCase.java | 26 +-
.../mysql/source/MySqlPipelineITCase.java | 30 +-
.../mysql/testutils/MySqSourceTestUtils.java | 19 +
.../sink/v2/bucket/BucketAssignOperator.java | 2 +-
.../cdc/connectors/values/ValuesDatabase.java | 3 +-
.../mysql/source/split/MySqlBinlogSplit.java | 31 +-
.../mysql/source/split/MySqlBinlogSplitTest.java | 140 ++++
.../sqlserver/source/utils/SqlServerTypeUtils.java | 4 +-
.../read/fetch/SqlServerScanFetchTaskTest.java | 26 +
.../src/test/resources/ddl/pk.sql} | 19 +-
.../cdc/pipeline/tests/MySqlToDorisE2eITCase.java | 10 +-
.../flink/cdc/pipeline/tests/MysqlE2eITCase.java | 243 ++++---
.../flink/cdc/pipeline/tests/RouteE2eITCase.java | 37 +-
.../cdc/pipeline/tests/SchemaEvolveE2eITCase.java | 34 +-
....java => SchemaEvolvingTransformE2eITCase.java} | 98 +--
.../cdc/pipeline/tests/TransformE2eITCase.java | 388 +++++++++-
.../flink/cdc/pipeline/tests/UdfE2eITCase.java | 8 +-
.../tests/utils/PipelineTestEnvironment.java | 2 +
.../tests/SchemaRegistryMigrationMock.java | 2 +
.../runtime/operators/schema/SchemaOperator.java | 145 ++--
.../schema/coordinator/SchemaManager.java | 84 +++
.../schema/coordinator/SchemaRegistry.java | 270 ++++---
.../schema/coordinator/SchemaRegistryProvider.java | 54 +-
.../coordinator/SchemaRegistryRequestHandler.java | 424 ++++++-----
.../event/ApplyEvolvedSchemaChangeRequest.java | 73 --
.../event/ApplyEvolvedSchemaChangeResponse.java | 32 -
.../event/ApplyOriginalSchemaChangeRequest.java | 71 --
.../event/ApplyOriginalSchemaChangeResponse.java | 31 -
.../schema/event/RefreshPendingListsRequest.java | 27 -
.../schema/event/RefreshPendingListsResponse.java | 26 -
.../schema/event/ReleaseUpstreamRequest.java | 32 -
.../event/SchemaChangeProcessingResponse.java | 4 +-
.../schema/event/SchemaChangeRequest.java | 15 +-
.../schema/event/SchemaChangeResponse.java | 75 +-
...sponse.java => SchemaChangeResultResponse.java} | 48 +-
.../operators/transform/PostTransformOperator.java | 55 +-
.../operators/transform/PreTransformOperator.java | 101 ++-
.../flink/cdc/runtime/parser/TransformParser.java | 22 +
.../runtime/partitioning/PrePartitionOperator.java | 13 +-
.../runtime/operators/schema/SchemaEvolveTest.java | 14 +-
.../transform/PreTransformOperatorTest.java | 11 +-
.../TransformOperatorWithSchemaEvolveTest.java | 792 +++++++++++++++++++++
.../operators/EventOperatorTestHarness.java | 44 +-
.../MockedOperatorCoordinatorContext.java | 29 +-
.../schema/CollectingMetadataApplier.java | 4 +-
62 files changed, 3575 insertions(+), 1167 deletions(-)
copy flink-cdc-common/src/main/java/org/apache/flink/cdc/common/sink/MetadataApplier.java => flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/main/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaMetaDataApplier.java (56%)
create mode 100644 flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/split/MySqlBinlogSplitTest.java
copy flink-cdc-connect/flink-cdc-source-connectors/{flink-connector-vitess-cdc/src/test/resources/ddl/inventory.sql => flink-connector-sqlserver-cdc/src/test/resources/ddl/pk.sql} (76%)
copy flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/{SchemaEvolveE2eITCase.java => SchemaEvolvingTransformE2eITCase.java} (75%)
delete mode 100644 flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyEvolvedSchemaChangeRequest.java
delete mode 100644 flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyEvolvedSchemaChangeResponse.java
delete mode 100644 flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyOriginalSchemaChangeRequest.java
delete mode 100644 flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyOriginalSchemaChangeResponse.java
delete mode 100644 flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/RefreshPendingListsRequest.java
delete mode 100644 flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/RefreshPendingListsResponse.java
delete mode 100644 flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ReleaseUpstreamRequest.java
rename flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/{ReleaseUpstreamResponse.java => SchemaChangeResultResponse.java} (53%)
create mode 100644 flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/transform/TransformOperatorWithSchemaEvolveTest.java
copy flink-cdc-connect/flink-cdc-source-connectors/flink-connector-oracle-cdc/src/main/java/org/apache/flink/cdc/connectors/oracle/source/reader/fetch/StoppableChangeEventSourceContext.java => flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/MockedOperatorCoordinatorContext.java (52%)
(flink-cdc) 05/12: [FLINK-36094][cdc-runtime] Improve the Exception that SchemaRegistryRequestHandler thrown
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 5a80757443ecc816ff6db0cb7618aaa4d4bce9b7
Author: Hongshun Wang <12...@users.noreply.github.com>
AuthorDate: Thu Aug 22 09:58:55 2024 +0800
[FLINK-36094][cdc-runtime] Improve the Exception that SchemaRegistryRequestHandler thrown
This closes #3558.
(cherry picked from commit 6205a5a0f16d2cf72ff751573351c4e15ea59efb)
---
.../common/exceptions/SchemaEvolveException.java | 6 +-
.../UnsupportedSchemaChangeEventException.java | 28 +++++-
.../cdc/connectors/values/ValuesDatabase.java | 3 +-
.../flink/cdc/pipeline/tests/MysqlE2eITCase.java | 2 +-
.../cdc/pipeline/tests/SchemaEvolveE2eITCase.java | 23 +++--
.../cdc/pipeline/tests/TransformE2eITCase.java | 2 +-
.../flink/cdc/pipeline/tests/UdfE2eITCase.java | 2 +-
.../runtime/operators/schema/SchemaOperator.java | 65 +-----------
.../schema/coordinator/SchemaRegistry.java | 110 +++++++++++----------
.../coordinator/SchemaRegistryRequestHandler.java | 56 +++++++----
.../event/ApplyEvolvedSchemaChangeRequest.java | 73 --------------
.../event/ApplyEvolvedSchemaChangeResponse.java | 32 ------
.../event/ApplyOriginalSchemaChangeRequest.java | 71 -------------
.../event/ApplyOriginalSchemaChangeResponse.java | 31 ------
.../schema/event/ReleaseUpstreamResponse.java | 42 +-------
.../runtime/operators/schema/SchemaEvolveTest.java | 2 +
.../operators/EventOperatorTestHarness.java | 11 +--
.../schema/CollectingMetadataApplier.java | 4 +-
18 files changed, 149 insertions(+), 414 deletions(-)
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/SchemaEvolveException.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/SchemaEvolveException.java
index fc5b482f1..52950fb32 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/SchemaEvolveException.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/SchemaEvolveException.java
@@ -24,9 +24,9 @@ import javax.annotation.Nullable;
/** An exception occurred during schema evolution. */
public class SchemaEvolveException extends FlinkRuntimeException {
- private final SchemaChangeEvent applyingEvent;
- private final String exceptionMessage;
- private final @Nullable Throwable cause;
+ protected final SchemaChangeEvent applyingEvent;
+ protected final String exceptionMessage;
+ protected final @Nullable Throwable cause;
public SchemaEvolveException(SchemaChangeEvent applyingEvent, String exceptionMessage) {
this(applyingEvent, exceptionMessage, null);
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/UnsupportedSchemaChangeEventException.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/UnsupportedSchemaChangeEventException.java
index 4be20525a..55d53e7d9 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/UnsupportedSchemaChangeEventException.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/exceptions/UnsupportedSchemaChangeEventException.java
@@ -19,10 +19,36 @@ package org.apache.flink.cdc.common.exceptions;
import org.apache.flink.cdc.common.event.SchemaChangeEvent;
+import javax.annotation.Nullable;
+
/** A special kind of {@link SchemaEvolveException} that sink doesn't support such event type. */
public class UnsupportedSchemaChangeEventException extends SchemaEvolveException {
public UnsupportedSchemaChangeEventException(SchemaChangeEvent applyingEvent) {
- super(applyingEvent, "Sink doesn't support such schema change event.", null);
+ this(applyingEvent, "Sink doesn't support such schema change event.");
+ }
+
+ public UnsupportedSchemaChangeEventException(
+ SchemaChangeEvent applyingEvent, String exceptionMessage) {
+ this(applyingEvent, exceptionMessage, null);
+ }
+
+ public UnsupportedSchemaChangeEventException(
+ SchemaChangeEvent applyingEvent, String exceptionMessage, @Nullable Throwable cause) {
+ super(applyingEvent, exceptionMessage, cause);
+ }
+
+ @Override
+ public String toString() {
+ return "UnsupportedSchemaChangeEventException{"
+ + "applyingEvent="
+ + applyingEvent
+ + ", exceptionMessage='"
+ + exceptionMessage
+ + '\''
+ + ", cause='"
+ + cause
+ + '\''
+ + '}';
}
}
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/ValuesDatabase.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/ValuesDatabase.java
index e19c4a844..bbe36a25a 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/ValuesDatabase.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/ValuesDatabase.java
@@ -29,6 +29,7 @@ import org.apache.flink.cdc.common.event.SchemaChangeEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEventType;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.exceptions.SchemaEvolveException;
+import org.apache.flink.cdc.common.exceptions.UnsupportedSchemaChangeEventException;
import org.apache.flink.cdc.common.schema.Column;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.sink.MetadataApplier;
@@ -150,7 +151,7 @@ public class ValuesDatabase {
tableId, ((CreateTableEvent) schemaChangeEvent).getSchema()));
}
} else {
- throw new SchemaEvolveException(
+ throw new UnsupportedSchemaChangeEventException(
schemaChangeEvent,
"Rejected schema change event since error.on.schema.change is enabled.",
null);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
index ada384e07..917873bb8 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
@@ -252,7 +252,7 @@ public class MysqlE2eITCase extends PipelineTestEnvironment {
long endTimeout = System.currentTimeMillis() + MysqlE2eITCase.EVENT_WAITING_TIMEOUT;
while (System.currentTimeMillis() < endTimeout) {
String stdout = taskManagerConsumer.toUtf8String();
- if (stdout.contains(event)) {
+ if (stdout.contains(event + "\n")) {
result = true;
break;
}
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
index d187853da..c84a8824f 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
@@ -110,9 +110,9 @@ public class SchemaEvolveE2eITCase extends PipelineTestEnvironment {
true,
false,
false,
- Collections.singletonList(
- "java.lang.IllegalStateException: Incompatible types: \"INT\" and \"DOUBLE\""),
- Collections.singletonList(
+ Collections.emptyList(),
+ Arrays.asList(
+ "java.lang.IllegalStateException: Incompatible types: \"INT\" and \"DOUBLE\"",
"org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy"));
}
@@ -123,11 +123,10 @@ public class SchemaEvolveE2eITCase extends PipelineTestEnvironment {
false,
true,
false,
- Collections.singletonList(
- "AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}"),
+ Collections.emptyList(),
Arrays.asList(
- "Failed to apply schema change event AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}.",
- "SchemaEvolveException{applyingEvent=AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}, exceptionMessage='Rejected schema change event since error.on.schema.change is enabled.', cause='null'}",
+ "Failed to apply schema change AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]} to table %s.members. Caused by: UnsupportedSchemaChangeEventException{applyingEvent=AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}, exceptionMessage='Rejected schema change event since error.on.schema.change is enabl [...]
+ "UnsupportedSchemaChangeEventException{applyingEvent=AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}, exceptionMessage='Rejected schema change event since error.on.schema.change is enabled.', cause='null'}",
"org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy"));
}
@@ -143,8 +142,8 @@ public class SchemaEvolveE2eITCase extends PipelineTestEnvironment {
"DataChangeEvent{tableId=%s.members, before=[], after=[1012, Eve, 17], op=INSERT, meta=()}",
"DataChangeEvent{tableId=%s.members, before=[], after=[1013, Fiona, null], op=INSERT, meta=()}"),
Arrays.asList(
- "Failed to apply schema change AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]} to table %s.members.",
- "SchemaEvolveException{applyingEvent=AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}, exceptionMessage='Rejected schema change event since error.on.schema.change is enabled.', cause='null'}"));
+ "Failed to apply schema change AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]} to table %s.members. Caused by: UnsupportedSchemaChangeEventException{applyingEvent=AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}, exceptionMessage='Rejected schema change event since error.on.schema.change is enabl [...]
+ "UnsupportedSchemaChangeEventException{applyingEvent=AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}, exceptionMessage='Rejected schema change event since error.on.schema.change is enabled.', cause='null'}"));
}
@Test
@@ -181,7 +180,7 @@ public class SchemaEvolveE2eITCase extends PipelineTestEnvironment {
false,
false,
Arrays.asList(
- "AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}",
+ "AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=LAST, existedColumnName=null}]}",
"DataChangeEvent{tableId=%s.members, before=[], after=[1012, Eve, 17, 0], op=INSERT, meta=()}",
"AlterColumnTypeEvent{tableId=%s.members, nameMapping={age=DOUBLE}}",
"AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`precise_age` DOUBLE, position=LAST, existedColumnName=null}]}",
@@ -348,7 +347,7 @@ public class SchemaEvolveE2eITCase extends PipelineTestEnvironment {
List<String> expectedJmEvents =
expectedJobManagerEvents.stream()
- .map(s -> String.format(s, dbName, dbName))
+ .map(s -> String.format(s, dbName, dbName, dbName))
.collect(Collectors.toList());
validateResult(expectedJmEvents, jobManagerConsumer);
@@ -392,7 +391,7 @@ public class SchemaEvolveE2eITCase extends PipelineTestEnvironment {
long endTimeout = System.currentTimeMillis() + SchemaEvolveE2eITCase.EVENT_WAITING_TIMEOUT;
while (System.currentTimeMillis() < endTimeout) {
String stdout = consumer.toUtf8String();
- if (stdout.contains(event)) {
+ if (stdout.contains(event + "\n")) {
result = true;
break;
}
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
index e27977153..6f612ca95 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
@@ -736,7 +736,7 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
long endTimeout = System.currentTimeMillis() + timeout;
while (System.currentTimeMillis() < endTimeout) {
String stdout = taskManagerConsumer.toUtf8String();
- if (stdout.contains(event)) {
+ if (stdout.contains(event + "\n")) {
result = true;
break;
}
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java
index c965e4dec..938e2d98e 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java
@@ -386,7 +386,7 @@ public class UdfE2eITCase extends PipelineTestEnvironment {
long endTimeout = System.currentTimeMillis() + timeout;
while (System.currentTimeMillis() < endTimeout) {
String stdout = taskManagerConsumer.toUtf8String();
- if (stdout.contains(event)) {
+ if (stdout.contains(event + "\n")) {
result = true;
break;
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java
index d3a9e158b..a700fd39c 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java
@@ -17,7 +17,6 @@
package org.apache.flink.cdc.runtime.operators.schema;
-import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.annotation.VisibleForTesting;
@@ -29,7 +28,6 @@ import org.apache.flink.cdc.common.event.FlushEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEventType;
import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.exceptions.SchemaEvolveException;
import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.route.RouteRule;
import org.apache.flink.cdc.common.schema.Column;
@@ -40,8 +38,6 @@ import org.apache.flink.cdc.common.types.DataTypeFamily;
import org.apache.flink.cdc.common.types.DataTypeRoot;
import org.apache.flink.cdc.common.utils.ChangeEventUtils;
import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry;
-import org.apache.flink.cdc.runtime.operators.schema.event.ApplyEvolvedSchemaChangeRequest;
-import org.apache.flink.cdc.runtime.operators.schema.event.ApplyOriginalSchemaChangeRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.CoordinationResponseUtils;
import org.apache.flink.cdc.runtime.operators.schema.event.RefreshPendingListsRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.ReleaseUpstreamRequest;
@@ -431,56 +427,9 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
ReleaseUpstreamResponse schemaEvolveResponse = requestReleaseUpstream();
List<SchemaChangeEvent> finishedSchemaChangeEvents =
schemaEvolveResponse.getFinishedSchemaChangeEvents();
- List<Tuple2<SchemaChangeEvent, Throwable>> failedSchemaChangeEvents =
- schemaEvolveResponse.getFailedSchemaChangeEvents();
- List<SchemaChangeEvent> ignoredSchemaChangeEvents =
- schemaEvolveResponse.getIgnoredSchemaChangeEvents();
-
- if (schemaChangeBehavior == SchemaChangeBehavior.EVOLVE
- || schemaChangeBehavior == SchemaChangeBehavior.EXCEPTION) {
- if (schemaEvolveResponse.hasException()) {
- throw new RuntimeException(
- String.format(
- "Failed to apply schema change event %s.\nExceptions: %s",
- schemaChangeEvent,
- schemaEvolveResponse.getPrintableFailedSchemaChangeEvents()));
- }
- } else if (schemaChangeBehavior == SchemaChangeBehavior.TRY_EVOLVE
- || schemaChangeBehavior == SchemaChangeBehavior.LENIENT
- || schemaChangeBehavior == SchemaChangeBehavior.IGNORE) {
- if (schemaEvolveResponse.hasException()) {
- schemaEvolveResponse
- .getFailedSchemaChangeEvents()
- .forEach(
- e ->
- LOG.warn(
- "Failed to apply event {}, but keeps running in tolerant mode. Caused by: {}",
- e.f0,
- e.f1));
- }
- } else {
- throw new SchemaEvolveException(
- schemaChangeEvent,
- "Unexpected schema change behavior: " + schemaChangeBehavior);
- }
// Update evolved schema changes based on apply results
- requestApplyEvolvedSchemaChanges(tableId, finishedSchemaChangeEvents);
finishedSchemaChangeEvents.forEach(e -> output.collect(new StreamRecord<>(e)));
-
- LOG.info(
- "Applied schema change event {} to downstream. Among {} total evolved events, {} succeeded, {} failed, and {} ignored.",
- schemaChangeEvent,
- expectedSchemaChangeEvents.size(),
- finishedSchemaChangeEvents.size(),
- failedSchemaChangeEvents.size(),
- ignoredSchemaChangeEvents.size());
-
- schemaOperatorMetrics.increaseFinishedSchemaChangeEvents(
- finishedSchemaChangeEvents.size());
- schemaOperatorMetrics.increaseFailedSchemaChangeEvents(failedSchemaChangeEvents.size());
- schemaOperatorMetrics.increaseIgnoredSchemaChangeEvents(
- ignoredSchemaChangeEvents.size());
}
}
@@ -489,16 +438,6 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
return sendRequestToCoordinator(new SchemaChangeRequest(tableId, schemaChangeEvent));
}
- private void requestApplyOriginalSchemaChanges(
- TableId tableId, SchemaChangeEvent schemaChangeEvent) {
- sendRequestToCoordinator(new ApplyOriginalSchemaChangeRequest(tableId, schemaChangeEvent));
- }
-
- private void requestApplyEvolvedSchemaChanges(
- TableId tableId, List<SchemaChangeEvent> schemaChangeEvents) {
- sendRequestToCoordinator(new ApplyEvolvedSchemaChangeRequest(tableId, schemaChangeEvents));
- }
-
private ReleaseUpstreamResponse requestReleaseUpstream()
throws InterruptedException, TimeoutException {
CoordinationResponse coordinationResponse =
@@ -538,7 +477,7 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
return optionalSchema.get();
} catch (Exception e) {
throw new IllegalStateException(
- String.format("Unable to get latest schema for table \"%s\"", tableId));
+ String.format("Unable to get latest schema for table \"%s\"", tableId), e);
}
}
@@ -553,7 +492,7 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
return optionalSchema.get();
} catch (Exception e) {
throw new IllegalStateException(
- String.format("Unable to get latest schema for table \"%s\"", tableId));
+ String.format("Unable to get latest schema for table \"%s\"", tableId), e);
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
index cc7c68207..9087ae4b3 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
@@ -17,15 +17,13 @@
package org.apache.flink.cdc.runtime.operators.schema.coordinator;
+import org.apache.flink.cdc.common.annotation.VisibleForTesting;
+import org.apache.flink.cdc.common.event.SchemaChangeEvent;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.route.RouteRule;
import org.apache.flink.cdc.common.sink.MetadataApplier;
import org.apache.flink.cdc.runtime.operators.schema.SchemaOperator;
-import org.apache.flink.cdc.runtime.operators.schema.event.ApplyEvolvedSchemaChangeRequest;
-import org.apache.flink.cdc.runtime.operators.schema.event.ApplyEvolvedSchemaChangeResponse;
-import org.apache.flink.cdc.runtime.operators.schema.event.ApplyOriginalSchemaChangeRequest;
-import org.apache.flink.cdc.runtime.operators.schema.event.ApplyOriginalSchemaChangeResponse;
import org.apache.flink.cdc.runtime.operators.schema.event.FlushSuccessEvent;
import org.apache.flink.cdc.runtime.operators.schema.event.GetEvolvedSchemaRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.GetEvolvedSchemaResponse;
@@ -151,18 +149,23 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
@Override
public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event)
throws Exception {
- if (event instanceof FlushSuccessEvent) {
- FlushSuccessEvent flushSuccessEvent = (FlushSuccessEvent) event;
- LOG.info(
- "Sink subtask {} succeed flushing for table {}.",
- flushSuccessEvent.getSubtask(),
- flushSuccessEvent.getTableId().toString());
- requestHandler.flushSuccess(
- flushSuccessEvent.getTableId(), flushSuccessEvent.getSubtask());
- } else if (event instanceof SinkWriterRegisterEvent) {
- requestHandler.registerSinkWriter(((SinkWriterRegisterEvent) event).getSubtask());
- } else {
- throw new FlinkException("Unrecognized Operator Event: " + event);
+ try {
+ if (event instanceof FlushSuccessEvent) {
+ FlushSuccessEvent flushSuccessEvent = (FlushSuccessEvent) event;
+ LOG.info(
+ "Sink subtask {} succeed flushing for table {}.",
+ flushSuccessEvent.getSubtask(),
+ flushSuccessEvent.getTableId().toString());
+ requestHandler.flushSuccess(
+ flushSuccessEvent.getTableId(), flushSuccessEvent.getSubtask());
+ } else if (event instanceof SinkWriterRegisterEvent) {
+ requestHandler.registerSinkWriter(((SinkWriterRegisterEvent) event).getSubtask());
+ } else {
+ throw new FlinkException("Unrecognized Operator Event: " + event);
+ }
+ } catch (Throwable t) {
+ context.failJob(t);
+ throw t;
}
}
@@ -180,6 +183,9 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
// Serialize SchemaDerivation mapping
SchemaDerivation.serializeDerivationMapping(schemaDerivation, out);
resultFuture.complete(baos.toByteArray());
+ } catch (Throwable t) {
+ context.failJob(t);
+ throw t;
}
}
@@ -191,33 +197,29 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
@Override
public CompletableFuture<CoordinationResponse> handleCoordinationRequest(
CoordinationRequest request) {
- if (request instanceof SchemaChangeRequest) {
- SchemaChangeRequest schemaChangeRequest = (SchemaChangeRequest) request;
- return requestHandler.handleSchemaChangeRequest(schemaChangeRequest);
- } else if (request instanceof ReleaseUpstreamRequest) {
- return requestHandler.handleReleaseUpstreamRequest();
- } else if (request instanceof GetEvolvedSchemaRequest) {
- return CompletableFuture.completedFuture(
- wrap(handleGetEvolvedSchemaRequest(((GetEvolvedSchemaRequest) request))));
- } else if (request instanceof GetOriginalSchemaRequest) {
- return CompletableFuture.completedFuture(
- wrap(handleGetOriginalSchemaRequest((GetOriginalSchemaRequest) request)));
- } else if (request instanceof ApplyOriginalSchemaChangeRequest) {
- return CompletableFuture.completedFuture(
- wrap(
- handleApplyOriginalSchemaChangeRequest(
- (ApplyOriginalSchemaChangeRequest) request)));
- } else if (request instanceof ApplyEvolvedSchemaChangeRequest) {
- return CompletableFuture.completedFuture(
- wrap(
- handleApplyEvolvedSchemaChangeRequest(
- (ApplyEvolvedSchemaChangeRequest) request)));
- } else if (request instanceof SchemaChangeResultRequest) {
- return requestHandler.getSchemaChangeResult();
- } else if (request instanceof RefreshPendingListsRequest) {
- return requestHandler.refreshPendingLists();
- } else {
- throw new IllegalArgumentException("Unrecognized CoordinationRequest type: " + request);
+ try {
+ if (request instanceof SchemaChangeRequest) {
+ SchemaChangeRequest schemaChangeRequest = (SchemaChangeRequest) request;
+ return requestHandler.handleSchemaChangeRequest(schemaChangeRequest);
+ } else if (request instanceof ReleaseUpstreamRequest) {
+ return requestHandler.handleReleaseUpstreamRequest();
+ } else if (request instanceof GetEvolvedSchemaRequest) {
+ return CompletableFuture.completedFuture(
+ wrap(handleGetEvolvedSchemaRequest(((GetEvolvedSchemaRequest) request))));
+ } else if (request instanceof GetOriginalSchemaRequest) {
+ return CompletableFuture.completedFuture(
+ wrap(handleGetOriginalSchemaRequest((GetOriginalSchemaRequest) request)));
+ } else if (request instanceof SchemaChangeResultRequest) {
+ return requestHandler.getSchemaChangeResult();
+ } else if (request instanceof RefreshPendingListsRequest) {
+ return requestHandler.refreshPendingLists();
+ } else {
+ throw new IllegalArgumentException(
+ "Unrecognized CoordinationRequest type: " + request);
+ }
+ } catch (Throwable t) {
+ context.failJob(t);
+ throw t;
}
}
@@ -275,6 +277,9 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
throw new IOException(
"Unrecognized serialization version " + schemaManagerSerializerVersion);
}
+ } catch (Throwable t) {
+ context.failJob(t);
+ throw t;
}
}
@@ -342,18 +347,15 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
}
}
- private ApplyOriginalSchemaChangeResponse handleApplyOriginalSchemaChangeRequest(
- ApplyOriginalSchemaChangeRequest applyOriginalSchemaChangeRequest) {
- schemaManager.applyOriginalSchemaChange(
- applyOriginalSchemaChangeRequest.getSchemaChangeEvent());
- return new ApplyOriginalSchemaChangeResponse();
+ // --------------------Only visible for test -----------------
+
+ @VisibleForTesting
+ public void handleApplyOriginalSchemaChangeEvent(SchemaChangeEvent schemaChangeEvent) {
+ schemaManager.applyOriginalSchemaChange(schemaChangeEvent);
}
- private ApplyEvolvedSchemaChangeResponse handleApplyEvolvedSchemaChangeRequest(
- ApplyEvolvedSchemaChangeRequest applyEvolvedSchemaChangeRequest) {
- applyEvolvedSchemaChangeRequest
- .getSchemaChangeEvent()
- .forEach(schemaManager::applyEvolvedSchemaChange);
- return new ApplyEvolvedSchemaChangeResponse();
+ @VisibleForTesting
+ public void handleApplyEvolvedSchemaChangeRequest(SchemaChangeEvent schemaChangeEvent) {
+ schemaManager.applyEvolvedSchemaChange(schemaChangeEvent);
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
index da88753e5..60280ce48 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
@@ -17,7 +17,6 @@
package org.apache.flink.cdc.runtime.operators.schema.coordinator;
-import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.event.AddColumnEvent;
import org.apache.flink.cdc.common.event.AlterColumnTypeEvent;
@@ -27,7 +26,7 @@ import org.apache.flink.cdc.common.event.RenameColumnEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEventType;
import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.exceptions.SchemaEvolveException;
+import org.apache.flink.cdc.common.exceptions.UnsupportedSchemaChangeEventException;
import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.schema.Column;
import org.apache.flink.cdc.common.schema.Schema;
@@ -87,7 +86,6 @@ public class SchemaRegistryRequestHandler implements Closeable {
private final List<PendingSchemaChange> pendingSchemaChanges;
private final List<SchemaChangeEvent> finishedSchemaChanges;
- private final List<Tuple2<SchemaChangeEvent, Throwable>> failedSchemaChanges;
private final List<SchemaChangeEvent> ignoredSchemaChanges;
/** Sink writers which have sent flush success events for the request. */
@@ -95,6 +93,8 @@ public class SchemaRegistryRequestHandler implements Closeable {
/** Status of the execution of current schema change request. */
private volatile boolean isSchemaChangeApplying;
+ /** Actual exception if failed to apply schema change. */
+ private volatile Throwable schemaChangeException;
/** Executor service to execute schema change. */
private final ExecutorService schemaChangeThreadPool;
@@ -110,7 +110,6 @@ public class SchemaRegistryRequestHandler implements Closeable {
this.flushedSinkWriters = new HashSet<>();
this.pendingSchemaChanges = new LinkedList<>();
this.finishedSchemaChanges = new LinkedList<>();
- this.failedSchemaChanges = new LinkedList<>();
this.ignoredSchemaChanges = new LinkedList<>();
this.schemaManager = schemaManager;
this.schemaDerivation = schemaDerivation;
@@ -128,8 +127,8 @@ public class SchemaRegistryRequestHandler implements Closeable {
private void applySchemaChange(
TableId tableId, List<SchemaChangeEvent> derivedSchemaChangeEvents) {
isSchemaChangeApplying = true;
+ schemaChangeException = null;
finishedSchemaChanges.clear();
- failedSchemaChanges.clear();
ignoredSchemaChanges.clear();
for (SchemaChangeEvent changeEvent : derivedSchemaChangeEvents) {
@@ -146,17 +145,27 @@ public class SchemaRegistryRequestHandler implements Closeable {
try {
metadataApplier.applySchemaChange(changeEvent);
LOG.debug("Applied schema change {} to table {}.", changeEvent, tableId);
+ schemaManager.applyEvolvedSchemaChange(changeEvent);
finishedSchemaChanges.add(changeEvent);
- } catch (SchemaEvolveException e) {
+ } catch (Throwable t) {
LOG.error(
"Failed to apply schema change {} to table {}. Caused by: {}",
changeEvent,
tableId,
- e);
- failedSchemaChanges.add(Tuple2.of(changeEvent, e));
+ t);
+ if (!shouldIgnoreException(t)) {
+ schemaChangeException = t;
+ break;
+ } else {
+ LOG.warn(
+ "Failed to apply event {}, but keeps running in tolerant mode. Caused by: {}",
+ changeEvent,
+ t);
+ }
}
}
}
+
PendingSchemaChange waitFlushSuccess = pendingSchemaChanges.get(0);
if (RECEIVED_RELEASE_REQUEST.equals(waitFlushSuccess.getStatus())) {
startNextSchemaChangeRequest();
@@ -241,6 +250,10 @@ public class SchemaRegistryRequestHandler implements Closeable {
() -> applySchemaChange(tableId, waitFlushSuccess.derivedSchemaChangeEvents));
Thread.sleep(1000);
+ if (schemaChangeException != null) {
+ throw new RuntimeException("Failed to apply schema change.", schemaChangeException);
+ }
+
if (isSchemaChangeApplying) {
waitFlushSuccess
.getResponseFuture()
@@ -248,12 +261,7 @@ public class SchemaRegistryRequestHandler implements Closeable {
} else {
waitFlushSuccess
.getResponseFuture()
- .complete(
- wrap(
- new ReleaseUpstreamResponse(
- finishedSchemaChanges,
- failedSchemaChanges,
- ignoredSchemaChanges)));
+ .complete(wrap(new ReleaseUpstreamResponse(finishedSchemaChanges)));
}
}
}
@@ -292,16 +300,15 @@ public class SchemaRegistryRequestHandler implements Closeable {
}
public CompletableFuture<CoordinationResponse> getSchemaChangeResult() {
+ if (schemaChangeException != null) {
+ throw new RuntimeException("Failed to apply schema change.", schemaChangeException);
+ }
+
if (isSchemaChangeApplying) {
return CompletableFuture.supplyAsync(() -> wrap(new SchemaChangeProcessingResponse()));
} else {
return CompletableFuture.supplyAsync(
- () ->
- wrap(
- new ReleaseUpstreamResponse(
- finishedSchemaChanges,
- failedSchemaChanges,
- ignoredSchemaChanges)));
+ () -> wrap(new ReleaseUpstreamResponse(finishedSchemaChanges)));
}
}
@@ -420,6 +427,15 @@ public class SchemaRegistryRequestHandler implements Closeable {
}
}
+ private boolean shouldIgnoreException(Throwable throwable) {
+
+ // In IGNORE mode, will never try to apply schema change events
+ // In EVOLVE and and LENIENT mode, such failure will not be tolerated
+ // In EXCEPTION mode, an exception will be thrown once captured
+ return (throwable instanceof UnsupportedSchemaChangeEventException)
+ && (schemaChangeBehavior == SchemaChangeBehavior.TRY_EVOLVE);
+ }
+
private static class PendingSchemaChange {
private final SchemaChangeRequest changeRequest;
private List<SchemaChangeEvent> derivedSchemaChangeEvents;
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyEvolvedSchemaChangeRequest.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyEvolvedSchemaChangeRequest.java
deleted file mode 100644
index f6798af51..000000000
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyEvolvedSchemaChangeRequest.java
+++ /dev/null
@@ -1,73 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.runtime.operators.schema.event;
-
-import org.apache.flink.cdc.common.event.SchemaChangeEvent;
-import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.runtime.operators.schema.SchemaOperator;
-import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry;
-import org.apache.flink.runtime.operators.coordination.CoordinationRequest;
-
-import java.util.List;
-import java.util.Objects;
-
-/**
- * The request from {@link SchemaOperator} to {@link SchemaRegistry} to request apply evolved schema
- * changes.
- */
-public class ApplyEvolvedSchemaChangeRequest implements CoordinationRequest {
-
- private static final long serialVersionUID = 1L;
-
- /** The sender of the request. */
- private final TableId tableId;
- /** The schema changes. */
- private final List<SchemaChangeEvent> schemaChangeEvent;
-
- public ApplyEvolvedSchemaChangeRequest(
- TableId tableId, List<SchemaChangeEvent> schemaChangeEvent) {
- this.tableId = tableId;
- this.schemaChangeEvent = schemaChangeEvent;
- }
-
- public TableId getTableId() {
- return tableId;
- }
-
- public List<SchemaChangeEvent> getSchemaChangeEvent() {
- return schemaChangeEvent;
- }
-
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (!(o instanceof ApplyEvolvedSchemaChangeRequest)) {
- return false;
- }
- ApplyEvolvedSchemaChangeRequest that = (ApplyEvolvedSchemaChangeRequest) o;
- return Objects.equals(tableId, that.tableId)
- && Objects.equals(schemaChangeEvent, that.schemaChangeEvent);
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(tableId, schemaChangeEvent);
- }
-}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyEvolvedSchemaChangeResponse.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyEvolvedSchemaChangeResponse.java
deleted file mode 100644
index 787adfc5e..000000000
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyEvolvedSchemaChangeResponse.java
+++ /dev/null
@@ -1,32 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.runtime.operators.schema.event;
-
-import org.apache.flink.cdc.runtime.operators.schema.SchemaOperator;
-import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry;
-import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
-
-/**
- * The response from {@link SchemaRegistry} to {@link SchemaOperator} to request apply original
- * schema changes, the evolved schema changes come from original schema changes with different
- * schema evolution strategy.
- */
-public class ApplyEvolvedSchemaChangeResponse implements CoordinationResponse {
-
- private static final long serialVersionUID = 1L;
-}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyOriginalSchemaChangeRequest.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyOriginalSchemaChangeRequest.java
deleted file mode 100644
index d4c5d7fee..000000000
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyOriginalSchemaChangeRequest.java
+++ /dev/null
@@ -1,71 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.runtime.operators.schema.event;
-
-import org.apache.flink.cdc.common.event.SchemaChangeEvent;
-import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.runtime.operators.schema.SchemaOperator;
-import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry;
-import org.apache.flink.runtime.operators.coordination.CoordinationRequest;
-
-import java.util.Objects;
-
-/**
- * The request from {@link SchemaOperator} to {@link SchemaRegistry} to request apply original
- * schema changes.
- */
-public class ApplyOriginalSchemaChangeRequest implements CoordinationRequest {
-
- private static final long serialVersionUID = 1L;
-
- /** The sender of the request. */
- private final TableId tableId;
- /** The schema changes. */
- private final SchemaChangeEvent schemaChangeEvent;
-
- public ApplyOriginalSchemaChangeRequest(TableId tableId, SchemaChangeEvent schemaChangeEvent) {
- this.tableId = tableId;
- this.schemaChangeEvent = schemaChangeEvent;
- }
-
- public TableId getTableId() {
- return tableId;
- }
-
- public SchemaChangeEvent getSchemaChangeEvent() {
- return schemaChangeEvent;
- }
-
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (!(o instanceof ApplyOriginalSchemaChangeRequest)) {
- return false;
- }
- ApplyOriginalSchemaChangeRequest that = (ApplyOriginalSchemaChangeRequest) o;
- return Objects.equals(tableId, that.tableId)
- && Objects.equals(schemaChangeEvent, that.schemaChangeEvent);
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(tableId, schemaChangeEvent);
- }
-}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyOriginalSchemaChangeResponse.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyOriginalSchemaChangeResponse.java
deleted file mode 100644
index 0a92e9656..000000000
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ApplyOriginalSchemaChangeResponse.java
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.runtime.operators.schema.event;
-
-import org.apache.flink.cdc.runtime.operators.schema.SchemaOperator;
-import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry;
-import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
-
-/**
- * The response from {@link SchemaRegistry} to {@link SchemaOperator} to request apply original
- * schema changes.
- */
-public class ApplyOriginalSchemaChangeResponse implements CoordinationResponse {
-
- private static final long serialVersionUID = 1L;
-}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ReleaseUpstreamResponse.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ReleaseUpstreamResponse.java
index f577f1120..bea880962 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ReleaseUpstreamResponse.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ReleaseUpstreamResponse.java
@@ -17,7 +17,6 @@
package org.apache.flink.cdc.runtime.operators.schema.event;
-import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.cdc.common.event.SchemaChangeEvent;
import org.apache.flink.cdc.runtime.operators.schema.SchemaOperator;
import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry;
@@ -25,7 +24,6 @@ import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
import java.util.List;
import java.util.Objects;
-import java.util.stream.Collectors;
/**
* The response for {@link ReleaseUpstreamRequest} from {@link SchemaRegistry} to {@link
@@ -41,50 +39,19 @@ public class ReleaseUpstreamResponse implements CoordinationResponse {
*/
private final List<SchemaChangeEvent> finishedSchemaChangeEvents;
- private final List<Tuple2<SchemaChangeEvent, Throwable>> failedSchemaChangeEvents;
-
- private final List<SchemaChangeEvent> ignoredSchemaChangeEvents;
-
- public ReleaseUpstreamResponse(
- List<SchemaChangeEvent> finishedSchemaChangeEvents,
- List<Tuple2<SchemaChangeEvent, Throwable>> failedSchemaChangeEvents,
- List<SchemaChangeEvent> ignoredSchemaChangeEvents) {
+ public ReleaseUpstreamResponse(List<SchemaChangeEvent> finishedSchemaChangeEvents) {
this.finishedSchemaChangeEvents = finishedSchemaChangeEvents;
- this.failedSchemaChangeEvents = failedSchemaChangeEvents;
- this.ignoredSchemaChangeEvents = ignoredSchemaChangeEvents;
}
public List<SchemaChangeEvent> getFinishedSchemaChangeEvents() {
return finishedSchemaChangeEvents;
}
- public List<Tuple2<SchemaChangeEvent, Throwable>> getFailedSchemaChangeEvents() {
- return failedSchemaChangeEvents;
- }
-
- public List<SchemaChangeEvent> getIgnoredSchemaChangeEvents() {
- return ignoredSchemaChangeEvents;
- }
-
- public String getPrintableFailedSchemaChangeEvents() {
- return failedSchemaChangeEvents.stream()
- .map(e -> "Failed to apply " + e.f0 + ". Caused by: " + e.f1)
- .collect(Collectors.joining("\n"));
- }
-
- public boolean hasException() {
- return !failedSchemaChangeEvents.isEmpty();
- }
-
@Override
public String toString() {
return "ReleaseUpstreamResponse{"
+ "finishedSchemaChangeEvents="
+ finishedSchemaChangeEvents
- + ", failedSchemaChangeEvents="
- + failedSchemaChangeEvents
- + ", ignoredSchemaChangeEvents="
- + ignoredSchemaChangeEvents
+ '}';
}
@@ -97,14 +64,11 @@ public class ReleaseUpstreamResponse implements CoordinationResponse {
return false;
}
ReleaseUpstreamResponse that = (ReleaseUpstreamResponse) object;
- return Objects.equals(finishedSchemaChangeEvents, that.finishedSchemaChangeEvents)
- && Objects.equals(failedSchemaChangeEvents, that.failedSchemaChangeEvents)
- && Objects.equals(ignoredSchemaChangeEvents, that.ignoredSchemaChangeEvents);
+ return Objects.equals(finishedSchemaChangeEvents, that.finishedSchemaChangeEvents);
}
@Override
public int hashCode() {
- return Objects.hash(
- finishedSchemaChangeEvents, failedSchemaChangeEvents, ignoredSchemaChangeEvents);
+ return Objects.hash(finishedSchemaChangeEvents);
}
}
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/SchemaEvolveTest.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/SchemaEvolveTest.java
index bf61ad260..d3e0de8d0 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/SchemaEvolveTest.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/SchemaEvolveTest.java
@@ -1036,6 +1036,8 @@ public class SchemaEvolveTest {
Column.physicalColumn(
"height", DOUBLE, "Height data")))));
Assertions.assertThatThrownBy(() -> processEvent(schemaOperator, addColumnEvents))
+ .cause()
+ .cause()
.isExactlyInstanceOf(RuntimeException.class)
.hasMessageContaining("Failed to apply schema change");
harness.close();
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
index b5262d2d5..354304599 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
@@ -25,8 +25,6 @@ import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry;
-import org.apache.flink.cdc.runtime.operators.schema.event.ApplyEvolvedSchemaChangeRequest;
-import org.apache.flink.cdc.runtime.operators.schema.event.ApplyOriginalSchemaChangeRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.FlushSuccessEvent;
import org.apache.flink.cdc.runtime.operators.schema.event.GetEvolvedSchemaRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.GetEvolvedSchemaResponse;
@@ -56,7 +54,6 @@ import org.apache.flink.util.SerializedValue;
import java.io.IOException;
import java.time.Duration;
import java.util.ArrayList;
-import java.util.Collections;
import java.util.LinkedList;
import java.util.Set;
@@ -166,14 +163,10 @@ public class EventOperatorTestHarness<OP extends AbstractStreamOperator<E>, E ex
}
public void registerTableSchema(TableId tableId, Schema schema) {
- schemaRegistry.handleCoordinationRequest(
- new ApplyOriginalSchemaChangeRequest(
- tableId, new CreateTableEvent(tableId, schema)));
+ schemaRegistry.handleApplyOriginalSchemaChangeEvent(new CreateTableEvent(tableId, schema));
schemaRegistry.handleCoordinationRequest(
new SchemaChangeRequest(tableId, new CreateTableEvent(tableId, schema)));
- schemaRegistry.handleCoordinationRequest(
- new ApplyEvolvedSchemaChangeRequest(
- tableId, Collections.singletonList(new CreateTableEvent(tableId, schema))));
+ schemaRegistry.handleApplyEvolvedSchemaChangeRequest(new CreateTableEvent(tableId, schema));
}
public Schema getLatestOriginalSchema(TableId tableId) throws Exception {
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/schema/CollectingMetadataApplier.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/schema/CollectingMetadataApplier.java
index 9398e1f1e..9384fa757 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/schema/CollectingMetadataApplier.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/schema/CollectingMetadataApplier.java
@@ -21,6 +21,7 @@ import org.apache.flink.cdc.common.event.SchemaChangeEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEventType;
import org.apache.flink.cdc.common.event.SchemaChangeEventTypeFamily;
import org.apache.flink.cdc.common.exceptions.SchemaEvolveException;
+import org.apache.flink.cdc.common.exceptions.UnsupportedSchemaChangeEventException;
import org.apache.flink.cdc.common.sink.MetadataApplier;
import java.time.Duration;
@@ -83,8 +84,7 @@ public class CollectingMetadataApplier implements MetadataApplier {
try {
Thread.sleep(duration.toMillis());
if (errorsOnEventTypes.contains(schemaChangeEvent.getType())) {
- throw new SchemaEvolveException(
- schemaChangeEvent, "Dummy metadata apply exception for test.", null);
+ throw new UnsupportedSchemaChangeEventException(schemaChangeEvent);
}
} catch (InterruptedException ignore) {
// Ignores sleep interruption
(flink-cdc) 12/12: [hotfix] Run schema coordinator logic asynchronously to avoid blocking the main thread
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 1ebefd6d2e59712a9fad72686e4cbcda0ddd2d96
Author: yuxiqian <34...@users.noreply.github.com>
AuthorDate: Tue Aug 27 10:21:14 2024 +0800
[hotfix] Run schema coordinator logic asynchronously to avoid blocking the main thread
---
.../tests/SchemaRegistryMigrationMock.java | 2 +
.../schema/coordinator/SchemaRegistry.java | 232 ++++++++++++++-------
.../schema/coordinator/SchemaRegistryProvider.java | 54 ++++-
.../coordinator/SchemaRegistryRequestHandler.java | 40 ++--
.../runtime/operators/schema/SchemaEvolveTest.java | 14 +-
.../operators/EventOperatorTestHarness.java | 32 ++-
.../MockedOperatorCoordinatorContext.java | 44 ++++
7 files changed, 314 insertions(+), 104 deletions(-)
diff --git a/flink-cdc-migration-tests/flink-cdc-release-snapshot/src/main/java/org/apache/flink/cdc/migration/tests/SchemaRegistryMigrationMock.java b/flink-cdc-migration-tests/flink-cdc-release-snapshot/src/main/java/org/apache/flink/cdc/migration/tests/SchemaRegistryMigrationMock.java
index bb19223e6..34cd1ea03 100644
--- a/flink-cdc-migration-tests/flink-cdc-release-snapshot/src/main/java/org/apache/flink/cdc/migration/tests/SchemaRegistryMigrationMock.java
+++ b/flink-cdc-migration-tests/flink-cdc-release-snapshot/src/main/java/org/apache/flink/cdc/migration/tests/SchemaRegistryMigrationMock.java
@@ -40,6 +40,7 @@ import java.util.Set;
import java.util.SortedMap;
import java.util.TreeMap;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
import java.util.stream.Collectors;
/** Dummy classes for migration test. Called via reflection. */
@@ -69,6 +70,7 @@ public class SchemaRegistryMigrationMock implements MigrationMockBase {
return new SchemaRegistry(
"Dummy Name",
null,
+ Executors.newFixedThreadPool(1),
new MetadataApplier() {
@Override
public boolean acceptsSchemaEvolutionType(
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
index 617daed92..64d4fa1bc 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
@@ -37,7 +37,9 @@ import org.apache.flink.runtime.operators.coordination.CoordinationRequestHandle
import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.util.ExceptionUtils;
import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.function.ThrowingRunnable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -55,6 +57,7 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
import static org.apache.flink.cdc.runtime.operators.schema.event.CoordinationResponseUtils.wrap;
@@ -83,6 +86,9 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
/** The name of the operator this SchemaOperatorCoordinator is associated with. */
private final String operatorName;
+ /** A single-thread executor to handle async execution of the coordinator. */
+ private final ExecutorService coordinatorExecutor;
+
/**
* Tracks the subtask failed reason to throw a more meaningful exception in {@link
* #subtaskReset}.
@@ -113,18 +119,27 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
public SchemaRegistry(
String operatorName,
OperatorCoordinator.Context context,
+ ExecutorService executorService,
MetadataApplier metadataApplier,
List<RouteRule> routes) {
- this(operatorName, context, metadataApplier, routes, SchemaChangeBehavior.EVOLVE);
+ this(
+ operatorName,
+ context,
+ executorService,
+ metadataApplier,
+ routes,
+ SchemaChangeBehavior.LENIENT);
}
public SchemaRegistry(
String operatorName,
OperatorCoordinator.Context context,
+ ExecutorService coordinatorExecutor,
MetadataApplier metadataApplier,
List<RouteRule> routes,
SchemaChangeBehavior schemaChangeBehavior) {
this.context = context;
+ this.coordinatorExecutor = coordinatorExecutor;
this.operatorName = operatorName;
this.failedReasons = new HashMap<>();
this.metadataApplier = metadataApplier;
@@ -133,7 +148,11 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
this.schemaDerivation = new SchemaDerivation(schemaManager, routes, new HashMap<>());
this.requestHandler =
new SchemaRegistryRequestHandler(
- metadataApplier, schemaManager, schemaDerivation, schemaChangeBehavior);
+ metadataApplier,
+ schemaManager,
+ schemaDerivation,
+ schemaChangeBehavior,
+ context);
this.schemaChangeBehavior = schemaChangeBehavior;
}
@@ -153,48 +172,87 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
}
@Override
- public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event)
- throws Exception {
- try {
- if (event instanceof FlushSuccessEvent) {
- FlushSuccessEvent flushSuccessEvent = (FlushSuccessEvent) event;
- LOG.info(
- "Sink subtask {} succeed flushing for table {}.",
- flushSuccessEvent.getSubtask(),
- flushSuccessEvent.getTableId().toString());
- requestHandler.flushSuccess(
- flushSuccessEvent.getTableId(),
- flushSuccessEvent.getSubtask(),
- currentParallelism);
- } else if (event instanceof SinkWriterRegisterEvent) {
- requestHandler.registerSinkWriter(((SinkWriterRegisterEvent) event).getSubtask());
- } else {
- throw new FlinkException("Unrecognized Operator Event: " + event);
- }
- } catch (Throwable t) {
- context.failJob(t);
- throw t;
- }
+ public void handleEventFromOperator(int subtask, int attemptNumber, OperatorEvent event) {
+ runInEventLoop(
+ () -> {
+ try {
+ if (event instanceof FlushSuccessEvent) {
+ FlushSuccessEvent flushSuccessEvent = (FlushSuccessEvent) event;
+ LOG.info(
+ "Sink subtask {} succeed flushing for table {}.",
+ flushSuccessEvent.getSubtask(),
+ flushSuccessEvent.getTableId().toString());
+ requestHandler.flushSuccess(
+ flushSuccessEvent.getTableId(),
+ flushSuccessEvent.getSubtask(),
+ currentParallelism);
+ } else if (event instanceof SinkWriterRegisterEvent) {
+ requestHandler.registerSinkWriter(
+ ((SinkWriterRegisterEvent) event).getSubtask());
+ } else {
+ throw new FlinkException("Unrecognized Operator Event: " + event);
+ }
+ } catch (Throwable t) {
+ context.failJob(t);
+ throw t;
+ }
+ },
+ "handling event %s from subTask %d",
+ event,
+ subtask);
}
@Override
- public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture)
- throws Exception {
- try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
- DataOutputStream out = new DataOutputStream(baos)) {
- // Serialize SchemaManager
- int schemaManagerSerializerVersion = SchemaManager.SERIALIZER.getVersion();
- out.writeInt(schemaManagerSerializerVersion);
- byte[] serializedSchemaManager = SchemaManager.SERIALIZER.serialize(schemaManager);
- out.writeInt(serializedSchemaManager.length);
- out.write(serializedSchemaManager);
- // Serialize SchemaDerivation mapping
- SchemaDerivation.serializeDerivationMapping(schemaDerivation, out);
- resultFuture.complete(baos.toByteArray());
- } catch (Throwable t) {
- context.failJob(t);
- throw t;
- }
+ public void checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> resultFuture) {
+ // we generate checkpoint in an async thread to not block the JobManager's main thread, the
+ // coordinator state might be large if there are many schema changes and monitor many
+ // tables.
+ runInEventLoop(
+ () -> {
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputStream out = new DataOutputStream(baos)) {
+ // Serialize SchemaManager
+ int schemaManagerSerializerVersion = SchemaManager.SERIALIZER.getVersion();
+ out.writeInt(schemaManagerSerializerVersion);
+ byte[] serializedSchemaManager =
+ SchemaManager.SERIALIZER.serialize(schemaManager);
+ out.writeInt(serializedSchemaManager.length);
+ out.write(serializedSchemaManager);
+ // Serialize SchemaDerivation mapping
+ SchemaDerivation.serializeDerivationMapping(schemaDerivation, out);
+ resultFuture.complete(baos.toByteArray());
+ } catch (Throwable t) {
+ context.failJob(t);
+ throw t;
+ }
+ },
+ "taking checkpoint %d",
+ checkpointId);
+ }
+
+ private void runInEventLoop(
+ final ThrowingRunnable<Throwable> action,
+ final String actionName,
+ final Object... actionNameFormatParameters) {
+ coordinatorExecutor.execute(
+ () -> {
+ try {
+ action.run();
+ } catch (Throwable t) {
+ // if we have a JVM critical error, promote it immediately, there is a good
+ // chance the logging or job failing will not succeed anymore
+ ExceptionUtils.rethrowIfFatalErrorOrOOM(t);
+
+ final String actionString =
+ String.format(actionName, actionNameFormatParameters);
+ LOG.error(
+ "Uncaught exception in the SchemaEvolutionCoordinator for {} while {}. Triggering job failover.",
+ operatorName,
+ actionString,
+ t);
+ context.failJob(t);
+ }
+ });
}
@Override
@@ -205,26 +263,34 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
@Override
public CompletableFuture<CoordinationResponse> handleCoordinationRequest(
CoordinationRequest request) {
- try {
- if (request instanceof SchemaChangeRequest) {
- SchemaChangeRequest schemaChangeRequest = (SchemaChangeRequest) request;
- return requestHandler.handleSchemaChangeRequest(schemaChangeRequest);
- } else if (request instanceof SchemaChangeResultRequest) {
- return requestHandler.getSchemaChangeResult();
- } else if (request instanceof GetEvolvedSchemaRequest) {
- return CompletableFuture.completedFuture(
- wrap(handleGetEvolvedSchemaRequest(((GetEvolvedSchemaRequest) request))));
- } else if (request instanceof GetOriginalSchemaRequest) {
- return CompletableFuture.completedFuture(
- wrap(handleGetOriginalSchemaRequest((GetOriginalSchemaRequest) request)));
- } else {
- throw new IllegalArgumentException(
- "Unrecognized CoordinationRequest type: " + request);
- }
- } catch (Throwable t) {
- context.failJob(t);
- throw t;
- }
+ CompletableFuture<CoordinationResponse> responseFuture = new CompletableFuture<>();
+ runInEventLoop(
+ () -> {
+ try {
+ if (request instanceof SchemaChangeRequest) {
+ SchemaChangeRequest schemaChangeRequest = (SchemaChangeRequest) request;
+ requestHandler.handleSchemaChangeRequest(
+ schemaChangeRequest, responseFuture);
+ } else if (request instanceof SchemaChangeResultRequest) {
+ requestHandler.getSchemaChangeResult(responseFuture);
+ } else if (request instanceof GetEvolvedSchemaRequest) {
+ handleGetEvolvedSchemaRequest(
+ ((GetEvolvedSchemaRequest) request), responseFuture);
+ } else if (request instanceof GetOriginalSchemaRequest) {
+ handleGetOriginalSchemaRequest(
+ (GetOriginalSchemaRequest) request, responseFuture);
+ } else {
+ throw new IllegalArgumentException(
+ "Unrecognized CoordinationRequest type: " + request);
+ }
+ } catch (Throwable t) {
+ context.failJob(t);
+ throw t;
+ }
+ },
+ "handling coordination request %s",
+ request);
+ return responseFuture;
}
@Override
@@ -253,7 +319,8 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
metadataApplier,
schemaManager,
schemaDerivation,
- schemaManager.getBehavior());
+ schemaManager.getBehavior(),
+ context);
break;
}
case 1:
@@ -274,7 +341,8 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
metadataApplier,
schemaManager,
schemaDerivation,
- schemaChangeBehavior);
+ schemaChangeBehavior,
+ context);
break;
}
default:
@@ -307,46 +375,56 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
// do nothing
}
- private GetEvolvedSchemaResponse handleGetEvolvedSchemaRequest(
- GetEvolvedSchemaRequest getEvolvedSchemaRequest) {
+ private void handleGetEvolvedSchemaRequest(
+ GetEvolvedSchemaRequest getEvolvedSchemaRequest,
+ CompletableFuture<CoordinationResponse> response) {
LOG.info("Handling evolved schema request: {}", getEvolvedSchemaRequest);
int schemaVersion = getEvolvedSchemaRequest.getSchemaVersion();
TableId tableId = getEvolvedSchemaRequest.getTableId();
if (schemaVersion == GetEvolvedSchemaRequest.LATEST_SCHEMA_VERSION) {
- return new GetEvolvedSchemaResponse(
- schemaManager.getLatestEvolvedSchema(tableId).orElse(null));
+ response.complete(
+ wrap(
+ new GetEvolvedSchemaResponse(
+ schemaManager.getLatestEvolvedSchema(tableId).orElse(null))));
} else {
try {
- return new GetEvolvedSchemaResponse(
- schemaManager.getEvolvedSchema(tableId, schemaVersion));
+ response.complete(
+ wrap(
+ new GetEvolvedSchemaResponse(
+ schemaManager.getEvolvedSchema(tableId, schemaVersion))));
} catch (IllegalArgumentException iae) {
LOG.warn(
"Some client is requesting an non-existed evolved schema for table {} with version {}",
tableId,
schemaVersion);
- return new GetEvolvedSchemaResponse(null);
+ response.complete(wrap(new GetEvolvedSchemaResponse(null)));
}
}
}
- private GetOriginalSchemaResponse handleGetOriginalSchemaRequest(
- GetOriginalSchemaRequest getOriginalSchemaRequest) {
+ private void handleGetOriginalSchemaRequest(
+ GetOriginalSchemaRequest getOriginalSchemaRequest,
+ CompletableFuture<CoordinationResponse> response) {
LOG.info("Handling original schema request: {}", getOriginalSchemaRequest);
int schemaVersion = getOriginalSchemaRequest.getSchemaVersion();
TableId tableId = getOriginalSchemaRequest.getTableId();
if (schemaVersion == GetOriginalSchemaRequest.LATEST_SCHEMA_VERSION) {
- return new GetOriginalSchemaResponse(
- schemaManager.getLatestOriginalSchema(tableId).orElse(null));
+ response.complete(
+ wrap(
+ new GetOriginalSchemaResponse(
+ schemaManager.getLatestOriginalSchema(tableId).orElse(null))));
} else {
try {
- return new GetOriginalSchemaResponse(
- schemaManager.getOriginalSchema(tableId, schemaVersion));
+ response.complete(
+ wrap(
+ new GetOriginalSchemaResponse(
+ schemaManager.getOriginalSchema(tableId, schemaVersion))));
} catch (IllegalArgumentException iae) {
LOG.warn(
"Some client is requesting an non-existed original schema for table {} with version {}",
tableId,
schemaVersion);
- return new GetOriginalSchemaResponse(null);
+ response.complete(wrap(new GetOriginalSchemaResponse(null)));
}
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryProvider.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryProvider.java
index dd7f2dc36..bc261e40f 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryProvider.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryProvider.java
@@ -23,8 +23,12 @@ import org.apache.flink.cdc.common.route.RouteRule;
import org.apache.flink.cdc.common.sink.MetadataApplier;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.util.FatalExitExceptionHandler;
import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
/** Provider of {@link SchemaRegistry}. */
@Internal
@@ -57,7 +61,55 @@ public class SchemaRegistryProvider implements OperatorCoordinator.Provider {
@Override
public OperatorCoordinator create(OperatorCoordinator.Context context) throws Exception {
+ CoordinatorExecutorThreadFactory coordinatorThreadFactory =
+ new CoordinatorExecutorThreadFactory(
+ "schema-evolution-coordinator", context.getUserCodeClassloader());
+ ExecutorService coordinatorExecutor =
+ Executors.newSingleThreadExecutor(coordinatorThreadFactory);
return new SchemaRegistry(
- operatorName, context, metadataApplier, routingRules, schemaChangeBehavior);
+ operatorName,
+ context,
+ coordinatorExecutor,
+ metadataApplier,
+ routingRules,
+ schemaChangeBehavior);
+ }
+
+ /** A thread factory class that provides some helper methods. */
+ public static class CoordinatorExecutorThreadFactory implements ThreadFactory {
+
+ private final String coordinatorThreadName;
+ private final ClassLoader cl;
+ private final Thread.UncaughtExceptionHandler errorHandler;
+
+ private Thread t;
+
+ CoordinatorExecutorThreadFactory(
+ final String coordinatorThreadName, final ClassLoader contextClassLoader) {
+ this(coordinatorThreadName, contextClassLoader, FatalExitExceptionHandler.INSTANCE);
+ }
+
+ CoordinatorExecutorThreadFactory(
+ final String coordinatorThreadName,
+ final ClassLoader contextClassLoader,
+ final Thread.UncaughtExceptionHandler errorHandler) {
+ this.coordinatorThreadName = coordinatorThreadName;
+ this.cl = contextClassLoader;
+ this.errorHandler = errorHandler;
+ }
+
+ @Override
+ public synchronized Thread newThread(Runnable r) {
+ if (t != null) {
+ throw new Error(
+ "This indicates that a fatal error has happened and caused the "
+ + "coordinator executor thread to exit. Check the earlier logs"
+ + "to see the root cause of the problem.");
+ }
+ t = new Thread(r, coordinatorThreadName);
+ t.setContextClassLoader(cl);
+ t.setUncaughtExceptionHandler(errorHandler);
+ return t;
+ }
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
index de5bde4e6..9cb0e3e00 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
@@ -38,6 +38,7 @@ import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeResponse;
import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeResultResponse;
import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -94,15 +95,19 @@ public class SchemaRegistryRequestHandler implements Closeable {
private final SchemaChangeBehavior schemaChangeBehavior;
+ private final OperatorCoordinator.Context context;
+
public SchemaRegistryRequestHandler(
MetadataApplier metadataApplier,
SchemaManager schemaManager,
SchemaDerivation schemaDerivation,
- SchemaChangeBehavior schemaChangeBehavior) {
+ SchemaChangeBehavior schemaChangeBehavior,
+ OperatorCoordinator.Context context) {
this.metadataApplier = metadataApplier;
this.schemaManager = schemaManager;
this.schemaDerivation = schemaDerivation;
this.schemaChangeBehavior = schemaChangeBehavior;
+ this.context = context;
this.activeSinkWriters = ConcurrentHashMap.newKeySet();
this.flushedSinkWriters = ConcurrentHashMap.newKeySet();
@@ -122,8 +127,8 @@ public class SchemaRegistryRequestHandler implements Closeable {
*
* @param request the received SchemaChangeRequest
*/
- public CompletableFuture<CoordinationResponse> handleSchemaChangeRequest(
- SchemaChangeRequest request) {
+ public void handleSchemaChangeRequest(
+ SchemaChangeRequest request, CompletableFuture<CoordinationResponse> response) {
// We use requester subTask ID as the pending ticket, because there will be at most 1 schema
// change requests simultaneously from each subTask
@@ -156,7 +161,8 @@ public class SchemaRegistryRequestHandler implements Closeable {
if (!pendingSubTaskIds.contains(requestSubTaskId)) {
pendingSubTaskIds.add(requestSubTaskId);
}
- return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.busy()));
+ response.complete(wrap(SchemaChangeResponse.busy()));
+ return;
}
SchemaChangeEvent event = request.getSchemaChangeEvent();
@@ -168,8 +174,8 @@ public class SchemaRegistryRequestHandler implements Closeable {
LOG.info(
"SchemaChangeStatus switched from WAITING_FOR_FLUSH to IDLE for request {} due to duplicated request.",
request);
- return CompletableFuture.completedFuture(
- wrap(SchemaChangeResponse.duplicate()));
+ response.complete(wrap(SchemaChangeResponse.duplicate()));
+ return;
}
schemaManager.applyOriginalSchemaChange(event);
List<SchemaChangeEvent> derivedSchemaChangeEvents =
@@ -183,7 +189,9 @@ public class SchemaRegistryRequestHandler implements Closeable {
LOG.info(
"SchemaChangeStatus switched from WAITING_FOR_FLUSH to IDLE for request {} due to ignored request.",
request);
- return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.ignored()));
+
+ response.complete(wrap(SchemaChangeResponse.ignored()));
+ return;
}
LOG.info(
@@ -191,8 +199,8 @@ public class SchemaRegistryRequestHandler implements Closeable {
// This request has been accepted.
schemaChangeStatus = RequestStatus.WAITING_FOR_FLUSH;
currentDerivedSchemaChangeEvents = new ArrayList<>(derivedSchemaChangeEvents);
- return CompletableFuture.completedFuture(
- wrap(SchemaChangeResponse.accepted(derivedSchemaChangeEvents)));
+
+ response.complete(wrap(SchemaChangeResponse.accepted(derivedSchemaChangeEvents)));
} else {
LOG.info(
"Schema Registry is busy processing a schema change request, could not handle request {} for now. Added {} to pending list ({}).",
@@ -202,7 +210,7 @@ public class SchemaRegistryRequestHandler implements Closeable {
if (!pendingSubTaskIds.contains(requestSubTaskId)) {
pendingSubTaskIds.add(requestSubTaskId);
}
- return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.busy()));
+ response.complete(wrap(SchemaChangeResponse.busy()));
}
}
}
@@ -299,7 +307,7 @@ public class SchemaRegistryRequestHandler implements Closeable {
}
}
- public CompletableFuture<CoordinationResponse> getSchemaChangeResult() {
+ public void getSchemaChangeResult(CompletableFuture<CoordinationResponse> response) {
Preconditions.checkState(
schemaChangeStatus != RequestStatus.IDLE,
"Illegal schemaChangeStatus: should not be IDLE before getting schema change request results.");
@@ -311,11 +319,12 @@ public class SchemaRegistryRequestHandler implements Closeable {
// This request has been finished, return it and prepare for the next request
List<SchemaChangeEvent> finishedEvents = clearCurrentSchemaChangeRequest();
- return CompletableFuture.supplyAsync(
- () -> wrap(new SchemaChangeResultResponse(finishedEvents)));
+ SchemaChangeResultResponse resultResponse =
+ new SchemaChangeResultResponse(finishedEvents);
+ response.complete(wrap(resultResponse));
} else {
// Still working on schema change request, waiting it
- return CompletableFuture.supplyAsync(() -> wrap(new SchemaChangeProcessingResponse()));
+ response.complete(wrap(new SchemaChangeProcessingResponse()));
}
}
@@ -444,7 +453,8 @@ public class SchemaRegistryRequestHandler implements Closeable {
private List<SchemaChangeEvent> clearCurrentSchemaChangeRequest() {
if (currentChangeException != null) {
- throw new RuntimeException("Failed to apply schema change.", currentChangeException);
+ context.failJob(
+ new RuntimeException("Failed to apply schema change.", currentChangeException));
}
List<SchemaChangeEvent> finishedSchemaChanges =
new ArrayList<>(currentFinishedSchemaChanges);
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/SchemaEvolveTest.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/SchemaEvolveTest.java
index d3e0de8d0..a3d5ce547 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/SchemaEvolveTest.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/schema/SchemaEvolveTest.java
@@ -30,6 +30,7 @@ import org.apache.flink.cdc.common.event.RenameColumnEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEventType;
import org.apache.flink.cdc.common.event.SchemaChangeEventTypeFamily;
import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.exceptions.UnsupportedSchemaChangeEventException;
import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.schema.Column;
import org.apache.flink.cdc.common.schema.Schema;
@@ -1035,11 +1036,16 @@ public class SchemaEvolveTest {
new AddColumnEvent.ColumnWithPosition(
Column.physicalColumn(
"height", DOUBLE, "Height data")))));
- Assertions.assertThatThrownBy(() -> processEvent(schemaOperator, addColumnEvents))
+ processEvent(schemaOperator, addColumnEvents);
+ Assertions.assertThat(harness.isJobFailed()).isEqualTo(true);
+ Assertions.assertThat(harness.getJobFailureCause())
.cause()
- .cause()
- .isExactlyInstanceOf(RuntimeException.class)
- .hasMessageContaining("Failed to apply schema change");
+ .isExactlyInstanceOf(UnsupportedSchemaChangeEventException.class)
+ .matches(
+ e ->
+ ((UnsupportedSchemaChangeEventException) e)
+ .getExceptionMessage()
+ .equals("Sink doesn't support such schema change event."));
harness.close();
}
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
index a469a3a91..d8b977029 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
@@ -38,7 +38,6 @@ import org.apache.flink.cdc.runtime.testutils.schema.TestingSchemaRegistryGatewa
import org.apache.flink.configuration.Configuration;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
-import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
import org.apache.flink.runtime.operators.testutils.DummyEnvironment;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.Output;
@@ -56,6 +55,7 @@ import java.time.Duration;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.Set;
+import java.util.concurrent.Executors;
import static org.apache.flink.cdc.runtime.operators.schema.event.CoordinationResponseUtils.unwrap;
@@ -81,6 +81,7 @@ public class EventOperatorTestHarness<OP extends AbstractStreamOperator<E>, E ex
private final SchemaRegistry schemaRegistry;
private final TestingSchemaRegistryGateway schemaRegistryGateway;
private final LinkedList<StreamRecord<E>> outputRecords = new LinkedList<>();
+ private final MockedOperatorCoordinatorContext mockedContext;
public EventOperatorTestHarness(OP operator, int numOutputs) {
this(operator, numOutputs, null, SchemaChangeBehavior.EVOLVE);
@@ -94,11 +95,14 @@ public class EventOperatorTestHarness<OP extends AbstractStreamOperator<E>, E ex
OP operator, int numOutputs, Duration duration, SchemaChangeBehavior behavior) {
this.operator = operator;
this.numOutputs = numOutputs;
+ this.mockedContext =
+ new MockedOperatorCoordinatorContext(
+ SCHEMA_OPERATOR_ID, Thread.currentThread().getContextClassLoader());
schemaRegistry =
new SchemaRegistry(
"SchemaOperator",
- new MockOperatorCoordinatorContext(
- SCHEMA_OPERATOR_ID, Thread.currentThread().getContextClassLoader()),
+ mockedContext,
+ Executors.newFixedThreadPool(1),
new CollectingMetadataApplier(duration),
new ArrayList<>(),
behavior);
@@ -113,11 +117,14 @@ public class EventOperatorTestHarness<OP extends AbstractStreamOperator<E>, E ex
Set<SchemaChangeEventType> enabledEventTypes) {
this.operator = operator;
this.numOutputs = numOutputs;
+ this.mockedContext =
+ new MockedOperatorCoordinatorContext(
+ SCHEMA_OPERATOR_ID, Thread.currentThread().getContextClassLoader());
schemaRegistry =
new SchemaRegistry(
"SchemaOperator",
- new MockOperatorCoordinatorContext(
- SCHEMA_OPERATOR_ID, Thread.currentThread().getContextClassLoader()),
+ mockedContext,
+ Executors.newFixedThreadPool(1),
new CollectingMetadataApplier(duration, enabledEventTypes),
new ArrayList<>(),
behavior);
@@ -133,11 +140,14 @@ public class EventOperatorTestHarness<OP extends AbstractStreamOperator<E>, E ex
Set<SchemaChangeEventType> errorsOnEventTypes) {
this.operator = operator;
this.numOutputs = numOutputs;
+ this.mockedContext =
+ new MockedOperatorCoordinatorContext(
+ SCHEMA_OPERATOR_ID, Thread.currentThread().getContextClassLoader());
schemaRegistry =
new SchemaRegistry(
"SchemaOperator",
- new MockOperatorCoordinatorContext(
- SCHEMA_OPERATOR_ID, Thread.currentThread().getContextClassLoader()),
+ mockedContext,
+ Executors.newFixedThreadPool(1),
new CollectingMetadataApplier(
duration, enabledEventTypes, errorsOnEventTypes),
new ArrayList<>(),
@@ -196,6 +206,14 @@ public class EventOperatorTestHarness<OP extends AbstractStreamOperator<E>, E ex
.orElse(null);
}
+ public boolean isJobFailed() {
+ return mockedContext.isJobFailed();
+ }
+
+ public Throwable getJobFailureCause() {
+ return mockedContext.getFailureCause();
+ }
+
@Override
public void close() throws Exception {
operator.close();
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/MockedOperatorCoordinatorContext.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/MockedOperatorCoordinatorContext.java
new file mode 100644
index 000000000..19ab961ee
--- /dev/null
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/MockedOperatorCoordinatorContext.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.runtime.testutils.operators;
+
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+
+/**
+ * This is a mocked version of Operator coordinator context that stores failure cause for testing
+ * purposes only.
+ */
+public class MockedOperatorCoordinatorContext extends MockOperatorCoordinatorContext {
+ public MockedOperatorCoordinatorContext(
+ OperatorID operatorID, ClassLoader userCodeClassLoader) {
+ super(operatorID, userCodeClassLoader);
+ }
+
+ private Throwable failureCause;
+
+ @Override
+ public void failJob(Throwable cause) {
+ super.failJob(cause);
+ failureCause = cause;
+ }
+
+ public Throwable getFailureCause() {
+ return failureCause;
+ }
+}
(flink-cdc) 08/12: [hotfix][cdc-runtime] Fix schema registry hanging in multiple parallelism
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 803d438657a1051e1944046f703c5f57ba030b04
Author: yuxiqian <34...@users.noreply.github.com>
AuthorDate: Thu Aug 22 20:52:06 2024 +0800
[hotfix][cdc-runtime] Fix schema registry hanging in multiple parallelism
---
.../flink/FlinkPipelineTransformITCase.java | 10 +-
.../flink/cdc/pipeline/tests/MysqlE2eITCase.java | 17 +--
.../tests/SchemaEvolvingTransformE2eITCase.java | 24 +---
.../cdc/pipeline/tests/TransformE2eITCase.java | 4 +-
.../schema/coordinator/SchemaManager.java | 152 ++++++++++-----------
.../schema/coordinator/SchemaRegistry.java | 14 +-
.../coordinator/SchemaRegistryRequestHandler.java | 60 ++++----
7 files changed, 130 insertions(+), 151 deletions(-)
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java
index c489b2e81..322a3358e 100644
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java
@@ -301,7 +301,7 @@ class FlinkPipelineTransformITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[3rd, 6, Fiona, 26, 3], after=[], op=DELETE, meta=()}",
// Alter column type stage
- "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, typeMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}, oldTypeMapping={gender=TINYINT, name=STRING, age=INT}}",
+ "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, nameMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[4th, 7, Gem, 19.0, -1], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[5th, 8, Helen, 18.0, -2], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[5th, 8, Helen, 18.0, -2], after=[5th, 8, Harry, 18.0, -3], op=UPDATE, meta=()}",
@@ -387,7 +387,7 @@ class FlinkPipelineTransformITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[3rd, 6, Fiona, 26, 3], after=[], op=DELETE, meta=()}",
// Alter column type stage
- "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, typeMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}, oldTypeMapping={gender=TINYINT, name=STRING, age=INT}}",
+ "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, nameMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[4th, 7, Gem, 19.0, -1], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[5th, 8, Helen, 18.0, -2], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[5th, 8, Helen, 18.0, -2], after=[5th, 8, Harry, 18.0, -3], op=UPDATE, meta=()}",
@@ -473,7 +473,7 @@ class FlinkPipelineTransformITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[5, Eve, 5 -> Eve], after=[5, Eva, 5 -> Eva], op=UPDATE, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[6, Fiona, 6 -> Fiona], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[6, Fiona, 6 -> Fiona], after=[], op=DELETE, meta=()}",
- "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, typeMapping={name=VARCHAR(17)}, oldTypeMapping={name=STRING}}",
+ "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, nameMapping={name=VARCHAR(17)}}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[7, Gem, 7 -> Gem], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[8, Helen, 8 -> Helen], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[8, Helen, 8 -> Helen], after=[8, Harry, 8 -> Harry], op=UPDATE, meta=()}",
@@ -559,7 +559,7 @@ class FlinkPipelineTransformITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[3rd, 6, Fiona, 26, 3, 6 -> Fiona], after=[], op=DELETE, meta=()}",
// Alter column type stage
- "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, typeMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}, oldTypeMapping={gender=TINYINT, name=STRING, age=INT}}",
+ "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, nameMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[4th, 7, Gem, 19.0, -1, 7 -> Gem], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[5th, 8, Helen, 18.0, -2, 8 -> Helen], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[5th, 8, Helen, 18.0, -2, 8 -> Helen], after=[5th, 8, Harry, 18.0, -3, 8 -> Harry], op=UPDATE, meta=()}",
@@ -651,7 +651,7 @@ class FlinkPipelineTransformITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[6 -> Fiona, 3rd, 6, Fiona, 26, 3], after=[], op=DELETE, meta=()}",
// Alter column type stage
- "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, typeMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}, oldTypeMapping={gender=TINYINT, name=STRING, age=INT}}",
+ "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, nameMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[7 -> Gem, 4th, 7, Gem, 19.0, -1], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[8 -> Helen, 5th, 8, Helen, 18.0, -2], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[8 -> Helen, 5th, 8, Helen, 18.0, -2], after=[8 -> Harry, 5th, 8, Harry, 18.0, -3], op=UPDATE, meta=()}",
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
index 1f730be62..c16f6de9c 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
@@ -296,22 +296,11 @@ public class MysqlE2eITCase extends PipelineTestEnvironment {
stat.execute("ALTER TABLE products DROP COLUMN new_column;");
stat.execute(
"INSERT INTO products VALUES (default,'evangelion','Eva',2.1728, null, null, null);"); // 114
-
- // Test TruncateTableEvent
- stat.execute("TRUNCATE TABLE products;");
-
- // Test DropTableEvent. It's all over.
- stat.execute("DROP TABLE products;");
} catch (SQLException e) {
LOG.error("Update table for CDC failed.", e);
throw e;
}
- waitUntilSpecificEvent(
- String.format(
- "DropTableEvent{tableId=%s.products}",
- mysqlInventoryDatabase.getDatabaseName()));
-
validateResult(
"DataChangeEvent{tableId=%s.products, before=[106, hammer, 16oz carpenter's hammer, 1.0, null, null, null], after=[106, hammer, 18oz carpenter hammer, 1.0, null, null, null], op=UPDATE, meta=()}",
"DataChangeEvent{tableId=%s.products, before=[107, rocks, box of assorted rocks, 5.3, null, null, null], after=[107, rocks, box of assorted rocks, 5.1, null, null, null], op=UPDATE, meta=()}",
@@ -321,14 +310,12 @@ public class MysqlE2eITCase extends PipelineTestEnvironment {
"DataChangeEvent{tableId=%s.products, before=[110, jacket, water resistent white wind breaker, 0.2, null, null, null, 1], after=[110, jacket, new water resistent white wind breaker, 0.5, null, null, null, 1], op=UPDATE, meta=()}",
"DataChangeEvent{tableId=%s.products, before=[111, scooter, Big 2-wheel scooter , 5.18, null, null, null, 1], after=[111, scooter, Big 2-wheel scooter , 5.17, null, null, null, 1], op=UPDATE, meta=()}",
"DataChangeEvent{tableId=%s.products, before=[111, scooter, Big 2-wheel scooter , 5.17, null, null, null, 1], after=[], op=DELETE, meta=()}",
- "AlterColumnTypeEvent{tableId=%s.products, typeMapping={new_col=BIGINT}, oldTypeMapping={new_col=INT}}",
+ "AlterColumnTypeEvent{tableId=%s.products, nameMapping={new_col=BIGINT}}",
"DataChangeEvent{tableId=%s.products, before=[], after=[112, derrida, forever 21, 2.1728, null, null, null, 2147483649], op=INSERT, meta=()}",
"RenameColumnEvent{tableId=%s.products, nameMapping={new_col=new_column}}",
"DataChangeEvent{tableId=%s.products, before=[], after=[113, dynazenon, SSSS, 2.1728, null, null, null, 2147483649], op=INSERT, meta=()}",
"DropColumnEvent{tableId=%s.products, droppedColumnNames=[new_column]}",
- "DataChangeEvent{tableId=%s.products, before=[], after=[114, evangelion, Eva, 2.1728, null, null, null], op=INSERT, meta=()}",
- "TruncateTableEvent{tableId=%s.products}",
- "DropTableEvent{tableId=%s.products}");
+ "DataChangeEvent{tableId=%s.products, before=[], after=[114, evangelion, Eva, 2.1728, null, null, null], op=INSERT, meta=()}");
}
private void validateResult(String... expectedEvents) throws Exception {
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
index 1d1f79f0f..a4a21c992 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
@@ -97,13 +97,11 @@ public class SchemaEvolvingTransformE2eITCase extends PipelineTestEnvironment {
Arrays.asList(
"AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}",
"DataChangeEvent{tableId=%s.members, before=[], after=[1012 -> Eve, 1012, Eve, 17, 0, 1024144, age < 20], op=INSERT, meta=()}",
- "AlterColumnTypeEvent{tableId=%s.members, typeMapping={age=DOUBLE}, oldTypeMapping={age=INT}}",
+ "AlterColumnTypeEvent{tableId=%s.members, nameMapping={age=DOUBLE}}",
"RenameColumnEvent{tableId=%s.members, nameMapping={gender=biological_sex}}",
"DropColumnEvent{tableId=%s.members, droppedColumnNames=[biological_sex]}",
"DataChangeEvent{tableId=%s.members, before=[], after=[1013 -> Fiona, 1013, Fiona, 16.0, 1026169, age < 20], op=INSERT, meta=()}",
- "TruncateTableEvent{tableId=%s.members}",
- "DataChangeEvent{tableId=%s.members, before=[], after=[1014 -> Gem, 1014, Gem, 17.0, 1028196, age < 20], op=INSERT, meta=()}",
- "DropTableEvent{tableId=%s.members}"));
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1014 -> Gem, 1014, Gem, 17.0, 1028196, age < 20], op=INSERT, meta=()}"));
}
@Test
@@ -186,10 +184,9 @@ public class SchemaEvolvingTransformE2eITCase extends PipelineTestEnvironment {
Arrays.asList(
"AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=LAST, existedColumnName=null}]}",
"DataChangeEvent{tableId=%s.members, before=[], after=[1012 -> Eve, 1012, Eve, 17, 1024144, age < 20, 0], op=INSERT, meta=()}",
- "AlterColumnTypeEvent{tableId=%s.members, typeMapping={age=DOUBLE}, oldTypeMapping={age=INT}}",
+ "AlterColumnTypeEvent{tableId=%s.members, nameMapping={age=DOUBLE}}",
"AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`biological_sex` TINYINT, position=LAST, existedColumnName=null}]}",
"DataChangeEvent{tableId=%s.members, before=[], after=[1013 -> Fiona, 1013, Fiona, 16.0, 1026169, age < 20, null, null], op=INSERT, meta=()}",
- "TruncateTableEvent{tableId=%s.members}",
"DataChangeEvent{tableId=%s.members, before=[], after=[1014 -> Gem, 1014, Gem, 17.0, 1028196, age < 20, null, null], op=INSERT, meta=()}"));
}
@@ -204,14 +201,12 @@ public class SchemaEvolvingTransformE2eITCase extends PipelineTestEnvironment {
Arrays.asList(
"AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}",
"DataChangeEvent{tableId=%s.members, before=[], after=[1012 -> Eve, 1012, Eve, 17, 0, 1024144, age < 20], op=INSERT, meta=()}",
- "AlterColumnTypeEvent{tableId=%s.members, typeMapping={age=DOUBLE}, oldTypeMapping={age=INT}}",
+ "AlterColumnTypeEvent{tableId=%s.members, nameMapping={age=DOUBLE}}",
"RenameColumnEvent{tableId=%s.members, nameMapping={gender=biological_sex}}",
"DataChangeEvent{tableId=%s.members, before=[], after=[1013 -> Fiona, 1013, Fiona, 16.0, null, 1026169, age < 20], op=INSERT, meta=()}",
- "TruncateTableEvent{tableId=%s.members}",
"DataChangeEvent{tableId=%s.members, before=[], after=[1014 -> Gem, 1014, Gem, 17.0, null, 1028196, age < 20], op=INSERT, meta=()}"),
- Arrays.asList(
- "Ignored schema change DropColumnEvent{tableId=%s.members, droppedColumnNames=[biological_sex]} to table %s.members.",
- "Ignored schema change DropTableEvent{tableId=%s.members} to table %s.members."));
+ Collections.singletonList(
+ "Ignored schema change DropColumnEvent{tableId=%s.members, droppedColumnNames=[biological_sex]} to table %s.members."));
}
@Test
@@ -352,13 +347,6 @@ public class SchemaEvolvingTransformE2eITCase extends PipelineTestEnvironment {
// triggers DropColumnEvent
stmt.execute("ALTER TABLE members DROP COLUMN biological_sex");
stmt.execute("INSERT INTO members VALUES (1013, 'Fiona', 16);");
-
- // triggers TruncateTableEvent
- stmt.execute("TRUNCATE TABLE members;");
- stmt.execute("INSERT INTO members VALUES (1014, 'Gem', 17);");
-
- // triggers DropTableEvent
- stmt.execute("DROP TABLE members;");
}
List<String> expectedTmEvents =
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
index 9c6130359..c1c8e8109 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
@@ -912,7 +912,7 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
validateEvents(
"AddColumnEvent{tableId=%s.TABLEALPHA, addedColumns=[ColumnWithPosition{column=`LAST` VARCHAR(17), position=AFTER, existedColumnName=NAMEALPHA}]}",
"DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3008, 8, 8, 80, 17, Jazz, Last, id -> 3008], op=INSERT, meta=()}",
- "AlterColumnTypeEvent{tableId=%s.TABLEALPHA, typeMapping={CODENAME=DOUBLE}, oldTypeMapping={CODENAME=TINYINT}}",
+ "AlterColumnTypeEvent{tableId=%s.TABLEALPHA, nameMapping={CODENAME=DOUBLE}}",
"RenameColumnEvent{tableId=%s.TABLEALPHA, nameMapping={CODENAME=CODE_NAME}}",
"RenameColumnEvent{tableId=%s.TABLEALPHA, nameMapping={CODE_NAME=CODE_NAME_EX}}",
"DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3009, 9, 9.0, 90, 18, Keka, Finale, id -> 3009], op=INSERT, meta=()}",
@@ -1019,7 +1019,7 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
"AddColumnEvent{tableId=%s.TABLEALPHA, addedColumns=[ColumnWithPosition{column=`CODENAME` TINYINT, position=AFTER, existedColumnName=VERSION}]}",
"AddColumnEvent{tableId=%s.TABLEALPHA, addedColumns=[ColumnWithPosition{column=`FIRST` VARCHAR(17), position=BEFORE, existedColumnName=ID}]}",
"DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3008 <- id, First, 3008, 8, 8, 80, 17, Jazz], op=INSERT, meta=()}",
- "AlterColumnTypeEvent{tableId=%s.TABLEALPHA, typeMapping={CODENAME=DOUBLE}, oldTypeMapping={CODENAME=TINYINT}}",
+ "AlterColumnTypeEvent{tableId=%s.TABLEALPHA, nameMapping={CODENAME=DOUBLE}}",
"RenameColumnEvent{tableId=%s.TABLEALPHA, nameMapping={CODENAME=CODE_NAME}}",
"RenameColumnEvent{tableId=%s.TABLEALPHA, nameMapping={CODE_NAME=CODE_NAME_EX}}",
"DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3009 <- id, 1st, 3009, 9, 9.0, 90, 18, Keka], op=INSERT, meta=()}",
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaManager.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaManager.java
index 60e70b7ed..75cc4eb32 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaManager.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaManager.java
@@ -19,10 +19,12 @@ package org.apache.flink.cdc.runtime.operators.schema.coordinator;
import org.apache.flink.cdc.common.annotation.Internal;
import org.apache.flink.cdc.common.event.AddColumnEvent;
+import org.apache.flink.cdc.common.event.AlterColumnTypeEvent;
import org.apache.flink.cdc.common.event.CreateTableEvent;
+import org.apache.flink.cdc.common.event.DropColumnEvent;
+import org.apache.flink.cdc.common.event.RenameColumnEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEvent;
import org.apache.flink.cdc.common.event.TableId;
-import org.apache.flink.cdc.common.event.visitor.SchemaChangeEventVisitor;
import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.schema.Column;
import org.apache.flink.cdc.common.schema.Schema;
@@ -106,87 +108,73 @@ public class SchemaManager {
public final boolean isOriginalSchemaChangeEventRedundant(SchemaChangeEvent event) {
TableId tableId = event.tableId();
Optional<Schema> latestSchema = getLatestOriginalSchema(tableId);
- return Boolean.TRUE.equals(
- SchemaChangeEventVisitor.visit(
- event,
- addColumnEvent -> {
- // It has not been applied if schema does not even exist
- if (!latestSchema.isPresent()) {
- return false;
- }
- List<Column> existedColumns = latestSchema.get().getColumns();
-
- // It has been applied only if all columns are present in existedColumns
- for (AddColumnEvent.ColumnWithPosition column :
- addColumnEvent.getAddedColumns()) {
- if (!existedColumns.contains(column.getAddColumn())) {
- return false;
- }
- }
- return true;
- },
- alterColumnTypeEvent -> {
- // It has not been applied if schema does not even exist
- if (!latestSchema.isPresent()) {
- return false;
- }
- Schema schema = latestSchema.get();
-
- // It has been applied only if all column types are set as expected
- for (Map.Entry<String, DataType> entry :
- alterColumnTypeEvent.getTypeMapping().entrySet()) {
- if (!schema.getColumn(entry.getKey()).isPresent()
- || !schema.getColumn(entry.getKey())
- .get()
- .getType()
- .equals(entry.getValue())) {
- return false;
- }
- }
- return true;
- },
- createTableEvent -> {
- // It has been applied if such table already exists
- return latestSchema.isPresent();
- },
- dropColumnEvent -> {
- // It has not been applied if schema does not even exist
- if (!latestSchema.isPresent()) {
- return false;
- }
- List<String> existedColumnNames = latestSchema.get().getColumnNames();
-
- // It has been applied only if corresponding column types do not exist
- return dropColumnEvent.getDroppedColumnNames().stream()
- .noneMatch(existedColumnNames::contains);
- },
- dropTableEvent -> {
- // It has been applied if such table does not exist
- return !latestSchema.isPresent();
- },
- renameColumnEvent -> {
- // It has been applied if such table already exists
- if (!latestSchema.isPresent()) {
- return false;
- }
- List<String> existedColumnNames = latestSchema.get().getColumnNames();
-
- // It has been applied only if all previous names do not exist, and all
- // new names already exist
- for (Map.Entry<String, String> entry :
- renameColumnEvent.getNameMapping().entrySet()) {
- if (existedColumnNames.contains(entry.getKey())
- || !existedColumnNames.contains(entry.getValue())) {
- return false;
- }
- }
- return true;
- },
- truncateTableEvent -> {
- // We have no way to ensure if a TruncateTableEvent has been applied
- // before. Just assume it's not.
- return false;
- }));
+
+ if (event instanceof AddColumnEvent) {
+ AddColumnEvent addColumnEvent = (AddColumnEvent) event;
+ if (!latestSchema.isPresent()) {
+ return false;
+ }
+ List<Column> existedColumns = latestSchema.get().getColumns();
+
+ // It has been applied only if all columns are present in existedColumns
+ for (AddColumnEvent.ColumnWithPosition column : addColumnEvent.getAddedColumns()) {
+ if (!existedColumns.contains(column.getAddColumn())) {
+ return false;
+ }
+ }
+ return true;
+ } else if (event instanceof AlterColumnTypeEvent) {
+ AlterColumnTypeEvent alterColumnTypeEvent = (AlterColumnTypeEvent) event;
+ // It has not been applied if schema does not even exist
+ if (!latestSchema.isPresent()) {
+ return false;
+ }
+ Schema schema = latestSchema.get();
+
+ // It has been applied only if all column types are set as expected
+ for (Map.Entry<String, DataType> entry :
+ alterColumnTypeEvent.getTypeMapping().entrySet()) {
+ if (!schema.getColumn(entry.getKey()).isPresent()
+ || !schema.getColumn(entry.getKey())
+ .get()
+ .getType()
+ .equals(entry.getValue())) {
+ return false;
+ }
+ }
+ return true;
+ } else if (event instanceof CreateTableEvent) {
+ return latestSchema.isPresent();
+ } else if (event instanceof DropColumnEvent) {
+ DropColumnEvent dropColumnEvent = (DropColumnEvent) event;
+ if (!latestSchema.isPresent()) {
+ return false;
+ }
+ List<String> existedColumnNames = latestSchema.get().getColumnNames();
+
+ // It has been applied only if corresponding column types do not exist
+ return dropColumnEvent.getDroppedColumnNames().stream()
+ .noneMatch(existedColumnNames::contains);
+ } else if (event instanceof RenameColumnEvent) {
+ RenameColumnEvent renameColumnEvent = (RenameColumnEvent) event;
+ // It has been applied if such table already exists
+ if (!latestSchema.isPresent()) {
+ return false;
+ }
+ List<String> existedColumnNames = latestSchema.get().getColumnNames();
+
+ // It has been applied only if all previous names do not exist, and all
+ // new names already exist
+ for (Map.Entry<String, String> entry : renameColumnEvent.getNameMapping().entrySet()) {
+ if (existedColumnNames.contains(entry.getKey())
+ || !existedColumnNames.contains(entry.getValue())) {
+ return false;
+ }
+ }
+ return true;
+ } else {
+ throw new RuntimeException("Unknown schema change event: " + event);
+ }
}
public final boolean schemaExists(
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
index 8ea3a1f93..617daed92 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
@@ -104,6 +104,12 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
private SchemaChangeBehavior schemaChangeBehavior;
+ /**
+ * Current parallelism. Use this to verify if Schema Registry has collected enough flush success
+ * events from sink operators.
+ */
+ private int currentParallelism;
+
public SchemaRegistry(
String operatorName,
OperatorCoordinator.Context context,
@@ -135,7 +141,9 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
public void start() throws Exception {
LOG.info("Starting SchemaRegistry for {}.", operatorName);
this.failedReasons.clear();
- LOG.info("Started SchemaRegistry for {}.", operatorName);
+ this.currentParallelism = context.currentParallelism();
+ LOG.info(
+ "Started SchemaRegistry for {}. Parallelism: {}", operatorName, currentParallelism);
}
@Override
@@ -155,7 +163,9 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
flushSuccessEvent.getSubtask(),
flushSuccessEvent.getTableId().toString());
requestHandler.flushSuccess(
- flushSuccessEvent.getTableId(), flushSuccessEvent.getSubtask());
+ flushSuccessEvent.getTableId(),
+ flushSuccessEvent.getSubtask(),
+ currentParallelism);
} else if (event instanceof SinkWriterRegisterEvent) {
requestHandler.registerSinkWriter(((SinkWriterRegisterEvent) event).getSubtask());
} else {
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
index 99019a6b4..9262a6a90 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
@@ -25,7 +25,6 @@ import org.apache.flink.cdc.common.event.DropColumnEvent;
import org.apache.flink.cdc.common.event.RenameColumnEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEventType;
-import org.apache.flink.cdc.common.event.SchemaChangeEventWithPreSchema;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.exceptions.UnsupportedSchemaChangeEventException;
import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
@@ -48,11 +47,11 @@ import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
-import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicReference;
@@ -103,8 +102,8 @@ public class SchemaRegistryRequestHandler implements Closeable {
this.schemaDerivation = schemaDerivation;
this.schemaChangeBehavior = schemaChangeBehavior;
- this.activeSinkWriters = new HashSet<>();
- this.flushedSinkWriters = new HashSet<>();
+ this.activeSinkWriters = ConcurrentHashMap.newKeySet();
+ this.flushedSinkWriters = ConcurrentHashMap.newKeySet();
this.schemaChangeThreadPool = Executors.newSingleThreadExecutor();
this.currentDerivedSchemaChangeEvents = new ArrayList<>();
@@ -122,7 +121,7 @@ public class SchemaRegistryRequestHandler implements Closeable {
SchemaChangeRequest request) {
if (schemaChangeStatus.compareAndSet(RequestStatus.IDLE, RequestStatus.WAITING_FOR_FLUSH)) {
LOG.info(
- "Received schema change event request {} from table {}. Start to buffer requests for others.",
+ "Received schema change event request {} from table {}. SchemaChangeStatus switched from IDLE to WAITING_FOR_FLUSH, other requests will be blocked.",
request.getSchemaChangeEvent(),
request.getTableId().toString());
SchemaChangeEvent event = request.getSchemaChangeEvent();
@@ -134,7 +133,11 @@ public class SchemaRegistryRequestHandler implements Closeable {
Preconditions.checkState(
schemaChangeStatus.compareAndSet(
RequestStatus.WAITING_FOR_FLUSH, RequestStatus.IDLE),
- "Illegal schemaChangeStatus state: should still in WAITING_FOR_FLUSH state if event was duplicated.");
+ "Illegal schemaChangeStatus state: should still in WAITING_FOR_FLUSH state if event was duplicated, not "
+ + schemaChangeStatus.get());
+ LOG.info(
+ "SchemaChangeStatus switched from WAITING_FOR_FLUSH to IDLE for request {} due to duplicated request.",
+ request);
return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.duplicate()));
}
schemaManager.applyOriginalSchemaChange(event);
@@ -149,22 +152,13 @@ public class SchemaRegistryRequestHandler implements Closeable {
Preconditions.checkState(
schemaChangeStatus.compareAndSet(
RequestStatus.WAITING_FOR_FLUSH, RequestStatus.IDLE),
- "Illegal schemaChangeStatus state: should still in WAITING_FOR_FLUSH state if event was ignored.");
+ "Illegal schemaChangeStatus state: should still in WAITING_FOR_FLUSH state if event was ignored, not "
+ + schemaChangeStatus.get());
+ LOG.info(
+ "SchemaChangeStatus switched from WAITING_FOR_FLUSH to IDLE for request {} due to ignored request.",
+ request);
return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.ignored()));
}
-
- // Backfill pre-schema info for sink applying
- derivedSchemaChangeEvents.forEach(
- e -> {
- if (e instanceof SchemaChangeEventWithPreSchema) {
- SchemaChangeEventWithPreSchema pe = (SchemaChangeEventWithPreSchema) e;
- if (!pe.hasPreSchema()) {
- schemaManager
- .getLatestEvolvedSchema(pe.tableId())
- .ifPresent(pe::fillPreSchema);
- }
- }
- });
currentDerivedSchemaChangeEvents = new ArrayList<>(derivedSchemaChangeEvents);
return CompletableFuture.completedFuture(
wrap(SchemaChangeResponse.accepted(derivedSchemaChangeEvents)));
@@ -220,7 +214,11 @@ public class SchemaRegistryRequestHandler implements Closeable {
}
Preconditions.checkState(
schemaChangeStatus.compareAndSet(RequestStatus.APPLYING, RequestStatus.FINISHED),
- "Illegal schemaChangeStatus state: should be APPLYING before applySchemaChange finishes");
+ "Illegal schemaChangeStatus state: should be APPLYING before applySchemaChange finishes, not "
+ + schemaChangeStatus.get());
+ LOG.info(
+ "SchemaChangeStatus switched from APPLYING to FINISHED for request {}.",
+ currentDerivedSchemaChangeEvents);
}
/**
@@ -239,13 +237,21 @@ public class SchemaRegistryRequestHandler implements Closeable {
* @param tableId the subtask in SchemaOperator and table that the FlushEvent is about
* @param sinkSubtask the sink subtask succeed flushing
*/
- public void flushSuccess(TableId tableId, int sinkSubtask) {
+ public void flushSuccess(TableId tableId, int sinkSubtask, int parallelism) {
flushedSinkWriters.add(sinkSubtask);
+ if (activeSinkWriters.size() < parallelism) {
+ LOG.info(
+ "Not all active sink writers have been registered. Current {}, expected {}.",
+ activeSinkWriters.size(),
+ parallelism);
+ return;
+ }
if (flushedSinkWriters.equals(activeSinkWriters)) {
Preconditions.checkState(
schemaChangeStatus.compareAndSet(
RequestStatus.WAITING_FOR_FLUSH, RequestStatus.APPLYING),
- "Illegal schemaChangeStatus state: should be WAITING_FOR_FLUSH before collecting enough FlushEvents");
+ "Illegal schemaChangeStatus state: should be WAITING_FOR_FLUSH before collecting enough FlushEvents, not "
+ + schemaChangeStatus);
LOG.info(
"All sink subtask have flushed for table {}. Start to apply schema change.",
tableId.toString());
@@ -259,6 +265,10 @@ public class SchemaRegistryRequestHandler implements Closeable {
!schemaChangeStatus.get().equals(RequestStatus.IDLE),
"Illegal schemaChangeStatus: should not be IDLE before getting schema change request results.");
if (schemaChangeStatus.compareAndSet(RequestStatus.FINISHED, RequestStatus.IDLE)) {
+ LOG.info(
+ "SchemaChangeStatus switched from FINISHED to IDLE for request {}",
+ currentDerivedSchemaChangeEvents);
+
// This request has been finished, return it and prepare for the next request
List<SchemaChangeEvent> finishedEvents = clearCurrentSchemaChangeRequest();
return CompletableFuture.supplyAsync(
@@ -379,10 +389,6 @@ public class SchemaRegistryRequestHandler implements Closeable {
}
return events;
}
- case DROP_TABLE:
- // We don't drop any tables in Lenient mode.
- LOG.info("A drop table event {} has been ignored in Lenient mode.", event);
- return Collections.emptyList();
default:
return Collections.singletonList(event);
}
(flink-cdc) 04/12: [FLINK-36081][cdc-connector][mysql] Remove the schemas of outdated tables in the BinlogSplit when restart (#3548)
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 39afc4fc891abcb937fa0517e651caad34df196d
Author: Kevin Wang <da...@163.com>
AuthorDate: Wed Aug 21 10:40:17 2024 +0800
[FLINK-36081][cdc-connector][mysql] Remove the schemas of outdated tables in the BinlogSplit when restart (#3548)
Co-authored-by: 云时 <mi...@alibaba-inc.com>
(cherry picked from commit 77c63385d947f3bb8e726561a7f01cd383941a96)
---
.../mysql/source/split/MySqlBinlogSplit.java | 31 +++--
.../mysql/source/split/MySqlBinlogSplitTest.java | 140 +++++++++++++++++++++
2 files changed, 164 insertions(+), 7 deletions(-)
diff --git a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/split/MySqlBinlogSplit.java b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/split/MySqlBinlogSplit.java
index 033844ab3..7a2a8ec20 100644
--- a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/split/MySqlBinlogSplit.java
+++ b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/split/MySqlBinlogSplit.java
@@ -183,30 +183,47 @@ public class MySqlBinlogSplit extends MySqlSplit {
*
* <p>When restore from a checkpoint, the finished split infos may contain some splits from the
* deleted tables. We need to remove these splits from the total finished split infos and update
- * the size.
+ * the size, while also removing the outdated tables from the table schemas of binlog split.
*/
public static MySqlBinlogSplit filterOutdatedSplitInfos(
MySqlBinlogSplit binlogSplit, Tables.TableFilter currentTableFilter) {
- Set<TableId> tablesToRemove =
+ Map<TableId, TableChange> filteredTableSchemas =
+ binlogSplit.getTableSchemas().entrySet().stream()
+ .filter(entry -> currentTableFilter.isIncluded(entry.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+
+ Set<TableId> tablesToRemoveInFinishedSnapshotSplitInfos =
binlogSplit.getFinishedSnapshotSplitInfos().stream()
.filter(i -> !currentTableFilter.isIncluded(i.getTableId()))
.map(split -> split.getTableId())
.collect(Collectors.toSet());
- if (tablesToRemove.isEmpty()) {
- return binlogSplit;
+ if (tablesToRemoveInFinishedSnapshotSplitInfos.isEmpty()) {
+ return new MySqlBinlogSplit(
+ binlogSplit.splitId,
+ binlogSplit.getStartingOffset(),
+ binlogSplit.getEndingOffset(),
+ binlogSplit.getFinishedSnapshotSplitInfos(),
+ filteredTableSchemas,
+ binlogSplit.totalFinishedSplitSize,
+ binlogSplit.isSuspended());
}
- LOG.info("Reader remove tables after restart: {}", tablesToRemove);
+ LOG.info(
+ "Reader remove tables after restart: {}",
+ tablesToRemoveInFinishedSnapshotSplitInfos);
List<FinishedSnapshotSplitInfo> allFinishedSnapshotSplitInfos =
binlogSplit.getFinishedSnapshotSplitInfos().stream()
- .filter(i -> !tablesToRemove.contains(i.getTableId()))
+ .filter(
+ i ->
+ !tablesToRemoveInFinishedSnapshotSplitInfos.contains(
+ i.getTableId()))
.collect(Collectors.toList());
return new MySqlBinlogSplit(
binlogSplit.splitId,
binlogSplit.getStartingOffset(),
binlogSplit.getEndingOffset(),
allFinishedSnapshotSplitInfos,
- binlogSplit.getTableSchemas(),
+ filteredTableSchemas,
binlogSplit.getTotalFinishedSplitSize()
- (binlogSplit.getFinishedSnapshotSplitInfos().size()
- allFinishedSnapshotSplitInfos.size()),
diff --git a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/split/MySqlBinlogSplitTest.java b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/split/MySqlBinlogSplitTest.java
new file mode 100644
index 000000000..6066c19f8
--- /dev/null
+++ b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/split/MySqlBinlogSplitTest.java
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.mysql.source.split;
+
+import org.apache.flink.cdc.connectors.mysql.source.offset.BinlogOffset;
+
+import io.debezium.relational.Column;
+import io.debezium.relational.Table;
+import io.debezium.relational.TableEditor;
+import io.debezium.relational.TableId;
+import io.debezium.relational.Tables;
+import io.debezium.relational.history.TableChanges;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/** Unit tests for {@link MySqlBinlogSplit}. */
+public class MySqlBinlogSplitTest {
+
+ @Test
+ public void filterOutdatedSplitInfos() {
+ Map<TableId, TableChanges.TableChange> tableSchemas = new HashMap<>();
+
+ // mock table1
+ TableId tableId1 = new TableId("catalog1", null, "table1");
+
+ TableChanges.TableChange tableChange1 =
+ new TableChanges.TableChange(
+ TableChanges.TableChangeType.CREATE,
+ new MockTable(TableId.parse("catalog1.table1")));
+
+ // mock table2
+ TableId tableId2 = new TableId("catalog2", null, "table2");
+
+ TableChanges.TableChange tableChange2 =
+ new TableChanges.TableChange(
+ TableChanges.TableChangeType.CREATE,
+ new MockTable(TableId.parse("catalog2.table2")));
+ tableSchemas.put(tableId1, tableChange1);
+ tableSchemas.put(tableId2, tableChange2);
+ MySqlBinlogSplit binlogSplit =
+ new MySqlBinlogSplit(
+ "binlog-split",
+ BinlogOffset.ofLatest(),
+ null,
+ new ArrayList<>(),
+ tableSchemas,
+ 0,
+ false);
+
+ // case 1: only include table1
+ Tables.TableFilter currentTableFilter = tableId -> tableId.table().equals("table1");
+
+ MySqlBinlogSplit mySqlBinlogSplit =
+ MySqlBinlogSplit.filterOutdatedSplitInfos(binlogSplit, currentTableFilter);
+ Map<TableId, TableChanges.TableChange> filterTableSchemas =
+ mySqlBinlogSplit.getTableSchemas();
+ Assert.assertEquals(1, filterTableSchemas.size());
+ Assert.assertEquals(tableChange1, filterTableSchemas.get(tableId1));
+
+ // case 2: include all tables
+ currentTableFilter = tableId -> tableId.table().startsWith("table");
+
+ mySqlBinlogSplit =
+ MySqlBinlogSplit.filterOutdatedSplitInfos(binlogSplit, currentTableFilter);
+ filterTableSchemas = mySqlBinlogSplit.getTableSchemas();
+ Assert.assertEquals(2, filterTableSchemas.size());
+ Assert.assertEquals(tableChange1, filterTableSchemas.get(tableId1));
+ Assert.assertEquals(tableChange2, filterTableSchemas.get(tableId2));
+ }
+
+ /** A mock implementation for {@link Table} which is used for unit tests. */
+ private static class MockTable implements Table {
+ private final TableId tableId;
+
+ public MockTable(TableId tableId) {
+ this.tableId = tableId;
+ }
+
+ @Override
+ public TableId id() {
+ return tableId;
+ }
+
+ @Override
+ public List<String> primaryKeyColumnNames() {
+ return Collections.emptyList();
+ }
+
+ @Override
+ public List<String> retrieveColumnNames() {
+ return Collections.emptyList();
+ }
+
+ @Override
+ public List<Column> columns() {
+ return Collections.emptyList();
+ }
+
+ @Override
+ public Column columnWithName(String name) {
+ throw new UnsupportedOperationException("Not implemented.");
+ }
+
+ @Override
+ public String defaultCharsetName() {
+ return "UTF-8";
+ }
+
+ @Override
+ public String comment() {
+ return null;
+ }
+
+ @Override
+ public TableEditor edit() {
+ throw new UnsupportedOperationException("Not implemented.");
+ }
+ }
+}
(flink-cdc) 07/12: [FLINK-36092][cdc-runtime] Fix schema evolution failure with wildcarded transform rule
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit dacbe7c34a4f310fdfaf0abd38f7efb0adc74fde
Author: yuxiqian <34...@users.noreply.github.com>
AuthorDate: Thu Aug 22 20:13:14 2024 +0800
[FLINK-36092][cdc-runtime] Fix schema evolution failure with wildcarded transform rule
This closes #3557.
(cherry picked from commit 3837887819021638a7b82018887d0dd146a4b4de)
---
.../apache/flink/cdc/common/utils/SchemaUtils.java | 71 ++
.../flink/FlinkPipelineComposerITCase.java | 8 +-
.../flink/FlinkPipelineTransformITCase.java | 622 ++++++++++++++++
.../cdc/composer/flink/FlinkPipelineUdfITCase.java | 20 +-
.../flink/cdc/pipeline/tests/RouteE2eITCase.java | 2 +-
.../tests/SchemaEvolvingTransformE2eITCase.java | 432 +++++++++++
.../cdc/pipeline/tests/TransformE2eITCase.java | 336 ++++++++-
.../operators/transform/PostTransformOperator.java | 55 +-
.../operators/transform/PreTransformOperator.java | 101 ++-
.../flink/cdc/runtime/parser/TransformParser.java | 22 +
.../transform/PreTransformOperatorTest.java | 11 +-
.../TransformOperatorWithSchemaEvolveTest.java | 792 +++++++++++++++++++++
12 files changed, 2412 insertions(+), 60 deletions(-)
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/utils/SchemaUtils.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/utils/SchemaUtils.java
index 68cdcd0e1..928a3b9db 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/utils/SchemaUtils.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/utils/SchemaUtils.java
@@ -39,7 +39,9 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedList;
import java.util.List;
+import java.util.Map;
import java.util.Objects;
+import java.util.Optional;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
@@ -352,4 +354,73 @@ public class SchemaUtils {
});
return oldSchema.copy(columns);
}
+
+ public static Optional<SchemaChangeEvent> transformSchemaChangeEvent(
+ boolean hasAsterisk, List<String> referencedColumns, SchemaChangeEvent event) {
+ Optional<SchemaChangeEvent> evolvedSchemaChangeEvent = Optional.empty();
+ if (event instanceof AddColumnEvent) {
+ // Send add column events to downstream iff there's an asterisk
+ if (hasAsterisk) {
+ List<AddColumnEvent.ColumnWithPosition> addedColumns =
+ ((AddColumnEvent) event)
+ .getAddedColumns().stream()
+ .map(
+ e -> {
+ if (AddColumnEvent.ColumnPosition.LAST.equals(
+ e.getPosition())) {
+ return new AddColumnEvent
+ .ColumnWithPosition(
+ e.getAddColumn(),
+ AddColumnEvent.ColumnPosition.AFTER,
+ referencedColumns.get(
+ referencedColumns.size()
+ - 1));
+ } else if (AddColumnEvent.ColumnPosition.FIRST
+ .equals(e.getPosition())) {
+ return new AddColumnEvent
+ .ColumnWithPosition(
+ e.getAddColumn(),
+ AddColumnEvent.ColumnPosition
+ .BEFORE,
+ referencedColumns.get(0));
+ } else {
+ return e;
+ }
+ })
+ .collect(Collectors.toList());
+ evolvedSchemaChangeEvent =
+ Optional.of(new AddColumnEvent(event.tableId(), addedColumns));
+ }
+ } else if (event instanceof AlterColumnTypeEvent) {
+ AlterColumnTypeEvent alterColumnTypeEvent = (AlterColumnTypeEvent) event;
+ if (hasAsterisk) {
+ // In wildcard mode, all alter column type events should be sent to downstream
+ evolvedSchemaChangeEvent = Optional.of(event);
+ } else {
+ // Or, we need to filter out those referenced columns and reconstruct
+ // SchemaChangeEvents
+ Map<String, DataType> newDataTypeMap =
+ alterColumnTypeEvent.getTypeMapping().entrySet().stream()
+ .filter(e -> referencedColumns.contains(e.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+ if (!newDataTypeMap.isEmpty()) {
+ evolvedSchemaChangeEvent =
+ Optional.of(
+ new AlterColumnTypeEvent(
+ alterColumnTypeEvent.tableId(), newDataTypeMap));
+ }
+ }
+ } else if (event instanceof RenameColumnEvent) {
+ if (hasAsterisk) {
+ evolvedSchemaChangeEvent = Optional.of(event);
+ }
+ } else if (event instanceof DropColumnEvent) {
+ if (hasAsterisk) {
+ evolvedSchemaChangeEvent = Optional.of(event);
+ }
+ } else {
+ evolvedSchemaChangeEvent = Optional.of(event);
+ }
+ return evolvedSchemaChangeEvent;
+ }
}
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposerITCase.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposerITCase.java
index 45cdbe9e8..8e471d886 100644
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposerITCase.java
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposerITCase.java
@@ -333,7 +333,7 @@ class FlinkPipelineComposerITCase {
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING,`col12` STRING}, primaryKeys=col1, partitionKeys=col12, options=({key1=value1})}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1, 10], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, 20], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, 10], after=[], op=DELETE, meta=()}",
@@ -393,7 +393,7 @@ class FlinkPipelineComposerITCase {
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING,`col12` STRING,`rk` STRING}, primaryKeys=col1, partitionKeys=col12, options=({key1=value1})}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1, 10, +I], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, 20, +I], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, 10, -D], after=[], op=DELETE, meta=()}",
@@ -461,7 +461,7 @@ class FlinkPipelineComposerITCase {
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING,`col12` STRING}, primaryKeys=col1, partitionKeys=col12, options=({key1=value1})}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1, 11], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, 22], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, 11], after=[], op=DELETE, meta=()}",
@@ -1010,7 +1010,7 @@ class FlinkPipelineComposerITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.merged, before=[4, Donald, 25, last_name, student], after=[], op=DELETE, meta=()}",
"AddColumnEvent{tableId=default_namespace.default_schema.merged, addedColumns=[ColumnWithPosition{column=`gender` STRING, position=LAST, existedColumnName=null}]}",
"DataChangeEvent{tableId=default_namespace.default_schema.merged, before=[], after=[5, Eliza, 24, last_name, null, null], op=INSERT, meta=()}",
- "DataChangeEvent{tableId=default_namespace.default_schema.merged, before=[], after=[6, Frank, 30, male, student, last_name], op=INSERT, meta=()}");
+ "DataChangeEvent{tableId=default_namespace.default_schema.merged, before=[], after=[6, Frank, 30, last_name, student, male], op=INSERT, meta=()}");
}
@ParameterizedTest
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java
index c23b8624d..c489b2e81 100644
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java
@@ -18,12 +18,18 @@
package org.apache.flink.cdc.composer.flink;
import org.apache.flink.cdc.common.configuration.Configuration;
+import org.apache.flink.cdc.common.data.binary.BinaryRecordData;
import org.apache.flink.cdc.common.data.binary.BinaryStringData;
+import org.apache.flink.cdc.common.event.AddColumnEvent;
+import org.apache.flink.cdc.common.event.AlterColumnTypeEvent;
import org.apache.flink.cdc.common.event.CreateTableEvent;
import org.apache.flink.cdc.common.event.DataChangeEvent;
+import org.apache.flink.cdc.common.event.DropColumnEvent;
import org.apache.flink.cdc.common.event.Event;
+import org.apache.flink.cdc.common.event.RenameColumnEvent;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.pipeline.PipelineOptions;
+import org.apache.flink.cdc.common.schema.Column;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.types.DataType;
import org.apache.flink.cdc.common.types.DataTypes;
@@ -41,6 +47,8 @@ import org.apache.flink.cdc.runtime.typeutils.BinaryRecordDataGenerator;
import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
import org.apache.flink.test.junit5.MiniClusterExtension;
+import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
+
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -233,6 +241,608 @@ class FlinkPipelineTransformITCase {
Arrays.stream(outputEvents).forEach(this::extractDataLines);
}
+ @Test
+ void testVanillaTransformWithSchemaEvolution() throws Exception {
+ FlinkPipelineComposer composer = FlinkPipelineComposer.ofMiniCluster();
+
+ // Setup value source
+ Configuration sourceConfig = new Configuration();
+ sourceConfig.set(
+ ValuesDataSourceOptions.EVENT_SET_ID,
+ ValuesDataSourceHelper.EventSetId.CUSTOM_SOURCE_EVENTS);
+
+ TableId tableId = TableId.tableId("default_namespace", "default_schema", "mytable1");
+ List<Event> events = generateSchemaEvolutionEvents(tableId);
+
+ ValuesDataSourceHelper.setSourceEvents(Collections.singletonList(events));
+
+ SourceDef sourceDef =
+ new SourceDef(ValuesDataFactory.IDENTIFIER, "Value Source", sourceConfig);
+
+ // Setup value sink
+ Configuration sinkConfig = new Configuration();
+ SinkDef sinkDef = new SinkDef(ValuesDataFactory.IDENTIFIER, "Value Sink", sinkConfig);
+
+ // Setup pipeline
+ Configuration pipelineConfig = new Configuration();
+ pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ PipelineDef pipelineDef =
+ new PipelineDef(
+ sourceDef,
+ sinkDef,
+ Collections.emptyList(),
+ Collections.emptyList(),
+ Collections.emptyList(),
+ pipelineConfig);
+
+ // Execute the pipeline
+ PipelineExecution execution = composer.compose(pipelineDef);
+ execution.execute();
+
+ // Check the order and content of all received events
+ String[] outputEvents = outCaptor.toString().trim().split("\n");
+
+ assertThat(outputEvents)
+ .containsExactly(
+ // Initial stage
+ "CreateTableEvent{tableId=default_namespace.default_schema.mytable1, schema=columns={`id` INT,`name` STRING,`age` INT}, primaryKeys=id, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[1, Alice, 21], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[2, Barcarolle, 22], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[3, Cecily, 23], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[3, Cecily, 23], after=[3, Colin, 24], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[2, Barcarolle, 22], after=[], op=DELETE, meta=()}",
+
+ // Add column stage
+ "AddColumnEvent{tableId=default_namespace.default_schema.mytable1, addedColumns=[ColumnWithPosition{column=`rank` STRING, position=FIRST, existedColumnName=null}, ColumnWithPosition{column=`gender` TINYINT, position=LAST, existedColumnName=null}]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[1st, 4, Derrida, 24, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[2nd, 5, Eve, 25, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[2nd, 5, Eve, 25, 1], after=[2nd, 5, Eva, 20, 2], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[3rd, 6, Fiona, 26, 3], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[3rd, 6, Fiona, 26, 3], after=[], op=DELETE, meta=()}",
+
+ // Alter column type stage
+ "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, typeMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}, oldTypeMapping={gender=TINYINT, name=STRING, age=INT}}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[4th, 7, Gem, 19.0, -1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[5th, 8, Helen, 18.0, -2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[5th, 8, Helen, 18.0, -2], after=[5th, 8, Harry, 18.0, -3], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[6th, 9, IINA, 17.0, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[6th, 9, IINA, 17.0, 0], after=[], op=DELETE, meta=()}",
+
+ // Rename column stage
+ "RenameColumnEvent{tableId=default_namespace.default_schema.mytable1, nameMapping={gender=biological_sex, age=toshi}}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[7th, 10, Julia, 24.0, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[8th, 11, Kalle, 23.0, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[8th, 11, Kalle, 23.0, 0], after=[8th, 11, Kella, 18.0, 0], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[9th, 12, Lynx, 17.0, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[9th, 12, Lynx, 17.0, 0], after=[], op=DELETE, meta=()}",
+
+ // Drop column stage
+ "DropColumnEvent{tableId=default_namespace.default_schema.mytable1, droppedColumnNames=[biological_sex, toshi]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[10th, 13, Munroe], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[11th, 14, Neko], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[11th, 14, Neko], after=[11th, 14, Nein], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[12th, 15, Oops], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[12th, 15, Oops], after=[], op=DELETE, meta=()}");
+ }
+
+ @Test
+ void testWildcardTransformWithSchemaEvolution() throws Exception {
+ FlinkPipelineComposer composer = FlinkPipelineComposer.ofMiniCluster();
+
+ // Setup value source
+ Configuration sourceConfig = new Configuration();
+ sourceConfig.set(
+ ValuesDataSourceOptions.EVENT_SET_ID,
+ ValuesDataSourceHelper.EventSetId.CUSTOM_SOURCE_EVENTS);
+
+ TableId tableId = TableId.tableId("default_namespace", "default_schema", "mytable1");
+ List<Event> events = generateSchemaEvolutionEvents(tableId);
+
+ ValuesDataSourceHelper.setSourceEvents(Collections.singletonList(events));
+
+ SourceDef sourceDef =
+ new SourceDef(ValuesDataFactory.IDENTIFIER, "Value Source", sourceConfig);
+
+ // Setup value sink
+ Configuration sinkConfig = new Configuration();
+ SinkDef sinkDef = new SinkDef(ValuesDataFactory.IDENTIFIER, "Value Sink", sinkConfig);
+
+ // Setup pipeline
+ Configuration pipelineConfig = new Configuration();
+ pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ PipelineDef pipelineDef =
+ new PipelineDef(
+ sourceDef,
+ sinkDef,
+ Collections.emptyList(),
+ Collections.singletonList(
+ new TransformDef(
+ tableId.toString(), "*", null, null, null, null, null)),
+ Collections.emptyList(),
+ pipelineConfig);
+
+ // Execute the pipeline
+ PipelineExecution execution = composer.compose(pipelineDef);
+ execution.execute();
+
+ // Check the order and content of all received events
+ String[] outputEvents = outCaptor.toString().trim().split("\n");
+
+ assertThat(outputEvents)
+ .containsExactly(
+ // Initial stage
+ "CreateTableEvent{tableId=default_namespace.default_schema.mytable1, schema=columns={`id` INT,`name` STRING,`age` INT}, primaryKeys=id, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[1, Alice, 21], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[2, Barcarolle, 22], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[3, Cecily, 23], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[3, Cecily, 23], after=[3, Colin, 24], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[2, Barcarolle, 22], after=[], op=DELETE, meta=()}",
+
+ // Add column stage
+ "AddColumnEvent{tableId=default_namespace.default_schema.mytable1, addedColumns=[ColumnWithPosition{column=`rank` STRING, position=BEFORE, existedColumnName=id}, ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[1st, 4, Derrida, 24, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[2nd, 5, Eve, 25, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[2nd, 5, Eve, 25, 1], after=[2nd, 5, Eva, 20, 2], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[3rd, 6, Fiona, 26, 3], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[3rd, 6, Fiona, 26, 3], after=[], op=DELETE, meta=()}",
+
+ // Alter column type stage
+ "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, typeMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}, oldTypeMapping={gender=TINYINT, name=STRING, age=INT}}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[4th, 7, Gem, 19.0, -1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[5th, 8, Helen, 18.0, -2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[5th, 8, Helen, 18.0, -2], after=[5th, 8, Harry, 18.0, -3], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[6th, 9, IINA, 17.0, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[6th, 9, IINA, 17.0, 0], after=[], op=DELETE, meta=()}",
+
+ // Rename column stage
+ "RenameColumnEvent{tableId=default_namespace.default_schema.mytable1, nameMapping={gender=biological_sex, age=toshi}}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[7th, 10, Julia, 24.0, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[8th, 11, Kalle, 23.0, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[8th, 11, Kalle, 23.0, 0], after=[8th, 11, Kella, 18.0, 0], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[9th, 12, Lynx, 17.0, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[9th, 12, Lynx, 17.0, 0], after=[], op=DELETE, meta=()}",
+
+ // Drop column stage
+ "DropColumnEvent{tableId=default_namespace.default_schema.mytable1, droppedColumnNames=[biological_sex, toshi]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[10th, 13, Munroe], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[11th, 14, Neko], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[11th, 14, Neko], after=[11th, 14, Nein], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[12th, 15, Oops], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[12th, 15, Oops], after=[], op=DELETE, meta=()}");
+ }
+
+ @Test
+ void testExplicitTransformWithSchemaEvolution() throws Exception {
+ FlinkPipelineComposer composer = FlinkPipelineComposer.ofMiniCluster();
+
+ // Setup value source
+ Configuration sourceConfig = new Configuration();
+ sourceConfig.set(
+ ValuesDataSourceOptions.EVENT_SET_ID,
+ ValuesDataSourceHelper.EventSetId.CUSTOM_SOURCE_EVENTS);
+
+ TableId tableId = TableId.tableId("default_namespace", "default_schema", "mytable1");
+ List<Event> events = generateSchemaEvolutionEvents(tableId);
+
+ ValuesDataSourceHelper.setSourceEvents(Collections.singletonList(events));
+
+ SourceDef sourceDef =
+ new SourceDef(ValuesDataFactory.IDENTIFIER, "Value Source", sourceConfig);
+
+ // Setup value sink
+ Configuration sinkConfig = new Configuration();
+ SinkDef sinkDef = new SinkDef(ValuesDataFactory.IDENTIFIER, "Value Sink", sinkConfig);
+
+ // Setup pipeline
+ Configuration pipelineConfig = new Configuration();
+ pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ PipelineDef pipelineDef =
+ new PipelineDef(
+ sourceDef,
+ sinkDef,
+ Collections.emptyList(),
+ Collections.singletonList(
+ new TransformDef(
+ tableId.toString(),
+ "id, name, CAST(id AS VARCHAR) || ' -> ' || name AS extend_id",
+ null,
+ null,
+ null,
+ null,
+ null)),
+ Collections.emptyList(),
+ pipelineConfig);
+
+ // Execute the pipeline
+ PipelineExecution execution = composer.compose(pipelineDef);
+ execution.execute();
+
+ // Check the order and content of all received events
+ String[] outputEvents = outCaptor.toString().trim().split("\n");
+
+ assertThat(outputEvents)
+ .containsExactly(
+ "CreateTableEvent{tableId=default_namespace.default_schema.mytable1, schema=columns={`id` INT,`name` STRING,`extend_id` STRING}, primaryKeys=id, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[1, Alice, 1 -> Alice], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[2, Barcarolle, 2 -> Barcarolle], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[3, Cecily, 3 -> Cecily], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[3, Cecily, 3 -> Cecily], after=[3, Colin, 3 -> Colin], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[2, Barcarolle, 2 -> Barcarolle], after=[], op=DELETE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[4, Derrida, 4 -> Derrida], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[5, Eve, 5 -> Eve], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[5, Eve, 5 -> Eve], after=[5, Eva, 5 -> Eva], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[6, Fiona, 6 -> Fiona], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[6, Fiona, 6 -> Fiona], after=[], op=DELETE, meta=()}",
+ "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, typeMapping={name=VARCHAR(17)}, oldTypeMapping={name=STRING}}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[7, Gem, 7 -> Gem], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[8, Helen, 8 -> Helen], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[8, Helen, 8 -> Helen], after=[8, Harry, 8 -> Harry], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[9, IINA, 9 -> IINA], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[9, IINA, 9 -> IINA], after=[], op=DELETE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[10, Julia, 10 -> Julia], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[11, Kalle, 11 -> Kalle], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[11, Kalle, 11 -> Kalle], after=[11, Kella, 11 -> Kella], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[12, Lynx, 12 -> Lynx], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[12, Lynx, 12 -> Lynx], after=[], op=DELETE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[13, Munroe, 13 -> Munroe], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[14, Neko, 14 -> Neko], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[14, Neko, 14 -> Neko], after=[14, Nein, 14 -> Nein], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[15, Oops, 15 -> Oops], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[15, Oops, 15 -> Oops], after=[], op=DELETE, meta=()}");
+ }
+
+ @Test
+ void testPreAsteriskWithSchemaEvolution() throws Exception {
+ FlinkPipelineComposer composer = FlinkPipelineComposer.ofMiniCluster();
+
+ // Setup value source
+ Configuration sourceConfig = new Configuration();
+ sourceConfig.set(
+ ValuesDataSourceOptions.EVENT_SET_ID,
+ ValuesDataSourceHelper.EventSetId.CUSTOM_SOURCE_EVENTS);
+
+ TableId tableId = TableId.tableId("default_namespace", "default_schema", "mytable1");
+ List<Event> events = generateSchemaEvolutionEvents(tableId);
+
+ ValuesDataSourceHelper.setSourceEvents(Collections.singletonList(events));
+
+ SourceDef sourceDef =
+ new SourceDef(ValuesDataFactory.IDENTIFIER, "Value Source", sourceConfig);
+
+ // Setup value sink
+ Configuration sinkConfig = new Configuration();
+ SinkDef sinkDef = new SinkDef(ValuesDataFactory.IDENTIFIER, "Value Sink", sinkConfig);
+
+ // Setup pipeline
+ Configuration pipelineConfig = new Configuration();
+ pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ PipelineDef pipelineDef =
+ new PipelineDef(
+ sourceDef,
+ sinkDef,
+ Collections.emptyList(),
+ Collections.singletonList(
+ new TransformDef(
+ tableId.toString(),
+ "*, CAST(id AS VARCHAR) || ' -> ' || name AS extend_id",
+ null,
+ null,
+ null,
+ null,
+ null)),
+ Collections.emptyList(),
+ pipelineConfig);
+
+ // Execute the pipeline
+ PipelineExecution execution = composer.compose(pipelineDef);
+ execution.execute();
+
+ // Check the order and content of all received events
+ String[] outputEvents = outCaptor.toString().trim().split("\n");
+
+ assertThat(outputEvents)
+ .containsExactly(
+ // Initial stage
+ "CreateTableEvent{tableId=default_namespace.default_schema.mytable1, schema=columns={`id` INT,`name` STRING,`age` INT,`extend_id` STRING}, primaryKeys=id, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[1, Alice, 21, 1 -> Alice], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[2, Barcarolle, 22, 2 -> Barcarolle], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[3, Cecily, 23, 3 -> Cecily], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[3, Cecily, 23, 3 -> Cecily], after=[3, Colin, 24, 3 -> Colin], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[2, Barcarolle, 22, 2 -> Barcarolle], after=[], op=DELETE, meta=()}",
+
+ // Add column stage
+ "AddColumnEvent{tableId=default_namespace.default_schema.mytable1, addedColumns=[ColumnWithPosition{column=`rank` STRING, position=BEFORE, existedColumnName=id}, ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[1st, 4, Derrida, 24, 0, 4 -> Derrida], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[2nd, 5, Eve, 25, 1, 5 -> Eve], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[2nd, 5, Eve, 25, 1, 5 -> Eve], after=[2nd, 5, Eva, 20, 2, 5 -> Eva], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[3rd, 6, Fiona, 26, 3, 6 -> Fiona], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[3rd, 6, Fiona, 26, 3, 6 -> Fiona], after=[], op=DELETE, meta=()}",
+
+ // Alter column type stage
+ "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, typeMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}, oldTypeMapping={gender=TINYINT, name=STRING, age=INT}}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[4th, 7, Gem, 19.0, -1, 7 -> Gem], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[5th, 8, Helen, 18.0, -2, 8 -> Helen], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[5th, 8, Helen, 18.0, -2, 8 -> Helen], after=[5th, 8, Harry, 18.0, -3, 8 -> Harry], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[6th, 9, IINA, 17.0, 0, 9 -> IINA], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[6th, 9, IINA, 17.0, 0, 9 -> IINA], after=[], op=DELETE, meta=()}",
+
+ // Rename column stage
+ "RenameColumnEvent{tableId=default_namespace.default_schema.mytable1, nameMapping={gender=biological_sex, age=toshi}}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[7th, 10, Julia, 24.0, 1, 10 -> Julia], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[8th, 11, Kalle, 23.0, 0, 11 -> Kalle], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[8th, 11, Kalle, 23.0, 0, 11 -> Kalle], after=[8th, 11, Kella, 18.0, 0, 11 -> Kella], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[9th, 12, Lynx, 17.0, 0, 12 -> Lynx], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[9th, 12, Lynx, 17.0, 0, 12 -> Lynx], after=[], op=DELETE, meta=()}",
+
+ // Drop column stage
+ "DropColumnEvent{tableId=default_namespace.default_schema.mytable1, droppedColumnNames=[biological_sex, toshi]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[10th, 13, Munroe, 13 -> Munroe], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[11th, 14, Neko, 14 -> Neko], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[11th, 14, Neko, 14 -> Neko], after=[11th, 14, Nein, 14 -> Nein], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[12th, 15, Oops, 15 -> Oops], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[12th, 15, Oops, 15 -> Oops], after=[], op=DELETE, meta=()}");
+ }
+
+ @Test
+ void testPostAsteriskWithSchemaEvolution() throws Exception {
+ FlinkPipelineComposer composer = FlinkPipelineComposer.ofMiniCluster();
+
+ // Setup value source
+ Configuration sourceConfig = new Configuration();
+ sourceConfig.set(
+ ValuesDataSourceOptions.EVENT_SET_ID,
+ ValuesDataSourceHelper.EventSetId.CUSTOM_SOURCE_EVENTS);
+
+ TableId tableId = TableId.tableId("default_namespace", "default_schema", "mytable1");
+ List<Event> events = generateSchemaEvolutionEvents(tableId);
+
+ ValuesDataSourceHelper.setSourceEvents(Collections.singletonList(events));
+
+ SourceDef sourceDef =
+ new SourceDef(ValuesDataFactory.IDENTIFIER, "Value Source", sourceConfig);
+
+ // Setup value sink
+ Configuration sinkConfig = new Configuration();
+ SinkDef sinkDef = new SinkDef(ValuesDataFactory.IDENTIFIER, "Value Sink", sinkConfig);
+
+ // Setup pipeline
+ Configuration pipelineConfig = new Configuration();
+ pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ PipelineDef pipelineDef =
+ new PipelineDef(
+ sourceDef,
+ sinkDef,
+ Collections.emptyList(),
+ Collections.singletonList(
+ new TransformDef(
+ tableId.toString(),
+ "CAST(id AS VARCHAR) || ' -> ' || name AS extend_id, *",
+ "id > 0",
+ null,
+ null,
+ null,
+ null)),
+ Collections.emptyList(),
+ pipelineConfig);
+
+ // Execute the pipeline
+ PipelineExecution execution = composer.compose(pipelineDef);
+ execution.execute();
+
+ // Check the order and content of all received events
+ String[] outputEvents = outCaptor.toString().trim().split("\n");
+
+ assertThat(outputEvents)
+ .containsExactly(
+ // Initial stage
+ "CreateTableEvent{tableId=default_namespace.default_schema.mytable1, schema=columns={`extend_id` STRING,`id` INT,`name` STRING,`age` INT}, primaryKeys=id, options=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[1 -> Alice, 1, Alice, 21], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[2 -> Barcarolle, 2, Barcarolle, 22], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[3 -> Cecily, 3, Cecily, 23], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[3 -> Cecily, 3, Cecily, 23], after=[3 -> Colin, 3, Colin, 24], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[2 -> Barcarolle, 2, Barcarolle, 22], after=[], op=DELETE, meta=()}",
+
+ // Add column stage
+ "AddColumnEvent{tableId=default_namespace.default_schema.mytable1, addedColumns=[ColumnWithPosition{column=`rank` STRING, position=BEFORE, existedColumnName=id}, ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[4 -> Derrida, 1st, 4, Derrida, 24, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[5 -> Eve, 2nd, 5, Eve, 25, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[5 -> Eve, 2nd, 5, Eve, 25, 1], after=[5 -> Eva, 2nd, 5, Eva, 20, 2], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[6 -> Fiona, 3rd, 6, Fiona, 26, 3], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[6 -> Fiona, 3rd, 6, Fiona, 26, 3], after=[], op=DELETE, meta=()}",
+
+ // Alter column type stage
+ "AlterColumnTypeEvent{tableId=default_namespace.default_schema.mytable1, typeMapping={gender=INT, name=VARCHAR(17), age=DOUBLE}, oldTypeMapping={gender=TINYINT, name=STRING, age=INT}}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[7 -> Gem, 4th, 7, Gem, 19.0, -1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[8 -> Helen, 5th, 8, Helen, 18.0, -2], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[8 -> Helen, 5th, 8, Helen, 18.0, -2], after=[8 -> Harry, 5th, 8, Harry, 18.0, -3], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[9 -> IINA, 6th, 9, IINA, 17.0, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[9 -> IINA, 6th, 9, IINA, 17.0, 0], after=[], op=DELETE, meta=()}",
+
+ // Rename column stage
+ "RenameColumnEvent{tableId=default_namespace.default_schema.mytable1, nameMapping={gender=biological_sex, age=toshi}}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[10 -> Julia, 7th, 10, Julia, 24.0, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[11 -> Kalle, 8th, 11, Kalle, 23.0, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[11 -> Kalle, 8th, 11, Kalle, 23.0, 0], after=[11 -> Kella, 8th, 11, Kella, 18.0, 0], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[12 -> Lynx, 9th, 12, Lynx, 17.0, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[12 -> Lynx, 9th, 12, Lynx, 17.0, 0], after=[], op=DELETE, meta=()}",
+
+ // Drop column stage
+ "DropColumnEvent{tableId=default_namespace.default_schema.mytable1, droppedColumnNames=[biological_sex, toshi]}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[13 -> Munroe, 10th, 13, Munroe], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[14 -> Neko, 11th, 14, Neko], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[14 -> Neko, 11th, 14, Neko], after=[14 -> Nein, 11th, 14, Nein], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[], after=[15 -> Oops, 12th, 15, Oops], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=default_namespace.default_schema.mytable1, before=[15 -> Oops, 12th, 15, Oops], after=[], op=DELETE, meta=()}");
+ }
+
+ private List<Event> generateSchemaEvolutionEvents(TableId tableId) {
+ List<Event> events = new ArrayList<>();
+
+ // Initial schema
+ {
+ Schema schemaV1 =
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build();
+
+ events.add(new CreateTableEvent(tableId, schemaV1));
+ events.add(DataChangeEvent.insertEvent(tableId, generate(schemaV1, 1, "Alice", 21)));
+ events.add(
+ DataChangeEvent.insertEvent(tableId, generate(schemaV1, 2, "Barcarolle", 22)));
+ events.add(DataChangeEvent.insertEvent(tableId, generate(schemaV1, 3, "Cecily", 23)));
+ events.add(
+ DataChangeEvent.updateEvent(
+ tableId,
+ generate(schemaV1, 3, "Cecily", 23),
+ generate(schemaV1, 3, "Colin", 24)));
+ events.add(
+ DataChangeEvent.deleteEvent(tableId, generate(schemaV1, 2, "Barcarolle", 22)));
+ }
+
+ // test AddColumnEvent
+ {
+ events.add(
+ new AddColumnEvent(
+ tableId,
+ Arrays.asList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn("rank", DataTypes.STRING()),
+ AddColumnEvent.ColumnPosition.FIRST,
+ null),
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "gender", DataTypes.TINYINT())))));
+ Schema schemaV2 =
+ Schema.newBuilder()
+ .physicalColumn("rank", DataTypes.STRING())
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .physicalColumn("gender", DataTypes.TINYINT())
+ .primaryKey("id")
+ .build();
+ events.add(
+ DataChangeEvent.insertEvent(
+ tableId, generate(schemaV2, "1st", 4, "Derrida", 24, (byte) 0)));
+ events.add(
+ DataChangeEvent.insertEvent(
+ tableId, generate(schemaV2, "2nd", 5, "Eve", 25, (byte) 1)));
+ events.add(
+ DataChangeEvent.updateEvent(
+ tableId,
+ generate(schemaV2, "2nd", 5, "Eve", 25, (byte) 1),
+ generate(schemaV2, "2nd", 5, "Eva", 20, (byte) 2)));
+ events.add(
+ DataChangeEvent.insertEvent(
+ tableId, generate(schemaV2, "3rd", 6, "Fiona", 26, (byte) 3)));
+ events.add(
+ DataChangeEvent.deleteEvent(
+ tableId, generate(schemaV2, "3rd", 6, "Fiona", 26, (byte) 3)));
+ }
+
+ // test AlterColumnTypeEvent
+ {
+ events.add(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "age",
+ DataTypes.DOUBLE(),
+ "gender",
+ DataTypes.INT(),
+ "name",
+ DataTypes.VARCHAR(17))));
+ Schema schemaV3 =
+ Schema.newBuilder()
+ .physicalColumn("rank", DataTypes.STRING())
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.DOUBLE())
+ .physicalColumn("gender", DataTypes.INT())
+ .primaryKey("id")
+ .build();
+ events.add(
+ DataChangeEvent.insertEvent(
+ tableId, generate(schemaV3, "4th", 7, "Gem", 19d, -1)));
+ events.add(
+ DataChangeEvent.insertEvent(
+ tableId, generate(schemaV3, "5th", 8, "Helen", 18d, -2)));
+ events.add(
+ DataChangeEvent.updateEvent(
+ tableId,
+ generate(schemaV3, "5th", 8, "Helen", 18d, -2),
+ generate(schemaV3, "5th", 8, "Harry", 18d, -3)));
+ events.add(
+ DataChangeEvent.insertEvent(
+ tableId, generate(schemaV3, "6th", 9, "IINA", 17d, 0)));
+ events.add(
+ DataChangeEvent.deleteEvent(
+ tableId, generate(schemaV3, "6th", 9, "IINA", 17d, 0)));
+ }
+
+ // test RenameColumnEvent
+ {
+ events.add(
+ new RenameColumnEvent(
+ tableId, ImmutableMap.of("gender", "biological_sex", "age", "toshi")));
+ Schema schemaV4 =
+ Schema.newBuilder()
+ .physicalColumn("rank", DataTypes.STRING())
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("toshi", DataTypes.DOUBLE())
+ .physicalColumn("biological_sex", DataTypes.INT())
+ .primaryKey("id")
+ .build();
+ events.add(
+ DataChangeEvent.insertEvent(
+ tableId, generate(schemaV4, "7th", 10, "Julia", 24d, 1)));
+ events.add(
+ DataChangeEvent.insertEvent(
+ tableId, generate(schemaV4, "8th", 11, "Kalle", 23d, 0)));
+ events.add(
+ DataChangeEvent.updateEvent(
+ tableId,
+ generate(schemaV4, "8th", 11, "Kalle", 23d, 0),
+ generate(schemaV4, "8th", 11, "Kella", 18d, 0)));
+ events.add(
+ DataChangeEvent.insertEvent(
+ tableId, generate(schemaV4, "9th", 12, "Lynx", 17d, 0)));
+ events.add(
+ DataChangeEvent.deleteEvent(
+ tableId, generate(schemaV4, "9th", 12, "Lynx", 17d, 0)));
+ }
+
+ // test DropColumnEvent
+ {
+ events.add(new DropColumnEvent(tableId, Arrays.asList("biological_sex", "toshi")));
+ Schema schemaV5 =
+ Schema.newBuilder()
+ .physicalColumn("rank", DataTypes.STRING())
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .primaryKey("id")
+ .build();
+ events.add(
+ DataChangeEvent.insertEvent(tableId, generate(schemaV5, "10th", 13, "Munroe")));
+ events.add(
+ DataChangeEvent.insertEvent(tableId, generate(schemaV5, "11th", 14, "Neko")));
+ events.add(
+ DataChangeEvent.updateEvent(
+ tableId,
+ generate(schemaV5, "11th", 14, "Neko"),
+ generate(schemaV5, "11th", 14, "Nein")));
+ events.add(
+ DataChangeEvent.insertEvent(tableId, generate(schemaV5, "12th", 15, "Oops")));
+ events.add(
+ DataChangeEvent.deleteEvent(tableId, generate(schemaV5, "12th", 15, "Oops")));
+ }
+ return events;
+ }
+
void extractDataLines(String line) {
if (!line.startsWith("DataChangeEvent{")) {
return;
@@ -285,4 +895,16 @@ class FlinkPipelineTransformITCase {
assertThat(instant.toEpochMilli() / milliSecondsInOneDay)
.isEqualTo(Long.parseLong(currentDate));
}
+
+ BinaryRecordData generate(Schema schema, Object... fields) {
+ return (new BinaryRecordDataGenerator(schema.getColumnDataTypes().toArray(new DataType[0])))
+ .generate(
+ Arrays.stream(fields)
+ .map(
+ e ->
+ (e instanceof String)
+ ? BinaryStringData.fromString((String) e)
+ : e)
+ .toArray());
+ }
}
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineUdfITCase.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineUdfITCase.java
index c85cf8050..f9be7d7dd 100644
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineUdfITCase.java
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineUdfITCase.java
@@ -160,7 +160,7 @@ public class FlinkPipelineUdfITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1, from 1 to z is lie], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, from 2 to z is lie], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3, from 3 to z is lie], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, from 1 to z is lie], after=[], op=DELETE, meta=()}",
@@ -227,7 +227,7 @@ public class FlinkPipelineUdfITCase {
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING,`collen` STRING}, primaryKeys=col1, options=({key1=value1})}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, 3], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3, 4], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[2, , 3], after=[2, x, 3], op=UPDATE, meta=()}");
@@ -294,7 +294,7 @@ public class FlinkPipelineUdfITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1, Boolean: true, Integer: 1, Double: 3.14, String: str], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, Boolean: true, Integer: 1, Double: 3.14, String: str], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3, Boolean: true, Integer: 1, Double: 3.14, String: str], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, Boolean: true, Integer: 1, Double: 3.14, String: str], after=[], op=DELETE, meta=()}",
@@ -363,7 +363,7 @@ public class FlinkPipelineUdfITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1, #0], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, #1], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3, #2], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, #3], after=[], op=DELETE, meta=()}",
@@ -432,7 +432,7 @@ public class FlinkPipelineUdfITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1, Forty-two], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, Forty-two], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3, Forty-two], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, Forty-two], after=[], op=DELETE, meta=()}",
@@ -514,7 +514,7 @@ public class FlinkPipelineUdfITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1, 3, Integer: 42, 1-42, #0], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, 4, Integer: 42, 2-42, #1], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3, 5, Integer: 42, 3-42, #2], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, 3, Integer: 42, 1-42, #3], after=[], op=DELETE, meta=()}",
@@ -586,7 +586,7 @@ public class FlinkPipelineUdfITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1, from 1 to z is lie], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, from 2 to z is lie], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3, from 3 to z is lie], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, from 1 to z is lie], after=[], op=DELETE, meta=()}",
@@ -652,7 +652,7 @@ public class FlinkPipelineUdfITCase {
"CreateTableEvent{tableId=default_namespace.default_schema.table1, schema=columns={`col1` STRING,`col2` STRING,`collen` STRING}, primaryKeys=col1, options=({key1=value1})}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, 3], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3, 4], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[2, , 3], after=[2, x, 3], op=UPDATE, meta=()}");
@@ -718,7 +718,7 @@ public class FlinkPipelineUdfITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1, Boolean: true, Integer: 1, Double: 3.14, String: str], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, Boolean: true, Integer: 1, Double: 3.14, String: str], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3, Boolean: true, Integer: 1, Double: 3.14, String: str], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, Boolean: true, Integer: 1, Double: 3.14, String: str], after=[], op=DELETE, meta=()}",
@@ -794,7 +794,7 @@ public class FlinkPipelineUdfITCase {
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[1, 1, 3, Integer: 42, 1-42], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[2, 2, 4, Integer: 42, 2-42], op=INSERT, meta=()}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[], after=[3, 3, 5, Integer: 42, 3-42], op=INSERT, meta=()}",
- "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=LAST, existedColumnName=null}]}",
+ "AddColumnEvent{tableId=default_namespace.default_schema.table1, addedColumns=[ColumnWithPosition{column=`col3` STRING, position=AFTER, existedColumnName=col2}]}",
"RenameColumnEvent{tableId=default_namespace.default_schema.table1, nameMapping={col2=newCol2, col3=newCol3}}",
"DropColumnEvent{tableId=default_namespace.default_schema.table1, droppedColumnNames=[newCol2]}",
"DataChangeEvent{tableId=default_namespace.default_schema.table1, before=[1, 1, 3, Integer: 42, 1-42], after=[], op=DELETE, meta=()}",
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java
index f4ea4d81a..b4db8d5be 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java
@@ -709,7 +709,7 @@ public class RouteE2eITCase extends PipelineTestEnvironment {
validateResult(
"AddColumnEvent{tableId=%s.ALL, addedColumns=[ColumnWithPosition{column=`NAME` VARCHAR(17), position=LAST, existedColumnName=null}]}",
- "DataChangeEvent{tableId=%s.ALL, before=[], after=[10001, 12, Derrida, extras], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.ALL, before=[], after=[10001, 12, extras, Derrida], op=INSERT, meta=()}",
"AddColumnEvent{tableId=%s.ALL, addedColumns=[ColumnWithPosition{column=`VERSION_EX` VARCHAR(17), position=LAST, existedColumnName=null}]}",
"DataChangeEvent{tableId=%s.ALL, before=[], after=[10002, null, extras, null, 15], op=INSERT, meta=()}",
"AlterColumnTypeEvent{tableId=%s.ALL, nameMapping={VERSION=STRING}}",
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
new file mode 100644
index 000000000..1d1f79f0f
--- /dev/null
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.pipeline.tests;
+
+import org.apache.flink.cdc.common.test.utils.TestUtils;
+import org.apache.flink.cdc.connectors.mysql.testutils.MySqlContainer;
+import org.apache.flink.cdc.connectors.mysql.testutils.MySqlVersion;
+import org.apache.flink.cdc.connectors.mysql.testutils.UniqueDatabase;
+import org.apache.flink.cdc.pipeline.tests.utils.PipelineTestEnvironment;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.containers.output.ToStringConsumer;
+
+import java.nio.file.Path;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.Statement;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/** E2e tests for Schema Evolution cases. */
+public class SchemaEvolvingTransformE2eITCase extends PipelineTestEnvironment {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(SchemaEvolvingTransformE2eITCase.class);
+
+ // ------------------------------------------------------------------------------------------
+ // MySQL Variables (we always use MySQL as the data source for easier verifying)
+ // ------------------------------------------------------------------------------------------
+ protected static final String MYSQL_TEST_USER = "mysqluser";
+ protected static final String MYSQL_TEST_PASSWORD = "mysqlpw";
+ protected static final String INTER_CONTAINER_MYSQL_ALIAS = "mysql";
+ protected static final long EVENT_WAITING_TIMEOUT = 60000L;
+
+ @ClassRule
+ public static final MySqlContainer MYSQL =
+ (MySqlContainer)
+ new MySqlContainer(
+ MySqlVersion.V8_0) // v8 support both ARM and AMD architectures
+ .withConfigurationOverride("docker/mysql/my.cnf")
+ .withSetupSQL("docker/mysql/setup.sql")
+ .withDatabaseName("flink-test")
+ .withUsername("flinkuser")
+ .withPassword("flinkpw")
+ .withNetwork(NETWORK)
+ .withNetworkAliases(INTER_CONTAINER_MYSQL_ALIAS)
+ .withLogConsumer(new Slf4jLogConsumer(LOG));
+
+ protected final UniqueDatabase schemaEvolveDatabase =
+ new UniqueDatabase(MYSQL, "schema_evolve", MYSQL_TEST_USER, MYSQL_TEST_PASSWORD);
+
+ @Before
+ public void before() throws Exception {
+ super.before();
+ schemaEvolveDatabase.createAndInitialize();
+ }
+
+ @After
+ public void after() {
+ super.after();
+ schemaEvolveDatabase.dropDatabase();
+ }
+
+ @Test
+ public void testSchemaEvolve() throws Exception {
+ testGenericSchemaEvolution(
+ "evolve",
+ false,
+ false,
+ false,
+ Arrays.asList(
+ "AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1012 -> Eve, 1012, Eve, 17, 0, 1024144, age < 20], op=INSERT, meta=()}",
+ "AlterColumnTypeEvent{tableId=%s.members, typeMapping={age=DOUBLE}, oldTypeMapping={age=INT}}",
+ "RenameColumnEvent{tableId=%s.members, nameMapping={gender=biological_sex}}",
+ "DropColumnEvent{tableId=%s.members, droppedColumnNames=[biological_sex]}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1013 -> Fiona, 1013, Fiona, 16.0, 1026169, age < 20], op=INSERT, meta=()}",
+ "TruncateTableEvent{tableId=%s.members}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1014 -> Gem, 1014, Gem, 17.0, 1028196, age < 20], op=INSERT, meta=()}",
+ "DropTableEvent{tableId=%s.members}"));
+ }
+
+ @Test
+ public void testSchemaEvolveWithIncompatibleChanges() throws Exception {
+ testGenericSchemaEvolution(
+ "evolve",
+ true,
+ false,
+ false,
+ Collections.emptyList(),
+ Arrays.asList(
+ "java.lang.IllegalStateException: Incompatible types: \"INT\" and \"DOUBLE\"",
+ "org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy"));
+ }
+
+ @Test
+ public void testSchemaEvolveWithException() throws Exception {
+ testGenericSchemaEvolution(
+ "evolve",
+ false,
+ true,
+ false,
+ Collections.emptyList(),
+ Arrays.asList(
+ "Failed to apply schema change AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]} to table %s.members. Caused by: UnsupportedSchemaChangeEventException{applyingEvent=AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}, exceptionMessage='Rejected schema change event since error.on.schema.change is enabl [...]
+ "UnsupportedSchemaChangeEventException{applyingEvent=AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}, exceptionMessage='Rejected schema change event since error.on.schema.change is enabled.', cause='null'}",
+ "org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy"));
+ }
+
+ @Test
+ public void testSchemaTryEvolveWithException() throws Exception {
+ testGenericSchemaEvolution(
+ "try_evolve",
+ false,
+ true,
+ false,
+ Arrays.asList(
+ // Add column never succeeded, so age column will not appear.
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1012 -> Eve, 1012, Eve, 17, 1024144, age < 20], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1013 -> Fiona, 1013, Fiona, null, 1026169, age < 20], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1014 -> Gem, 1014, Gem, null, 1028196, age < 20], op=INSERT, meta=()}"),
+ Arrays.asList(
+ "Failed to apply schema change AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]} to table %s.members. Caused by: UnsupportedSchemaChangeEventException{applyingEvent=AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}, exceptionMessage='Rejected schema change event since error.on.schema.change is enabl [...]
+ "UnsupportedSchemaChangeEventException{applyingEvent=AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}, exceptionMessage='Rejected schema change event since error.on.schema.change is enabled.', cause='null'}"));
+ }
+
+ @Test
+ public void testSchemaIgnore() throws Exception {
+
+ testGenericSchemaEvolution(
+ "ignore",
+ false,
+ false,
+ false,
+ Arrays.asList(
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1012 -> Eve, 1012, Eve, 17, 1024144, age < 20], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1013 -> Fiona, 1013, Fiona, null, 1026169, age < 20], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1014 -> Gem, 1014, Gem, null, 1028196, age < 20], op=INSERT, meta=()}"));
+ }
+
+ @Test
+ public void testSchemaException() throws Exception {
+ testGenericSchemaEvolution(
+ "exception",
+ false,
+ false,
+ false,
+ Collections.emptyList(),
+ Collections.singletonList(
+ "java.lang.RuntimeException: Refused to apply schema change event AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]} in EXCEPTION mode."));
+ }
+
+ @Test
+ public void testLenientSchemaEvolution() throws Exception {
+ testGenericSchemaEvolution(
+ "lenient",
+ false,
+ false,
+ false,
+ Arrays.asList(
+ "AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=LAST, existedColumnName=null}]}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1012 -> Eve, 1012, Eve, 17, 1024144, age < 20, 0], op=INSERT, meta=()}",
+ "AlterColumnTypeEvent{tableId=%s.members, typeMapping={age=DOUBLE}, oldTypeMapping={age=INT}}",
+ "AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`biological_sex` TINYINT, position=LAST, existedColumnName=null}]}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1013 -> Fiona, 1013, Fiona, 16.0, 1026169, age < 20, null, null], op=INSERT, meta=()}",
+ "TruncateTableEvent{tableId=%s.members}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1014 -> Gem, 1014, Gem, 17.0, 1028196, age < 20, null, null], op=INSERT, meta=()}"));
+ }
+
+ @Test
+ public void testFineGrainedSchemaEvolution() throws Exception {
+
+ testGenericSchemaEvolution(
+ "evolve",
+ false,
+ false,
+ true,
+ Arrays.asList(
+ "AddColumnEvent{tableId=%s.members, addedColumns=[ColumnWithPosition{column=`gender` TINYINT, position=AFTER, existedColumnName=age}]}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1012 -> Eve, 1012, Eve, 17, 0, 1024144, age < 20], op=INSERT, meta=()}",
+ "AlterColumnTypeEvent{tableId=%s.members, typeMapping={age=DOUBLE}, oldTypeMapping={age=INT}}",
+ "RenameColumnEvent{tableId=%s.members, nameMapping={gender=biological_sex}}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1013 -> Fiona, 1013, Fiona, 16.0, null, 1026169, age < 20], op=INSERT, meta=()}",
+ "TruncateTableEvent{tableId=%s.members}",
+ "DataChangeEvent{tableId=%s.members, before=[], after=[1014 -> Gem, 1014, Gem, 17.0, null, 1028196, age < 20], op=INSERT, meta=()}"),
+ Arrays.asList(
+ "Ignored schema change DropColumnEvent{tableId=%s.members, droppedColumnNames=[biological_sex]} to table %s.members.",
+ "Ignored schema change DropTableEvent{tableId=%s.members} to table %s.members."));
+ }
+
+ @Test
+ public void testUnexpectedBehavior() {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: mysql\n"
+ + " hostname: %s\n"
+ + " port: 3306\n"
+ + " username: %s\n"
+ + " password: %s\n"
+ + " tables: %s.members\n"
+ + " server-id: 5400-5404\n"
+ + " server-time-zone: UTC\n"
+ + "\n"
+ + "sink:\n"
+ + " type: values\n"
+ + "\n"
+ + "pipeline:\n"
+ + " schema.change.behavior: unexpected\n"
+ + " parallelism: 1",
+ INTER_CONTAINER_MYSQL_ALIAS,
+ MYSQL_TEST_USER,
+ MYSQL_TEST_PASSWORD,
+ schemaEvolveDatabase.getDatabaseName());
+ Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
+ Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
+ Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
+
+ // Submitting job should fail given an unknown schema change behavior configuration
+ Assert.assertThrows(
+ AssertionError.class,
+ () -> submitPipelineJob(pipelineJob, mysqlCdcJar, valuesCdcJar, mysqlDriverJar));
+ }
+
+ private void testGenericSchemaEvolution(
+ String behavior,
+ boolean mergeTable,
+ boolean triggerError,
+ boolean fineGrained,
+ List<String> expectedTaskManagerEvents)
+ throws Exception {
+ testGenericSchemaEvolution(
+ behavior,
+ mergeTable,
+ triggerError,
+ fineGrained,
+ expectedTaskManagerEvents,
+ Collections.emptyList());
+ }
+
+ private void testGenericSchemaEvolution(
+ String behavior,
+ boolean mergeTable,
+ boolean triggerError,
+ boolean fineGrained,
+ List<String> expectedTaskManagerEvents,
+ List<String> expectedJobManagerEvents)
+ throws Exception {
+
+ String dbName = schemaEvolveDatabase.getDatabaseName();
+
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: mysql\n"
+ + " hostname: %s\n"
+ + " port: 3306\n"
+ + " username: %s\n"
+ + " password: %s\n"
+ + " tables: %s.%s\n"
+ + " server-id: 5400-5404\n"
+ + " server-time-zone: UTC\n"
+ + "\n"
+ + "sink:\n"
+ + " type: values\n"
+ + (fineGrained
+ ? " exclude.schema.changes:\n" + " - drop\n"
+ : "")
+ + (triggerError ? " error.on.schema.change: true\n" : "\n")
+ + "transform:\n"
+ + " - source-table: %s.\\.*\n"
+ + " projection: CAST(id AS VARCHAR) || ' -> ' || name AS uid, *, id * id AS id_square, 'age < 20' as tag\n"
+ + " filter: age < 20\n"
+ + " - source-table: %s.\\.*\n"
+ + " projection: CAST(id AS VARCHAR) || ' -> ' || name AS uid, *, 0 - id * id AS id_square, 'age >= 20' as tag\n"
+ + " filter: age >= 20\n"
+ + (mergeTable
+ ? String.format(
+ "route:\n"
+ + " - source-table: %s.(members|new_members)\n"
+ + " sink-table: %s.merged",
+ dbName, dbName)
+ : "")
+ + "\n"
+ + "pipeline:\n"
+ + " schema.change.behavior: %s\n"
+ + " parallelism: 1",
+ INTER_CONTAINER_MYSQL_ALIAS,
+ MYSQL_TEST_USER,
+ MYSQL_TEST_PASSWORD,
+ dbName,
+ mergeTable ? "(members|new_members)" : "members",
+ dbName,
+ dbName,
+ behavior);
+ Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
+ Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
+ Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
+ submitPipelineJob(pipelineJob, mysqlCdcJar, valuesCdcJar, mysqlDriverJar);
+ waitUntilJobRunning(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+ validateSnapshotData(dbName, mergeTable ? "merged" : "members");
+
+ LOG.info("Starting schema evolution");
+ String mysqlJdbcUrl =
+ String.format(
+ "jdbc:mysql://%s:%s/%s", MYSQL.getHost(), MYSQL.getDatabasePort(), dbName);
+
+ try (Connection conn =
+ DriverManager.getConnection(
+ mysqlJdbcUrl, MYSQL_TEST_USER, MYSQL_TEST_PASSWORD);
+ Statement stmt = conn.createStatement()) {
+
+ waitForIncrementalStage(dbName, mergeTable ? "merged" : "members", stmt);
+
+ // triggers AddColumnEvent
+ stmt.execute("ALTER TABLE members ADD COLUMN gender TINYINT AFTER age;");
+ stmt.execute("INSERT INTO members VALUES (1012, 'Eve', 17, 0);");
+
+ // triggers AlterColumnTypeEvent
+ stmt.execute("ALTER TABLE members MODIFY COLUMN age DOUBLE;");
+
+ // triggers RenameColumnEvent
+ stmt.execute("ALTER TABLE members RENAME COLUMN gender TO biological_sex;");
+
+ // triggers DropColumnEvent
+ stmt.execute("ALTER TABLE members DROP COLUMN biological_sex");
+ stmt.execute("INSERT INTO members VALUES (1013, 'Fiona', 16);");
+
+ // triggers TruncateTableEvent
+ stmt.execute("TRUNCATE TABLE members;");
+ stmt.execute("INSERT INTO members VALUES (1014, 'Gem', 17);");
+
+ // triggers DropTableEvent
+ stmt.execute("DROP TABLE members;");
+ }
+
+ List<String> expectedTmEvents =
+ expectedTaskManagerEvents.stream()
+ .map(s -> String.format(s, dbName, dbName))
+ .collect(Collectors.toList());
+
+ validateResult(expectedTmEvents, taskManagerConsumer);
+
+ List<String> expectedJmEvents =
+ expectedJobManagerEvents.stream()
+ .map(s -> String.format(s, dbName, dbName, dbName))
+ .collect(Collectors.toList());
+
+ validateResult(expectedJmEvents, jobManagerConsumer);
+ }
+
+ private void validateSnapshotData(String dbName, String tableName) throws Exception {
+ List<String> expected =
+ Stream.of(
+ "CreateTableEvent{tableId=%s.%s, schema=columns={`uid` STRING,`id` INT NOT NULL,`name` VARCHAR(17),`age` INT,`id_square` INT,`tag` STRING}, primaryKeys=id, options=()}",
+ "DataChangeEvent{tableId=%s.%s, before=[], after=[1009 -> Bob, 1009, Bob, 20, -1018081, age >= 20], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.%s, before=[], after=[1008 -> Alice, 1008, Alice, 21, -1016064, age >= 20], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.%s, before=[], after=[1011 -> Derrida, 1011, Derrida, 18, 1022121, age < 20], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.%s, before=[], after=[1010 -> Carol, 1010, Carol, 19, 1020100, age < 20], op=INSERT, meta=()}")
+ .map(s -> String.format(s, dbName, tableName))
+ .collect(Collectors.toList());
+
+ validateResult(expected, taskManagerConsumer);
+ }
+
+ private void waitForIncrementalStage(String dbName, String tableName, Statement stmt)
+ throws Exception {
+ stmt.execute("INSERT INTO members VALUES (0, '__fence__', 0);");
+
+ // Ensure we change schema after incremental stage
+ waitUntilSpecificEvent(
+ String.format(
+ "DataChangeEvent{tableId=%s.%s, before=[], after=[0 -> __fence__, 0, __fence__, 0, 0, age < 20], op=INSERT, meta=()}",
+ dbName, tableName),
+ taskManagerConsumer);
+ }
+
+ private void validateResult(List<String> expectedEvents, ToStringConsumer consumer)
+ throws Exception {
+ for (String event : expectedEvents) {
+ waitUntilSpecificEvent(event, consumer);
+ }
+ }
+
+ private void waitUntilSpecificEvent(String event, ToStringConsumer consumer) throws Exception {
+ boolean result = false;
+ long endTimeout =
+ System.currentTimeMillis() + SchemaEvolvingTransformE2eITCase.EVENT_WAITING_TIMEOUT;
+ while (System.currentTimeMillis() < endTimeout) {
+ String stdout = consumer.toUtf8String();
+ if (stdout.contains(event + "\n")) {
+ result = true;
+ break;
+ }
+ Thread.sleep(1000);
+ }
+ if (!result) {
+ throw new TimeoutException(
+ "failed to get specific event: "
+ + event
+ + " from stdout: "
+ + consumer.toUtf8String());
+ }
+ }
+}
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
index ae516fc06..9c6130359 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
@@ -677,21 +677,6 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
"DataChangeEvent{tableId=%s.TABLEALPHA, before=[1009, 108.1, 18 - Bob], after=[1009, 200.0, 18 - Bob], op=UPDATE, meta=()}",
"DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3007, 107.0, 16 - IINA], op=INSERT, meta=()}",
"DataChangeEvent{tableId=%s.TABLEBETA, before=[2011, 111.0, 21 - Eva], after=[], op=DELETE, meta=()}");
-
- String stdout = taskManagerConsumer.toUtf8String();
- System.out.println(stdout);
- }
-
- private void validateEvents(String... expectedEvents) throws Exception {
- for (String event : expectedEvents) {
- waitUntilSpecificEvent(
- String.format(
- event,
- transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName()),
- 20000L);
- }
}
@Test
@@ -733,6 +718,327 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
waitForTemporaryRecords(8, 60000L);
}
+ @Test
+ public void testTransformWithSchemaEvolution() throws Exception {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: mysql\n"
+ + " hostname: %s\n"
+ + " port: 3306\n"
+ + " username: %s\n"
+ + " password: %s\n"
+ + " tables: %s.TABLEALPHA\n"
+ + " server-id: 5400-5404\n"
+ + " server-time-zone: UTC\n"
+ + "sink:\n"
+ + " type: values\n"
+ + "transform:\n"
+ + " - source-table: %s.TABLEALPHA\n"
+ + " projection: ID, 'id -> ' || ID AS UID, PRICEALPHA AS PRICE\n"
+ + " filter: ID > 1008\n"
+ + "pipeline:\n"
+ + " parallelism: 1",
+ INTER_CONTAINER_MYSQL_ALIAS,
+ MYSQL_TEST_USER,
+ MYSQL_TEST_PASSWORD,
+ transformTestDatabase.getDatabaseName(),
+ transformTestDatabase.getDatabaseName());
+ Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
+ Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
+ Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
+ submitPipelineJob(pipelineJob, mysqlCdcJar, valuesCdcJar, mysqlDriverJar);
+ waitUntilJobRunning(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+
+ waitUntilSpecificEvent(
+ String.format(
+ "CreateTableEvent{tableId=%s.TABLEALPHA, schema=columns={`ID` INT NOT NULL,`UID` STRING,`PRICE` INT}, primaryKeys=ID, options=()}",
+ transformTestDatabase.getDatabaseName()),
+ 60000L);
+
+ validateEvents(
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[1009, id -> 1009, 0], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[1010, id -> 1010, 99], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[1011, id -> 1011, 59], op=INSERT, meta=()}");
+
+ LOG.info("Begin incremental reading stage.");
+ // generate binlogs
+ String mysqlJdbcUrl =
+ String.format(
+ "jdbc:mysql://%s:%s/%s",
+ MYSQL.getHost(),
+ MYSQL.getDatabasePort(),
+ transformTestDatabase.getDatabaseName());
+ try (Connection conn =
+ DriverManager.getConnection(
+ mysqlJdbcUrl, MYSQL_TEST_USER, MYSQL_TEST_PASSWORD);
+ Statement stat = conn.createStatement()) {
+ stat.execute("UPDATE TABLEALPHA SET VERSION='100' WHERE id=1009;");
+ stat.execute("INSERT INTO TABLEALPHA VALUES (3007, '7', 79, 16, 'IINA');");
+ stat.execute("DELETE FROM TABLEBETA WHERE id=2011;");
+ } catch (SQLException e) {
+ LOG.error("Update table for CDC failed.", e);
+ throw e;
+ }
+
+ validateEvents(
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[1009, id -> 1009, 0], after=[1009, id -> 1009, 0], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3007, id -> 3007, 79], op=INSERT, meta=()}");
+
+ LOG.info("Start schema evolution.");
+ try (Connection conn =
+ DriverManager.getConnection(
+ mysqlJdbcUrl, MYSQL_TEST_USER, MYSQL_TEST_PASSWORD);
+ Statement stmt = conn.createStatement()) {
+
+ // triggers AddColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA ADD COLUMN CODENAME TINYINT AFTER VERSION;");
+ stmt.execute("ALTER TABLE TABLEALPHA ADD COLUMN LAST VARCHAR(17);");
+ stmt.execute("INSERT INTO TABLEALPHA VALUES (3008, '8', 8, 80, 17, 'Jazz', 'Last');");
+
+ // triggers AlterColumnTypeEvent and RenameColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA CHANGE COLUMN CODENAME CODE_NAME DOUBLE;");
+
+ // triggers RenameColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA RENAME COLUMN CODE_NAME TO CODE_NAME_EX;");
+ stmt.execute("INSERT INTO TABLEALPHA VALUES (3009, '9', 9, 90, 18, 'Keka', 'Finale');");
+
+ // triggers DropColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA DROP COLUMN CODE_NAME_EX");
+ stmt.execute("INSERT INTO TABLEALPHA VALUES (3010, '10', 10, 97, 19, 'Lynx');");
+ }
+
+ validateEvents(
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3008, id -> 3008, 80], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3009, id -> 3009, 90], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3010, id -> 3010, 10], op=INSERT, meta=()}");
+ }
+
+ @Test
+ public void testTransformWildcardPrefixWithSchemaEvolution() throws Exception {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: mysql\n"
+ + " hostname: %s\n"
+ + " port: 3306\n"
+ + " username: %s\n"
+ + " password: %s\n"
+ + " tables: %s.TABLEALPHA\n"
+ + " server-id: 5400-5404\n"
+ + " server-time-zone: UTC\n"
+ + "sink:\n"
+ + " type: values\n"
+ + "transform:\n"
+ + " - source-table: %s.TABLEALPHA\n"
+ + " projection: \\*, 'id -> ' || ID AS UID\n"
+ + " filter: ID > 1008\n"
+ + "pipeline:\n"
+ + " parallelism: 1",
+ INTER_CONTAINER_MYSQL_ALIAS,
+ MYSQL_TEST_USER,
+ MYSQL_TEST_PASSWORD,
+ transformTestDatabase.getDatabaseName(),
+ transformTestDatabase.getDatabaseName());
+ Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
+ Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
+ Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
+ submitPipelineJob(pipelineJob, mysqlCdcJar, valuesCdcJar, mysqlDriverJar);
+ waitUntilJobRunning(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+
+ waitUntilSpecificEvent(
+ String.format(
+ "CreateTableEvent{tableId=%s.TABLEALPHA, schema=columns={`ID` INT NOT NULL,`VERSION` VARCHAR(17),`PRICEALPHA` INT,`AGEALPHA` INT,`NAMEALPHA` VARCHAR(128),`UID` STRING}, primaryKeys=ID, options=()}",
+ transformTestDatabase.getDatabaseName()),
+ 60000L);
+
+ validateEvents(
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[1009, 8.1, 0, 18, Bob, id -> 1009], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[1010, 10, 99, 19, Carol, id -> 1010], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[1011, 11, 59, 20, Dave, id -> 1011], op=INSERT, meta=()}");
+
+ LOG.info("Begin incremental reading stage.");
+ // generate binlogs
+ String mysqlJdbcUrl =
+ String.format(
+ "jdbc:mysql://%s:%s/%s",
+ MYSQL.getHost(),
+ MYSQL.getDatabasePort(),
+ transformTestDatabase.getDatabaseName());
+ try (Connection conn =
+ DriverManager.getConnection(
+ mysqlJdbcUrl, MYSQL_TEST_USER, MYSQL_TEST_PASSWORD);
+ Statement stat = conn.createStatement()) {
+ stat.execute("UPDATE TABLEALPHA SET VERSION='100' WHERE id=1009;");
+ stat.execute("INSERT INTO TABLEALPHA VALUES (3007, '7', 79, 16, 'IINA');");
+ stat.execute("DELETE FROM TABLEBETA WHERE id=2011;");
+ } catch (SQLException e) {
+ LOG.error("Update table for CDC failed.", e);
+ throw e;
+ }
+
+ validateEvents(
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[1009, 8.1, 0, 18, Bob, id -> 1009], after=[1009, 100, 0, 18, Bob, id -> 1009], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3007, 7, 79, 16, IINA, id -> 3007], op=INSERT, meta=()}");
+
+ LOG.info("Start schema evolution.");
+ try (Connection conn =
+ DriverManager.getConnection(
+ mysqlJdbcUrl, MYSQL_TEST_USER, MYSQL_TEST_PASSWORD);
+ Statement stmt = conn.createStatement()) {
+
+ // triggers AddColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA ADD COLUMN CODENAME TINYINT AFTER VERSION;");
+ stmt.execute("ALTER TABLE TABLEALPHA ADD COLUMN LAST VARCHAR(17);");
+ stmt.execute("INSERT INTO TABLEALPHA VALUES (3008, '8', 8, 80, 17, 'Jazz', 'Last');");
+
+ // triggers AlterColumnTypeEvent and RenameColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA CHANGE COLUMN CODENAME CODE_NAME DOUBLE;");
+
+ // triggers RenameColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA RENAME COLUMN CODE_NAME TO CODE_NAME_EX;");
+ stmt.execute("INSERT INTO TABLEALPHA VALUES (3009, '9', 9, 90, 18, 'Keka', 'Finale');");
+
+ // triggers DropColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA DROP COLUMN CODE_NAME_EX");
+ stmt.execute("INSERT INTO TABLEALPHA VALUES (3010, '10', 10, 97, 19, 'Lynx');");
+ } catch (SQLException e) {
+ LOG.error("Update table for CDC failed.", e);
+ throw e;
+ }
+
+ validateEvents(
+ "AddColumnEvent{tableId=%s.TABLEALPHA, addedColumns=[ColumnWithPosition{column=`LAST` VARCHAR(17), position=AFTER, existedColumnName=NAMEALPHA}]}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3008, 8, 8, 80, 17, Jazz, Last, id -> 3008], op=INSERT, meta=()}",
+ "AlterColumnTypeEvent{tableId=%s.TABLEALPHA, typeMapping={CODENAME=DOUBLE}, oldTypeMapping={CODENAME=TINYINT}}",
+ "RenameColumnEvent{tableId=%s.TABLEALPHA, nameMapping={CODENAME=CODE_NAME}}",
+ "RenameColumnEvent{tableId=%s.TABLEALPHA, nameMapping={CODE_NAME=CODE_NAME_EX}}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3009, 9, 9.0, 90, 18, Keka, Finale, id -> 3009], op=INSERT, meta=()}",
+ "DropColumnEvent{tableId=%s.TABLEALPHA, droppedColumnNames=[CODE_NAME_EX]}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3010, 10, 10, 97, 19, Lynx, id -> 3010], op=INSERT, meta=()}");
+ }
+
+ @Test
+ public void testTransformWildcardSuffixWithSchemaEvolution() throws Exception {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: mysql\n"
+ + " hostname: %s\n"
+ + " port: 3306\n"
+ + " username: %s\n"
+ + " password: %s\n"
+ + " tables: %s.TABLEALPHA\n"
+ + " server-id: 5400-5404\n"
+ + " server-time-zone: UTC\n"
+ + "sink:\n"
+ + " type: values\n"
+ + "transform:\n"
+ + " - source-table: %s.TABLEALPHA\n"
+ + " projection: ID || ' <- id' AS UID, *\n"
+ + " filter: ID > 1008\n"
+ + "pipeline:\n"
+ + " parallelism: 1",
+ INTER_CONTAINER_MYSQL_ALIAS,
+ MYSQL_TEST_USER,
+ MYSQL_TEST_PASSWORD,
+ transformTestDatabase.getDatabaseName(),
+ transformTestDatabase.getDatabaseName());
+ Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
+ Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
+ Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
+ submitPipelineJob(pipelineJob, mysqlCdcJar, valuesCdcJar, mysqlDriverJar);
+ waitUntilJobRunning(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+
+ waitUntilSpecificEvent(
+ String.format(
+ "CreateTableEvent{tableId=%s.TABLEALPHA, schema=columns={`UID` STRING,`ID` INT NOT NULL,`VERSION` VARCHAR(17),`PRICEALPHA` INT,`AGEALPHA` INT,`NAMEALPHA` VARCHAR(128)}, primaryKeys=ID, options=()}",
+ transformTestDatabase.getDatabaseName()),
+ 60000L);
+
+ validateEvents(
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[1009 <- id, 1009, 8.1, 0, 18, Bob], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[1010 <- id, 1010, 10, 99, 19, Carol], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[1011 <- id, 1011, 11, 59, 20, Dave], op=INSERT, meta=()}");
+
+ LOG.info("Begin incremental reading stage.");
+ // generate binlogs
+ String mysqlJdbcUrl =
+ String.format(
+ "jdbc:mysql://%s:%s/%s",
+ MYSQL.getHost(),
+ MYSQL.getDatabasePort(),
+ transformTestDatabase.getDatabaseName());
+ try (Connection conn =
+ DriverManager.getConnection(
+ mysqlJdbcUrl, MYSQL_TEST_USER, MYSQL_TEST_PASSWORD);
+ Statement stat = conn.createStatement()) {
+ stat.execute("UPDATE TABLEALPHA SET VERSION='100' WHERE id=1009;");
+ stat.execute("INSERT INTO TABLEALPHA VALUES (3007, '7', 79, 16, 'IINA');");
+ stat.execute("DELETE FROM TABLEBETA WHERE id=2011;");
+ } catch (SQLException e) {
+ LOG.error("Update table for CDC failed.", e);
+ throw e;
+ }
+
+ validateEvents(
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[1009 <- id, 1009, 8.1, 0, 18, Bob], after=[1009 <- id, 1009, 100, 0, 18, Bob], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3007 <- id, 3007, 7, 79, 16, IINA], op=INSERT, meta=()}");
+
+ LOG.info("Start schema evolution.");
+ try (Connection conn =
+ DriverManager.getConnection(
+ mysqlJdbcUrl, MYSQL_TEST_USER, MYSQL_TEST_PASSWORD);
+ Statement stmt = conn.createStatement()) {
+
+ // triggers AddColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA ADD COLUMN CODENAME TINYINT AFTER VERSION;");
+ stmt.execute("ALTER TABLE TABLEALPHA ADD COLUMN FIRST VARCHAR(17) FIRST;");
+ stmt.execute("INSERT INTO TABLEALPHA VALUES ('First', 3008, '8', 8, 80, 17, 'Jazz');");
+
+ // triggers AlterColumnTypeEvent and RenameColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA CHANGE COLUMN CODENAME CODE_NAME DOUBLE;");
+
+ // triggers RenameColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA RENAME COLUMN CODE_NAME TO CODE_NAME_EX;");
+ stmt.execute("INSERT INTO TABLEALPHA VALUES ('1st', 3009, '9', 9, 90, 18, 'Keka');");
+
+ // triggers DropColumnEvent
+ stmt.execute("ALTER TABLE TABLEALPHA DROP COLUMN CODE_NAME_EX");
+ stmt.execute(
+ "INSERT INTO TABLEALPHA VALUES ('Beginning', 3010, '10', 10, 97, 'Lemon');");
+ } catch (SQLException e) {
+ LOG.error("Update table for CDC failed.", e);
+ throw e;
+ }
+
+ validateEvents(
+ "AddColumnEvent{tableId=%s.TABLEALPHA, addedColumns=[ColumnWithPosition{column=`CODENAME` TINYINT, position=AFTER, existedColumnName=VERSION}]}",
+ "AddColumnEvent{tableId=%s.TABLEALPHA, addedColumns=[ColumnWithPosition{column=`FIRST` VARCHAR(17), position=BEFORE, existedColumnName=ID}]}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3008 <- id, First, 3008, 8, 8, 80, 17, Jazz], op=INSERT, meta=()}",
+ "AlterColumnTypeEvent{tableId=%s.TABLEALPHA, typeMapping={CODENAME=DOUBLE}, oldTypeMapping={CODENAME=TINYINT}}",
+ "RenameColumnEvent{tableId=%s.TABLEALPHA, nameMapping={CODENAME=CODE_NAME}}",
+ "RenameColumnEvent{tableId=%s.TABLEALPHA, nameMapping={CODE_NAME=CODE_NAME_EX}}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3009 <- id, 1st, 3009, 9, 9.0, 90, 18, Keka], op=INSERT, meta=()}",
+ "DropColumnEvent{tableId=%s.TABLEALPHA, droppedColumnNames=[CODE_NAME_EX]}",
+ "DataChangeEvent{tableId=%s.TABLEALPHA, before=[], after=[3010 <- id, Beginning, 3010, 10, 10, 97, Lemon], op=INSERT, meta=()}");
+ }
+
+ private void validateEvents(String... expectedEvents) throws Exception {
+ for (String event : expectedEvents) {
+ waitUntilSpecificEvent(
+ String.format(
+ event,
+ transformTestDatabase.getDatabaseName(),
+ transformTestDatabase.getDatabaseName(),
+ transformTestDatabase.getDatabaseName()),
+ 20000L);
+ }
+ }
+
private void validateResult(List<String> expectedEvents) throws Exception {
for (String event : expectedEvents) {
waitUntilSpecificEvent(event, 6000L);
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperator.java
index 36955f8e1..30c78d202 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperator.java
@@ -30,6 +30,7 @@ import org.apache.flink.cdc.common.pipeline.PipelineOptions;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.schema.Selectors;
import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.runtime.parser.TransformParser;
import org.apache.flink.streaming.api.graph.StreamConfig;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
@@ -43,10 +44,12 @@ import java.io.Serializable;
import java.lang.reflect.InvocationTargetException;
import java.time.ZoneId;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
+import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;
@@ -74,6 +77,8 @@ public class PostTransformOperator extends AbstractStreamOperator<Event>
transformProjectionProcessorMap;
private transient Map<Tuple2<TableId, TransformFilter>, TransformFilterProcessor>
transformFilterProcessorMap;
+ private final Map<TableId, Boolean> hasAsteriskMap;
+ private final Map<TableId, List<String>> projectedColumnsMap;
public static PostTransformOperator.Builder newBuilder() {
return new PostTransformOperator.Builder();
@@ -140,6 +145,8 @@ public class PostTransformOperator extends AbstractStreamOperator<Event>
this.transformProjectionProcessorMap = new ConcurrentHashMap<>();
this.udfFunctions = udfFunctions;
this.udfFunctionInstances = new ConcurrentHashMap<>();
+ this.hasAsteriskMap = new HashMap<>();
+ this.projectedColumnsMap = new HashMap<>();
}
@Override
@@ -220,8 +227,10 @@ public class PostTransformOperator extends AbstractStreamOperator<Event>
transformProjectionProcessorMap
.keySet()
.removeIf(e -> Objects.equals(e.f0, schemaChangeEvent.tableId()));
- event = cacheSchema(schemaChangeEvent);
- output.collect(new StreamRecord<>(event));
+ transformFilterProcessorMap
+ .keySet()
+ .removeIf(e -> Objects.equals(e.f0, schemaChangeEvent.tableId()));
+ cacheSchema(schemaChangeEvent).ifPresent(e -> output.collect(new StreamRecord<>(e)));
} else if (event instanceof DataChangeEvent) {
Optional<DataChangeEvent> dataChangeEventOptional =
processDataChangeEvent(((DataChangeEvent) event));
@@ -231,8 +240,42 @@ public class PostTransformOperator extends AbstractStreamOperator<Event>
}
}
- private SchemaChangeEvent cacheSchema(SchemaChangeEvent event) throws Exception {
+ private Optional<SchemaChangeEvent> cacheSchema(SchemaChangeEvent event) throws Exception {
TableId tableId = event.tableId();
+ if (event instanceof CreateTableEvent) {
+ CreateTableEvent createTableEvent = (CreateTableEvent) event;
+ Set<String> projectedColumnsSet =
+ transforms.stream()
+ .filter(t -> t.getSelectors().isMatch(tableId))
+ .flatMap(
+ rule ->
+ TransformParser.generateProjectionColumns(
+ rule.getProjection()
+ .map(TransformProjection::getProjection)
+ .orElse(null),
+ createTableEvent.getSchema().getColumns(),
+ udfDescriptors)
+ .stream())
+ .map(ProjectionColumn::getColumnName)
+ .collect(Collectors.toSet());
+ boolean hasAsterisk =
+ transforms.stream()
+ .filter(t -> t.getSelectors().isMatch(tableId))
+ .anyMatch(
+ t ->
+ TransformParser.hasAsterisk(
+ t.getProjection()
+ .map(TransformProjection::getProjection)
+ .orElse(null)));
+
+ hasAsteriskMap.put(tableId, hasAsterisk);
+ projectedColumnsMap.put(
+ tableId,
+ createTableEvent.getSchema().getColumnNames().stream()
+ .filter(projectedColumnsSet::contains)
+ .collect(Collectors.toList()));
+ }
+
Schema schema;
if (event instanceof CreateTableEvent) {
CreateTableEvent createTableEvent = (CreateTableEvent) event;
@@ -248,9 +291,11 @@ public class PostTransformOperator extends AbstractStreamOperator<Event>
tableId, PostTransformChangeInfo.of(tableId, projectedSchema, schema));
if (event instanceof CreateTableEvent) {
- return new CreateTableEvent(event.tableId(), projectedSchema);
+ return Optional.of(new CreateTableEvent(tableId, projectedSchema));
+ } else {
+ return SchemaUtils.transformSchemaChangeEvent(
+ hasAsteriskMap.get(tableId), projectedColumnsMap.get(tableId), event);
}
- return event;
}
private PostTransformChangeInfo getPostTransformChangeInfo(TableId tableId) {
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java
index b8dc2448e..4da0f0b99 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperator.java
@@ -27,9 +27,11 @@ import org.apache.flink.cdc.common.event.DataChangeEvent;
import org.apache.flink.cdc.common.event.Event;
import org.apache.flink.cdc.common.event.SchemaChangeEvent;
import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.schema.Column;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.schema.Selectors;
import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.runtime.parser.TransformParser;
import org.apache.flink.runtime.state.StateInitializationContext;
import org.apache.flink.runtime.state.StateSnapshotContext;
import org.apache.flink.streaming.api.graph.StreamConfig;
@@ -47,6 +49,8 @@ import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;
@@ -66,6 +70,8 @@ public class PreTransformOperator extends AbstractStreamOperator<Event>
private final List<Tuple2<String, String>> udfFunctions;
private List<UserDefinedFunctionDescriptor> udfDescriptors;
private Map<TableId, PreTransformProcessor> preTransformProcessorMap;
+ private Map<TableId, Boolean> hasAsteriskMap;
+ private Map<TableId, List<String>> referencedColumnsMap;
public static PreTransformOperator.Builder newBuilder() {
return new PreTransformOperator.Builder();
@@ -132,12 +138,10 @@ public class PreTransformOperator extends AbstractStreamOperator<Event>
this.udfFunctions.stream()
.map(udf -> new UserDefinedFunctionDescriptor(udf.f0, udf.f1))
.collect(Collectors.toList());
- }
- @Override
- public void open() throws Exception {
- super.open();
- transforms = new ArrayList<>();
+ // Initialize data fields in advance because they might be accessed in
+ // `::initializeState` function when restoring from a previous state.
+ this.transforms = new ArrayList<>();
for (TransformRule transformRule : transformRules) {
String tableInclusions = transformRule.getTableInclusions();
String projection = transformRule.getProjection();
@@ -158,6 +162,8 @@ public class PreTransformOperator extends AbstractStreamOperator<Event>
new SchemaMetadataTransform(primaryKeys, partitionKeys, tableOptions)));
}
this.preTransformProcessorMap = new ConcurrentHashMap<>();
+ this.hasAsteriskMap = new ConcurrentHashMap<>();
+ this.referencedColumnsMap = new ConcurrentHashMap<>();
}
@Override
@@ -175,13 +181,18 @@ public class PreTransformOperator extends AbstractStreamOperator<Event>
serializedTableInfo);
preTransformChangeInfoMap.put(
stateTableChangeInfo.getTableId(), stateTableChangeInfo);
+
+ CreateTableEvent restoredCreateTableEvent =
+ new CreateTableEvent(
+ stateTableChangeInfo.getTableId(),
+ stateTableChangeInfo.getPreTransformedSchema());
+ // hasAsteriskMap and referencedColumnsMap needs to be recalculated after restoring
+ // from a checkpoint.
+ cacheTransformRuleInfo(restoredCreateTableEvent);
+
// Since PostTransformOperator doesn't preserve state, pre-transformed schema
// information needs to be passed by PreTransformOperator.
- output.collect(
- new StreamRecord<>(
- new CreateTableEvent(
- stateTableChangeInfo.getTableId(),
- stateTableChangeInfo.getPreTransformedSchema())));
+ output.collect(new StreamRecord<>(restoredCreateTableEvent));
}
}
}
@@ -223,16 +234,14 @@ public class PreTransformOperator extends AbstractStreamOperator<Event>
if (event instanceof CreateTableEvent) {
CreateTableEvent createTableEvent = (CreateTableEvent) event;
preTransformProcessorMap.remove(createTableEvent.tableId());
- event = cacheCreateTable(createTableEvent);
- output.collect(new StreamRecord<>(event));
+ output.collect(new StreamRecord<>(cacheCreateTable(createTableEvent)));
} else if (event instanceof SchemaChangeEvent) {
SchemaChangeEvent schemaChangeEvent = (SchemaChangeEvent) event;
preTransformProcessorMap.remove(schemaChangeEvent.tableId());
- event = cacheChangeSchema(schemaChangeEvent);
- output.collect(new StreamRecord<>(event));
+ cacheChangeSchema(schemaChangeEvent)
+ .ifPresent(e -> output.collect(new StreamRecord<>(e)));
} else if (event instanceof DataChangeEvent) {
- DataChangeEvent dataChangeEvent = processDataChangeEvent(((DataChangeEvent) event));
- output.collect(new StreamRecord<>(dataChangeEvent));
+ output.collect(new StreamRecord<>(processDataChangeEvent(((DataChangeEvent) event))));
}
}
@@ -246,23 +255,66 @@ public class PreTransformOperator extends AbstractStreamOperator<Event>
return event;
}
- private SchemaChangeEvent cacheChangeSchema(SchemaChangeEvent event) {
+ private Optional<SchemaChangeEvent> cacheChangeSchema(SchemaChangeEvent event) {
TableId tableId = event.tableId();
PreTransformChangeInfo tableChangeInfo = preTransformChangeInfoMap.get(tableId);
Schema originalSchema =
SchemaUtils.applySchemaChangeEvent(tableChangeInfo.getSourceSchema(), event);
- Schema newSchema =
- SchemaUtils.applySchemaChangeEvent(
- tableChangeInfo.getPreTransformedSchema(), event);
+ Schema preTransformedSchema = tableChangeInfo.getPreTransformedSchema();
+ Optional<SchemaChangeEvent> schemaChangeEvent =
+ SchemaUtils.transformSchemaChangeEvent(
+ hasAsteriskMap.get(tableId), referencedColumnsMap.get(tableId), event);
+ if (schemaChangeEvent.isPresent()) {
+ preTransformedSchema =
+ SchemaUtils.applySchemaChangeEvent(
+ tableChangeInfo.getPreTransformedSchema(), schemaChangeEvent.get());
+ }
preTransformChangeInfoMap.put(
- tableId, PreTransformChangeInfo.of(tableId, originalSchema, newSchema));
- return event;
+ tableId, PreTransformChangeInfo.of(tableId, originalSchema, preTransformedSchema));
+ return schemaChangeEvent;
+ }
+
+ private void cacheTransformRuleInfo(CreateTableEvent createTableEvent) {
+ TableId tableId = createTableEvent.tableId();
+ Set<String> referencedColumnsSet =
+ transforms.stream()
+ .filter(t -> t.getSelectors().isMatch(tableId))
+ .flatMap(
+ rule ->
+ TransformParser.generateReferencedColumns(
+ rule.getProjection()
+ .map(TransformProjection::getProjection)
+ .orElse(null),
+ rule.getFilter()
+ .map(TransformFilter::getExpression)
+ .orElse(null),
+ createTableEvent.getSchema().getColumns())
+ .stream())
+ .map(Column::getName)
+ .collect(Collectors.toSet());
+
+ boolean hasAsterisk =
+ transforms.stream()
+ .filter(t -> t.getSelectors().isMatch(tableId))
+ .anyMatch(
+ t ->
+ TransformParser.hasAsterisk(
+ t.getProjection()
+ .map(TransformProjection::getProjection)
+ .orElse(null)));
+
+ hasAsteriskMap.put(createTableEvent.tableId(), hasAsterisk);
+ referencedColumnsMap.put(
+ createTableEvent.tableId(),
+ createTableEvent.getSchema().getColumnNames().stream()
+ .filter(referencedColumnsSet::contains)
+ .collect(Collectors.toList()));
}
private CreateTableEvent transformCreateTableEvent(CreateTableEvent createTableEvent) {
TableId tableId = createTableEvent.tableId();
PreTransformChangeInfo tableChangeInfo = preTransformChangeInfoMap.get(tableId);
-
+ cacheTransformRuleInfo(createTableEvent);
for (Tuple2<Selectors, SchemaMetadataTransform> transform : schemaMetadataTransformers) {
Selectors selectors = transform.f0;
if (selectors.isMatch(tableId)) {
@@ -288,7 +340,8 @@ public class PreTransformOperator extends AbstractStreamOperator<Event>
}
PreTransformProcessor preTransformProcessor =
preTransformProcessorMap.get(tableId);
- // filter out unreferenced columns in pre-transform process
+ // TODO: Currently this code wrongly filters out rows that weren't referenced in
+ // the first matching transform rule but in the following transform rules.
return preTransformProcessor.preTransformCreateTableEvent(createTableEvent);
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java
index c7a1b718e..38121f35d 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java
@@ -585,4 +585,26 @@ public class TransformParser {
return filter;
}
}
+
+ public static boolean hasAsterisk(@Nullable String projection) {
+ if (isNullOrWhitespaceOnly(projection)) {
+ return false;
+ }
+ return parseProjectionExpression(projection).getOperandList().stream()
+ .anyMatch(TransformParser::hasAsterisk);
+ }
+
+ private static boolean hasAsterisk(SqlNode sqlNode) {
+ if (sqlNode instanceof SqlIdentifier) {
+ return ((SqlIdentifier) sqlNode).isStar();
+ } else if (sqlNode instanceof SqlBasicCall) {
+ SqlBasicCall sqlBasicCall = (SqlBasicCall) sqlNode;
+ return sqlBasicCall.getOperandList().stream().anyMatch(TransformParser::hasAsterisk);
+ } else if (sqlNode instanceof SqlNodeList) {
+ SqlNodeList sqlNodeList = (SqlNodeList) sqlNode;
+ return sqlNodeList.getList().stream().anyMatch(TransformParser::hasAsterisk);
+ } else {
+ return false;
+ }
+ }
}
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperatorTest.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperatorTest.java
index c447d51c2..cc257401e 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperatorTest.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/transform/PreTransformOperatorTest.java
@@ -254,7 +254,16 @@ public class PreTransformOperatorTest {
transform.processElement(new StreamRecord<>(addColumnEvent));
Assertions.assertThat(
transformFunctionEventEventOperatorTestHarness.getOutputRecords().poll())
- .isEqualTo(new StreamRecord<>(addColumnEvent));
+ .isEqualTo(
+ new StreamRecord<>(
+ new AddColumnEvent(
+ CUSTOMERS_TABLEID,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "col3", DataTypes.STRING()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "col2")))));
transform.processElement(new StreamRecord<>(insertEvent));
Assertions.assertThat(
transformFunctionEventEventOperatorTestHarness.getOutputRecords().poll())
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/transform/TransformOperatorWithSchemaEvolveTest.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/transform/TransformOperatorWithSchemaEvolveTest.java
new file mode 100644
index 000000000..7613c930b
--- /dev/null
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/transform/TransformOperatorWithSchemaEvolveTest.java
@@ -0,0 +1,792 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.runtime.operators.transform;
+
+import org.apache.flink.cdc.common.data.binary.BinaryStringData;
+import org.apache.flink.cdc.common.event.AddColumnEvent;
+import org.apache.flink.cdc.common.event.AlterColumnTypeEvent;
+import org.apache.flink.cdc.common.event.CreateTableEvent;
+import org.apache.flink.cdc.common.event.DropColumnEvent;
+import org.apache.flink.cdc.common.event.Event;
+import org.apache.flink.cdc.common.event.RenameColumnEvent;
+import org.apache.flink.cdc.common.event.SchemaChangeEvent;
+import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.schema.Column;
+import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.types.DataTypes;
+import org.apache.flink.cdc.runtime.testutils.operators.EventOperatorTestHarness;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
+
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * Unit tests for the {@link PreTransformOperator} and {@link PostTransformOperator} handling schema
+ * evolution events.
+ */
+public class TransformOperatorWithSchemaEvolveTest {
+
+ /** Defines a unified transform test cases. */
+ static class TransformWithSchemaEvolveTestCase {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TransformWithSchemaEvolveTestCase.class);
+
+ private final TableId tableId;
+ private final String projectionExpression;
+ private final String filterExpression;
+
+ private Schema sourceSchema;
+ private Schema preTransformedSchema;
+ private Schema postTransformedSchema;
+
+ private final List<Event> sourceEvents;
+ private final List<Event> preTransformedEvents;
+ private final List<Event> postTransformedEvents;
+
+ private PreTransformOperator preTransformOperator;
+ private PostTransformOperator postTransformOperator;
+
+ private EventOperatorTestHarness<PreTransformOperator, Event> preTransformOperatorHarness;
+ private EventOperatorTestHarness<PostTransformOperator, Event> postTransformOperatorHarness;
+
+ public static TransformWithSchemaEvolveTestCase of(
+ TableId tableId,
+ String projectionExpression,
+ String filterExpression,
+ Schema sourceSchema,
+ Schema preTransformedSchema,
+ Schema postTransformedSchema) {
+ return new TransformWithSchemaEvolveTestCase(
+ tableId,
+ projectionExpression,
+ filterExpression,
+ sourceSchema,
+ preTransformedSchema,
+ postTransformedSchema);
+ }
+
+ private Object[] stringify(Object... objects) {
+ return Arrays.stream(objects)
+ .map(o -> o instanceof String ? new BinaryStringData((String) o) : o)
+ .toArray();
+ }
+
+ public TransformWithSchemaEvolveTestCase evolveFromSource(SchemaChangeEvent event) {
+ sourceEvents.add(event);
+ return this;
+ }
+
+ public TransformWithSchemaEvolveTestCase expectNothingInPreTransformed() {
+ preTransformedEvents.add(null);
+ return this;
+ }
+
+ public TransformWithSchemaEvolveTestCase expectInPreTransformed(SchemaChangeEvent event) {
+ preTransformedEvents.add(event);
+ return this;
+ }
+
+ public TransformWithSchemaEvolveTestCase expectNothingInPostTransformed() {
+ postTransformedEvents.add(null);
+ return this;
+ }
+
+ public TransformWithSchemaEvolveTestCase expectInPostTransformed(SchemaChangeEvent event) {
+ postTransformedEvents.add(event);
+ return this;
+ }
+
+ private TransformWithSchemaEvolveTestCase(
+ TableId tableId,
+ String projectionExpression,
+ String filterExpression,
+ Schema sourceSchema,
+ Schema preTransformedSchema,
+ Schema postTransformedSchema) {
+ this.tableId = tableId;
+ this.projectionExpression = projectionExpression;
+ this.filterExpression = filterExpression;
+
+ this.sourceSchema = sourceSchema;
+ this.preTransformedSchema = preTransformedSchema;
+ this.postTransformedSchema = postTransformedSchema;
+
+ this.sourceEvents = new ArrayList<>();
+ this.preTransformedEvents = new ArrayList<>();
+ this.postTransformedEvents = new ArrayList<>();
+
+ this.sourceEvents.add(new CreateTableEvent(tableId, sourceSchema));
+ this.preTransformedEvents.add(new CreateTableEvent(tableId, preTransformedSchema));
+ this.postTransformedEvents.add(new CreateTableEvent(tableId, postTransformedSchema));
+ }
+
+ private TransformWithSchemaEvolveTestCase initializeHarness() throws Exception {
+ preTransformOperator =
+ PreTransformOperator.newBuilder()
+ .addTransform(
+ tableId.identifier(), projectionExpression, filterExpression)
+ .build();
+ preTransformOperatorHarness = new EventOperatorTestHarness<>(preTransformOperator, 1);
+ preTransformOperatorHarness.open();
+
+ postTransformOperator =
+ PostTransformOperator.newBuilder()
+ .addTransform(
+ tableId.identifier(), projectionExpression, filterExpression)
+ .build();
+ postTransformOperatorHarness = new EventOperatorTestHarness<>(postTransformOperator, 1);
+ postTransformOperatorHarness.open();
+ return this;
+ }
+
+ private void destroyHarness() throws Exception {
+ if (preTransformOperatorHarness != null) {
+ preTransformOperatorHarness.close();
+ }
+ if (postTransformOperatorHarness != null) {
+ postTransformOperatorHarness.close();
+ }
+ }
+
+ public TransformWithSchemaEvolveTestCase runTests(String comment) throws Exception {
+ LOG.info("Running {}#{}", getClass().getSimpleName(), comment);
+ for (int i = 0; i < sourceEvents.size(); i++) {
+ Event sourceEvent = sourceEvents.get(i);
+
+ preTransformOperator.processElement(new StreamRecord<>(sourceEvent));
+
+ Event expectedPreTransformEvent = preTransformedEvents.get(i);
+ Event actualPreTransformEvent =
+ Optional.ofNullable(preTransformOperatorHarness.getOutputRecords().poll())
+ .map(StreamRecord::getValue)
+ .orElse(null);
+
+ Assertions.assertThat(actualPreTransformEvent).isEqualTo(expectedPreTransformEvent);
+
+ postTransformOperator.processElement(
+ new StreamRecord<>(preTransformedEvents.get(i)));
+ Event expectedPostTransformEvent = postTransformedEvents.get(i);
+ Event actualPostTransformEvent =
+ Optional.ofNullable(postTransformOperatorHarness.getOutputRecords().poll())
+ .map(StreamRecord::getValue)
+ .orElse(null);
+ Assertions.assertThat(actualPostTransformEvent)
+ .isEqualTo(expectedPostTransformEvent);
+ }
+
+ sourceEvents.clear();
+ preTransformedEvents.clear();
+ postTransformedEvents.clear();
+ return this;
+ }
+
+ public TransformWithSchemaEvolveTestCase runTestsAndExpect(
+ String comment, Class<? extends Throwable> exceptionClass, String message) {
+ try {
+ Assertions.assertThatThrownBy(() -> runTests(comment))
+ .isInstanceOf(exceptionClass)
+ .hasMessageContaining(message);
+ } finally {
+ sourceEvents.clear();
+ preTransformedEvents.clear();
+ postTransformedEvents.clear();
+ }
+ return this;
+ }
+ }
+
+ /** This case tests when schema evolution happens with unspecified columns. */
+ @Test
+ public void testIrrelevantSchemaChangeInExplicitTransformRules() throws Exception {
+ TableId tableId = TableId.tableId("my_company", "my_branch", "data_changes");
+ TransformWithSchemaEvolveTestCase.of(
+ tableId,
+ "id, age, id + age as computed",
+ "id > 100",
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("age", DataTypes.INT())
+ .physicalColumn("computed", DataTypes.INT())
+ .primaryKey("id")
+ .build())
+ .initializeHarness()
+ .runTests("initializing table")
+ .evolveFromSource(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn("extras", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.LAST,
+ null))))
+ .expectNothingInPreTransformed()
+ .expectNothingInPostTransformed()
+ .runTests("inserting unspecified columns")
+ .evolveFromSource(
+ new RenameColumnEvent(tableId, Collections.singletonMap("name", "namae")))
+ .expectNothingInPreTransformed()
+ .expectNothingInPostTransformed()
+ .runTests("renaming unspecified columns")
+ .evolveFromSource(
+ new AlterColumnTypeEvent(
+ tableId, Collections.singletonMap("extras", DataTypes.DOUBLE())))
+ .expectNothingInPreTransformed()
+ .expectNothingInPostTransformed()
+ .runTests("altering unspecified columns' type")
+ .evolveFromSource(new DropColumnEvent(tableId, Arrays.asList("namae", "extras")))
+ .expectNothingInPreTransformed()
+ .expectNothingInPostTransformed()
+ .runTests("dropping unspecified columns")
+ .destroyHarness();
+ }
+
+ /** This case tests when schema evolution happens with referenced-only columns. */
+ @Test
+ public void testSemiRelevantSchemaChangeInExplicitTransformRules() throws Exception {
+ TableId tableId = TableId.tableId("my_company", "my_branch", "data_changes");
+ TransformWithSchemaEvolveTestCase.of(
+ tableId,
+ "id, age, id + age as computed",
+ "name > 100",
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("age", DataTypes.INT())
+ .physicalColumn("computed", DataTypes.INT())
+ .primaryKey("id")
+ .build())
+ .initializeHarness()
+ .runTests("initializing table")
+ .evolveFromSource(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn("extras", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.LAST,
+ null))))
+ .expectNothingInPreTransformed()
+ .expectNothingInPostTransformed()
+ .runTests("inserting unspecified columns")
+ .evolveFromSource(
+ new AlterColumnTypeEvent(
+ tableId, Collections.singletonMap("extras", DataTypes.DOUBLE())))
+ .expectNothingInPreTransformed()
+ .expectNothingInPostTransformed()
+ .runTests("altering unspecified columns' type")
+ .evolveFromSource(
+ new AlterColumnTypeEvent(
+ tableId, Collections.singletonMap("name", DataTypes.VARCHAR(17))))
+ .expectInPreTransformed(
+ new AlterColumnTypeEvent(
+ tableId, Collections.singletonMap("name", DataTypes.VARCHAR(17))))
+ .expectNothingInPostTransformed()
+ .runTests("altering referenced columns' type")
+ .evolveFromSource(new DropColumnEvent(tableId, Collections.singletonList("extras")))
+ .expectNothingInPreTransformed()
+ .expectNothingInPostTransformed()
+ .runTests("dropping unspecified columns")
+ .destroyHarness();
+ }
+
+ /** This case tests when schema evolution happens with explicitly-written columns. */
+ @Test
+ public void testRelevantColumnSchemaInExplicitTransformRules() throws Exception {
+ TableId tableId = TableId.tableId("my_company", "my_branch", "data_changes");
+ TransformWithSchemaEvolveTestCase.of(
+ tableId,
+ "id, age, name, id + age as computed",
+ "name <> 'Alice'",
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("age", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("computed", DataTypes.INT())
+ .primaryKey("id")
+ .build())
+ .initializeHarness()
+ .runTests("initializing table")
+ .evolveFromSource(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .expectInPreTransformed(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .expectInPostTransformed(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .runTests("schema evolution on relevant columns");
+ }
+
+ /** This case tests when schema evolution happens with a wildcard character at first. */
+ @Test
+ public void testSchemaChangeWithPreWildcard() throws Exception {
+ TableId tableId = TableId.tableId("my_company", "my_branch", "data_changes");
+ TransformWithSchemaEvolveTestCase.of(
+ tableId,
+ "*, id + age as computed",
+ "name <> 'Alice'",
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .physicalColumn("computed", DataTypes.INT())
+ .primaryKey("id")
+ .build())
+ .initializeHarness()
+ .runTests("initializing table")
+ .evolveFromSource(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .expectInPreTransformed(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .expectInPostTransformed(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .runTests("schema evolution on relevant columns")
+ .evolveFromSource(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_first", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.FIRST,
+ null))))
+ .expectInPreTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_first", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.BEFORE,
+ "id"))))
+ .expectInPostTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_first", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.BEFORE,
+ "id"))))
+ .runTests("inserting columns at first")
+ .evolveFromSource(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_last", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.LAST,
+ null))))
+ .expectInPreTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_last", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "age"))))
+ .expectInPostTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_last", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "age"))))
+ .runTests("inserting columns at last")
+ .evolveFromSource(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_middle", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "name"))))
+ .expectInPreTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_middle", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "name"))))
+ .expectInPostTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_middle", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "name"))))
+ .runTests("inserting columns at last");
+ }
+
+ /** This case tests when schema evolution happens with a wildcard character in the middle. */
+ @Test
+ public void testSchemaChangeWithMidWildcard() throws Exception {
+ TableId tableId = TableId.tableId("my_company", "my_branch", "data_changes");
+ TransformWithSchemaEvolveTestCase.of(
+ tableId,
+ "id + age as computed1, *, id * age as computed2",
+ "name <> 'Alice'",
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("computed1", DataTypes.INT())
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .physicalColumn("computed2", DataTypes.INT())
+ .primaryKey("id")
+ .build())
+ .initializeHarness()
+ .runTests("initializing table")
+ .evolveFromSource(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .expectInPreTransformed(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .expectInPostTransformed(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .runTests("schema evolution on relevant columns")
+ .evolveFromSource(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_first", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.FIRST,
+ null))))
+ .expectInPreTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_first", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.BEFORE,
+ "id"))))
+ .expectInPostTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_first", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.BEFORE,
+ "id"))))
+ .runTests("inserting columns at first")
+ .evolveFromSource(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_last", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.LAST,
+ null))))
+ .expectInPreTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_last", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "age"))))
+ .expectInPostTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_last", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "age"))))
+ .runTests("inserting columns at last")
+ .evolveFromSource(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_middle", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "name"))))
+ .expectInPreTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_middle", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "name"))))
+ .expectInPostTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_middle", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "name"))))
+ .runTests("inserting columns at last");
+ }
+
+ /** This case tests when schema evolution happens with a wildcard character at last. */
+ @Test
+ public void testSchemaChangeWithPostWildcard() throws Exception {
+ TableId tableId = TableId.tableId("my_company", "my_branch", "data_changes");
+ TransformWithSchemaEvolveTestCase.of(
+ tableId,
+ "id + age as computed, *",
+ "name <> 'Alice'",
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build(),
+ Schema.newBuilder()
+ .physicalColumn("computed", DataTypes.INT())
+ .physicalColumn("id", DataTypes.INT())
+ .physicalColumn("name", DataTypes.STRING())
+ .physicalColumn("age", DataTypes.INT())
+ .primaryKey("id")
+ .build())
+ .initializeHarness()
+ .runTests("initializing table")
+ .evolveFromSource(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .expectInPreTransformed(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .expectInPostTransformed(
+ new AlterColumnTypeEvent(
+ tableId,
+ ImmutableMap.of(
+ "name", DataTypes.VARCHAR(17),
+ "age", DataTypes.DOUBLE())))
+ .runTests("schema evolution on relevant columns")
+ .evolveFromSource(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_first", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.FIRST,
+ null))))
+ .expectInPreTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_first", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.BEFORE,
+ "id"))))
+ .expectInPostTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_first", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.BEFORE,
+ "id"))))
+ .runTests("inserting columns at first")
+ .evolveFromSource(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_last", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.LAST,
+ null))))
+ .expectInPreTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_last", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "age"))))
+ .expectInPostTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_last", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "age"))))
+ .runTests("inserting columns at last")
+ .evolveFromSource(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_middle", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "name"))))
+ .expectInPreTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_middle", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "name"))))
+ .expectInPostTransformed(
+ new AddColumnEvent(
+ tableId,
+ Collections.singletonList(
+ new AddColumnEvent.ColumnWithPosition(
+ Column.physicalColumn(
+ "extras_middle", DataTypes.FLOAT()),
+ AddColumnEvent.ColumnPosition.AFTER,
+ "name"))))
+ .runTests("inserting columns at last");
+ }
+}
(flink-cdc) 10/12: [FLINK-36128][cdc-runtime] Fix potential unrecoverable in-flight data exception by promoting LENIENT as the default schema change behavior
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit debd43cdd75eb53ceed8c2483e0a14edc1b265c6
Author: yuxiqian <34...@users.noreply.github.com>
AuthorDate: Mon Aug 26 17:46:37 2024 +0800
[FLINK-36128][cdc-runtime] Fix potential unrecoverable in-flight data exception by promoting LENIENT as the default schema change behavior
This closes #3574.
(cherry picked from commit 2e938a92f5335515f7eb46077254aee25acd9107)
---
.../cli/parser/YamlPipelineDefinitionParser.java | 38 ++++++++----
.../parser/YamlPipelineDefinitionParserTest.java | 36 +++++++++++-
.../cdc/common/event/SchemaChangeEventType.java | 20 +++++--
.../flink/cdc/common/pipeline/PipelineOptions.java | 2 +-
.../flink/cdc/common/utils/ChangeEventUtils.java | 8 +--
.../cdc/common/utils/ChangeEventUtilsTest.java | 67 ++++++++++++----------
.../flink/FlinkPipelineComposerITCase.java | 21 +++++++
.../flink/FlinkPipelineTransformITCase.java | 13 +++++
.../cdc/composer/flink/FlinkPipelineUdfITCase.java | 21 +++++++
.../flink/cdc/pipeline/tests/MysqlE2eITCase.java | 3 +-
.../tests/SchemaEvolvingTransformE2eITCase.java | 10 ++--
.../cdc/pipeline/tests/TransformE2eITCase.java | 18 ++++--
.../runtime/partitioning/PrePartitionOperator.java | 6 +-
13 files changed, 195 insertions(+), 68 deletions(-)
diff --git a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java
index d5df8eda1..ffba60784 100644
--- a/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java
+++ b/flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParser.java
@@ -19,6 +19,8 @@ package org.apache.flink.cdc.cli.parser;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.SchemaChangeEventType;
+import org.apache.flink.cdc.common.event.SchemaChangeEventTypeFamily;
+import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.utils.StringUtils;
import org.apache.flink.cdc.composer.definition.PipelineDef;
import org.apache.flink.cdc.composer.definition.RouteDef;
@@ -35,11 +37,13 @@ import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.yaml.YA
import java.nio.file.Path;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import static org.apache.flink.cdc.common.pipeline.PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR;
import static org.apache.flink.cdc.common.utils.ChangeEventUtils.resolveSchemaEvolutionOptions;
import static org.apache.flink.cdc.common.utils.Preconditions.checkNotNull;
@@ -99,6 +103,19 @@ public class YamlPipelineDefinitionParser implements PipelineDefinitionParser {
private PipelineDef parse(JsonNode pipelineDefJsonNode, Configuration globalPipelineConfig)
throws Exception {
+
+ // UDFs are optional. We parse UDF first and remove it from the pipelineDefJsonNode since
+ // it's not of plain data types and must be removed before calling toPipelineConfig.
+ List<UdfDef> udfDefs = new ArrayList<>();
+ Optional.ofNullable(((ObjectNode) pipelineDefJsonNode.get(PIPELINE_KEY)).remove(UDF_KEY))
+ .ifPresent(node -> node.forEach(udf -> udfDefs.add(toUdfDef(udf))));
+
+ // Pipeline configs are optional
+ Configuration userPipelineConfig = toPipelineConfig(pipelineDefJsonNode.get(PIPELINE_KEY));
+
+ SchemaChangeBehavior schemaChangeBehavior =
+ userPipelineConfig.get(PIPELINE_SCHEMA_CHANGE_BEHAVIOR);
+
// Source is required
SourceDef sourceDef =
toSourceDef(
@@ -113,7 +130,8 @@ public class YamlPipelineDefinitionParser implements PipelineDefinitionParser {
checkNotNull(
pipelineDefJsonNode.get(SINK_KEY),
"Missing required field \"%s\" in pipeline definition",
- SINK_KEY));
+ SINK_KEY),
+ schemaChangeBehavior);
// Transforms are optional
List<TransformDef> transformDefs = new ArrayList<>();
@@ -128,14 +146,6 @@ public class YamlPipelineDefinitionParser implements PipelineDefinitionParser {
Optional.ofNullable(pipelineDefJsonNode.get(ROUTE_KEY))
.ifPresent(node -> node.forEach(route -> routeDefs.add(toRouteDef(route))));
- // UDFs are optional
- List<UdfDef> udfDefs = new ArrayList<>();
- Optional.ofNullable(((ObjectNode) pipelineDefJsonNode.get(PIPELINE_KEY)).remove(UDF_KEY))
- .ifPresent(node -> node.forEach(udf -> udfDefs.add(toUdfDef(udf))));
-
- // Pipeline configs are optional
- Configuration userPipelineConfig = toPipelineConfig(pipelineDefJsonNode.get(PIPELINE_KEY));
-
// Merge user config into global config
Configuration pipelineConfig = new Configuration();
pipelineConfig.addAll(globalPipelineConfig);
@@ -162,7 +172,7 @@ public class YamlPipelineDefinitionParser implements PipelineDefinitionParser {
return new SourceDef(type, name, Configuration.fromMap(sourceMap));
}
- private SinkDef toSinkDef(JsonNode sinkNode) {
+ private SinkDef toSinkDef(JsonNode sinkNode, SchemaChangeBehavior schemaChangeBehavior) {
List<String> includedSETypes = new ArrayList<>();
List<String> excludedSETypes = new ArrayList<>();
@@ -172,6 +182,14 @@ public class YamlPipelineDefinitionParser implements PipelineDefinitionParser {
Optional.ofNullable(sinkNode.get(EXCLUDE_SCHEMA_EVOLUTION_TYPES))
.ifPresent(e -> e.forEach(tag -> excludedSETypes.add(tag.asText())));
+ if (includedSETypes.isEmpty()) {
+ // If no schema evolution types are specified, include all schema evolution types by
+ // default.
+ Arrays.stream(SchemaChangeEventTypeFamily.ALL)
+ .map(SchemaChangeEventType::getTag)
+ .forEach(includedSETypes::add);
+ }
+
Set<SchemaChangeEventType> declaredSETypes =
resolveSchemaEvolutionOptions(includedSETypes, excludedSETypes);
diff --git a/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParserTest.java b/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParserTest.java
index f57dd62c7..2ecf45870 100644
--- a/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParserTest.java
+++ b/flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/parser/YamlPipelineDefinitionParserTest.java
@@ -27,6 +27,7 @@ import org.apache.flink.cdc.composer.definition.TransformDef;
import org.apache.flink.cdc.composer.definition.UdfDef;
import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableMap;
+import org.apache.flink.shaded.guava31.com.google.common.collect.ImmutableSet;
import org.apache.flink.shaded.guava31.com.google.common.io.Resources;
import org.junit.jupiter.api.Test;
@@ -37,6 +38,11 @@ import java.time.Duration;
import java.util.Arrays;
import java.util.Collections;
+import static org.apache.flink.cdc.common.event.SchemaChangeEventType.ADD_COLUMN;
+import static org.apache.flink.cdc.common.event.SchemaChangeEventType.ALTER_COLUMN_TYPE;
+import static org.apache.flink.cdc.common.event.SchemaChangeEventType.CREATE_TABLE;
+import static org.apache.flink.cdc.common.event.SchemaChangeEventType.DROP_COLUMN;
+import static org.apache.flink.cdc.common.event.SchemaChangeEventType.RENAME_COLUMN;
import static org.apache.flink.cdc.common.pipeline.PipelineOptions.PIPELINE_LOCAL_TIME_ZONE;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
@@ -384,7 +390,13 @@ class YamlPipelineDefinitionParserTest {
Configuration.fromMap(
ImmutableMap.<String, String>builder()
.put("bootstrap-servers", "localhost:9092")
- .build())),
+ .build()),
+ ImmutableSet.of(
+ DROP_COLUMN,
+ ALTER_COLUMN_TYPE,
+ ADD_COLUMN,
+ CREATE_TABLE,
+ RENAME_COLUMN)),
Collections.singletonList(
new RouteDef(
"mydb.default.app_order_.*",
@@ -401,7 +413,16 @@ class YamlPipelineDefinitionParserTest {
private final PipelineDef minimizedDef =
new PipelineDef(
new SourceDef("mysql", null, new Configuration()),
- new SinkDef("kafka", null, new Configuration()),
+ new SinkDef(
+ "kafka",
+ null,
+ new Configuration(),
+ ImmutableSet.of(
+ DROP_COLUMN,
+ ALTER_COLUMN_TYPE,
+ ADD_COLUMN,
+ CREATE_TABLE,
+ RENAME_COLUMN)),
Collections.emptyList(),
Collections.emptyList(),
Collections.emptyList(),
@@ -474,7 +495,16 @@ class YamlPipelineDefinitionParserTest {
private final PipelineDef pipelineDefWithUdf =
new PipelineDef(
new SourceDef("values", null, new Configuration()),
- new SinkDef("values", null, new Configuration()),
+ new SinkDef(
+ "values",
+ null,
+ new Configuration(),
+ ImmutableSet.of(
+ DROP_COLUMN,
+ ALTER_COLUMN_TYPE,
+ ADD_COLUMN,
+ CREATE_TABLE,
+ RENAME_COLUMN)),
Collections.emptyList(),
Collections.singletonList(
new TransformDef(
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/SchemaChangeEventType.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/SchemaChangeEventType.java
index 668ea76a4..3e03ec4cf 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/SchemaChangeEventType.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/event/SchemaChangeEventType.java
@@ -22,11 +22,21 @@ import org.apache.flink.cdc.common.annotation.PublicEvolving;
/** An enumeration of schema change event types for {@link SchemaChangeEvent}. */
@PublicEvolving
public enum SchemaChangeEventType {
- ADD_COLUMN,
- ALTER_COLUMN_TYPE,
- CREATE_TABLE,
- DROP_COLUMN,
- RENAME_COLUMN;
+ ADD_COLUMN("add.column"),
+ ALTER_COLUMN_TYPE("alter.column.type"),
+ CREATE_TABLE("create.table"),
+ DROP_COLUMN("drop.column"),
+ RENAME_COLUMN("rename.column");
+
+ private final String tag;
+
+ SchemaChangeEventType(String tag) {
+ this.tag = tag;
+ }
+
+ public String getTag() {
+ return tag;
+ }
public static SchemaChangeEventType ofEvent(SchemaChangeEvent event) {
if (event instanceof AddColumnEvent) {
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/pipeline/PipelineOptions.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/pipeline/PipelineOptions.java
index 48a4fbb13..343e99270 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/pipeline/PipelineOptions.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/pipeline/PipelineOptions.java
@@ -48,7 +48,7 @@ public class PipelineOptions {
public static final ConfigOption<SchemaChangeBehavior> PIPELINE_SCHEMA_CHANGE_BEHAVIOR =
ConfigOptions.key("schema.change.behavior")
.enumType(SchemaChangeBehavior.class)
- .defaultValue(SchemaChangeBehavior.EVOLVE)
+ .defaultValue(SchemaChangeBehavior.LENIENT)
.withDescription(
Description.builder()
.text("Behavior for handling schema change events. ")
diff --git a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/utils/ChangeEventUtils.java b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/utils/ChangeEventUtils.java
index ee72ef6e2..e2c289e35 100644
--- a/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/utils/ChangeEventUtils.java
+++ b/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/utils/ChangeEventUtils.java
@@ -95,12 +95,8 @@ public class ChangeEventUtils {
List<String> includedSchemaEvolutionTypes, List<String> excludedSchemaEvolutionTypes) {
List<SchemaChangeEventType> resultTypes = new ArrayList<>();
- if (includedSchemaEvolutionTypes.isEmpty()) {
- resultTypes.addAll(Arrays.asList(SchemaChangeEventTypeFamily.ALL));
- } else {
- for (String includeTag : includedSchemaEvolutionTypes) {
- resultTypes.addAll(resolveSchemaEvolutionTag(includeTag));
- }
+ for (String includeTag : includedSchemaEvolutionTypes) {
+ resultTypes.addAll(resolveSchemaEvolutionTag(includeTag));
}
for (String excludeTag : excludedSchemaEvolutionTypes) {
diff --git a/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/utils/ChangeEventUtilsTest.java b/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/utils/ChangeEventUtilsTest.java
index 0a34a7501..e159fb1d1 100644
--- a/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/utils/ChangeEventUtilsTest.java
+++ b/flink-cdc-common/src/test/java/org/apache/flink/cdc/common/utils/ChangeEventUtilsTest.java
@@ -17,59 +17,67 @@
package org.apache.flink.cdc.common.utils;
-import org.assertj.core.api.Assertions;
+import org.apache.flink.cdc.common.event.SchemaChangeEventType;
+import org.apache.flink.cdc.common.event.SchemaChangeEventTypeFamily;
+
import org.assertj.core.util.Sets;
import org.junit.jupiter.api.Test;
import java.util.Arrays;
import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
import static org.apache.flink.cdc.common.event.SchemaChangeEventType.ADD_COLUMN;
import static org.apache.flink.cdc.common.event.SchemaChangeEventType.ALTER_COLUMN_TYPE;
import static org.apache.flink.cdc.common.event.SchemaChangeEventType.CREATE_TABLE;
import static org.apache.flink.cdc.common.event.SchemaChangeEventType.DROP_COLUMN;
import static org.apache.flink.cdc.common.event.SchemaChangeEventType.RENAME_COLUMN;
+import static org.apache.flink.cdc.common.testutils.assertions.EventAssertions.assertThat;
/** A test for the {@link org.apache.flink.cdc.common.utils.ChangeEventUtils}. */
public class ChangeEventUtilsTest {
@Test
public void testResolveSchemaEvolutionOptions() {
- Assertions.assertThat(
- ChangeEventUtils.resolveSchemaEvolutionOptions(
- Collections.emptyList(), Collections.emptyList()))
+
+ List<String> allTags =
+ Arrays.stream(SchemaChangeEventTypeFamily.ALL)
+ .map(SchemaChangeEventType::getTag)
+ .collect(Collectors.toList());
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionOptions(allTags, Collections.emptyList()))
.isEqualTo(
Sets.set(
+ RENAME_COLUMN,
CREATE_TABLE,
- ADD_COLUMN,
ALTER_COLUMN_TYPE,
- DROP_COLUMN,
- RENAME_COLUMN));
+ ADD_COLUMN,
+ DROP_COLUMN));
- Assertions.assertThat(
+ assertThat(
ChangeEventUtils.resolveSchemaEvolutionOptions(
- Collections.emptyList(), Collections.singletonList("drop")))
- .isEqualTo(Sets.set(CREATE_TABLE, ADD_COLUMN, ALTER_COLUMN_TYPE, RENAME_COLUMN));
+ allTags, Collections.singletonList("drop")))
+ .isEqualTo(Sets.set(ADD_COLUMN, ALTER_COLUMN_TYPE, RENAME_COLUMN, CREATE_TABLE));
- Assertions.assertThat(
+ assertThat(
ChangeEventUtils.resolveSchemaEvolutionOptions(
Arrays.asList("create", "add"), Collections.emptyList()))
- .isEqualTo(Sets.set(CREATE_TABLE, ADD_COLUMN));
+ .isEqualTo(Sets.set(ADD_COLUMN, CREATE_TABLE));
- Assertions.assertThat(
+ assertThat(
ChangeEventUtils.resolveSchemaEvolutionOptions(
Collections.singletonList("column"),
Collections.singletonList("drop.column")))
.isEqualTo(Sets.set(ADD_COLUMN, ALTER_COLUMN_TYPE, RENAME_COLUMN));
- Assertions.assertThat(
+ assertThat(
ChangeEventUtils.resolveSchemaEvolutionOptions(
- Collections.emptyList(), Collections.singletonList("drop.column")))
- .isEqualTo(Sets.set(CREATE_TABLE, ADD_COLUMN, ALTER_COLUMN_TYPE, RENAME_COLUMN));
+ allTags, Collections.singletonList("drop.column")))
+ .isEqualTo(Sets.set(ADD_COLUMN, RENAME_COLUMN, ALTER_COLUMN_TYPE, CREATE_TABLE));
}
@Test
public void testResolveSchemaEvolutionTag() {
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("all"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("all"))
.isEqualTo(
Arrays.asList(
ADD_COLUMN,
@@ -78,41 +86,38 @@ public class ChangeEventUtilsTest {
DROP_COLUMN,
RENAME_COLUMN));
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("column"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("column"))
.isEqualTo(
Arrays.asList(ADD_COLUMN, ALTER_COLUMN_TYPE, DROP_COLUMN, RENAME_COLUMN));
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("table"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("table"))
.isEqualTo(Collections.singletonList(CREATE_TABLE));
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("rename"))
- .isEqualTo(Collections.singletonList(RENAME_COLUMN));
-
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("rename.column"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("rename.column"))
.isEqualTo(Collections.singletonList(RENAME_COLUMN));
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("drop"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("drop"))
.isEqualTo(Collections.singletonList(DROP_COLUMN));
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("drop.column"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("drop.column"))
.isEqualTo(Collections.singletonList(DROP_COLUMN));
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("create"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("create"))
.isEqualTo(Collections.singletonList(CREATE_TABLE));
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("create.table"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("create.table"))
.isEqualTo(Collections.singletonList(CREATE_TABLE));
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("alter"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("alter"))
.isEqualTo(Collections.singletonList(ALTER_COLUMN_TYPE));
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("alter.column.type"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("alter.column.type"))
.isEqualTo(Collections.singletonList(ALTER_COLUMN_TYPE));
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("add"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("add"))
.isEqualTo(Collections.singletonList(ADD_COLUMN));
- Assertions.assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("add.column"))
+ assertThat(ChangeEventUtils.resolveSchemaEvolutionTag("add.column"))
.isEqualTo(Collections.singletonList(ADD_COLUMN));
}
}
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposerITCase.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposerITCase.java
index 8e471d886..4e16d2380 100644
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposerITCase.java
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposerITCase.java
@@ -26,6 +26,7 @@ import org.apache.flink.cdc.common.event.Event;
import org.apache.flink.cdc.common.event.RenameColumnEvent;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.pipeline.PipelineOptions;
+import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.schema.Column;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.types.DataType;
@@ -134,6 +135,8 @@ class FlinkPipelineComposerITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -191,6 +194,8 @@ class FlinkPipelineComposerITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -313,6 +318,8 @@ class FlinkPipelineComposerITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -373,6 +380,8 @@ class FlinkPipelineComposerITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -441,6 +450,8 @@ class FlinkPipelineComposerITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -496,6 +507,8 @@ class FlinkPipelineComposerITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -569,6 +582,8 @@ class FlinkPipelineComposerITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -766,6 +781,8 @@ class FlinkPipelineComposerITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -972,6 +989,8 @@ class FlinkPipelineComposerITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -1035,6 +1054,8 @@ class FlinkPipelineComposerITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java
index 322a3358e..54ae2c225 100644
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java
@@ -29,6 +29,7 @@ import org.apache.flink.cdc.common.event.Event;
import org.apache.flink.cdc.common.event.RenameColumnEvent;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.pipeline.PipelineOptions;
+import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.common.schema.Column;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.types.DataType;
@@ -213,6 +214,8 @@ class FlinkPipelineTransformITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
pipelineConfig.set(PipelineOptions.PIPELINE_LOCAL_TIME_ZONE, "America/Los_Angeles");
PipelineDef pipelineDef =
new PipelineDef(
@@ -266,6 +269,8 @@ class FlinkPipelineTransformITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -350,6 +355,8 @@ class FlinkPipelineTransformITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -436,6 +443,8 @@ class FlinkPipelineTransformITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -516,6 +525,8 @@ class FlinkPipelineTransformITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -608,6 +619,8 @@ class FlinkPipelineTransformITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
diff --git a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineUdfITCase.java b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineUdfITCase.java
index f9be7d7dd..c3b412dc7 100644
--- a/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineUdfITCase.java
+++ b/flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineUdfITCase.java
@@ -19,6 +19,7 @@ package org.apache.flink.cdc.composer.flink;
import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.pipeline.PipelineOptions;
+import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
import org.apache.flink.cdc.composer.PipelineExecution;
import org.apache.flink.cdc.composer.definition.PipelineDef;
import org.apache.flink.cdc.composer.definition.SinkDef;
@@ -139,6 +140,8 @@ public class FlinkPipelineUdfITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -207,6 +210,8 @@ public class FlinkPipelineUdfITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -273,6 +278,8 @@ public class FlinkPipelineUdfITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -341,6 +348,8 @@ public class FlinkPipelineUdfITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -411,6 +420,8 @@ public class FlinkPipelineUdfITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -472,6 +483,8 @@ public class FlinkPipelineUdfITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -565,6 +578,8 @@ public class FlinkPipelineUdfITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -632,6 +647,8 @@ public class FlinkPipelineUdfITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -697,6 +714,8 @@ public class FlinkPipelineUdfITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
@@ -758,6 +777,8 @@ public class FlinkPipelineUdfITCase {
// Setup pipeline
Configuration pipelineConfig = new Configuration();
pipelineConfig.set(PipelineOptions.PIPELINE_PARALLELISM, 1);
+ pipelineConfig.set(
+ PipelineOptions.PIPELINE_SCHEMA_CHANGE_BEHAVIOR, SchemaChangeBehavior.EVOLVE);
PipelineDef pipelineDef =
new PipelineDef(
sourceDef,
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
index c16f6de9c..5cb22d089 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
@@ -210,7 +210,8 @@ public class MysqlE2eITCase extends PipelineTestEnvironment {
+ " type: values\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: %d",
+ + " parallelism: %d\n"
+ + " schema.change.behavior: evolve",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
index a4a21c992..8b306d1e7 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
@@ -228,11 +228,12 @@ public class SchemaEvolvingTransformE2eITCase extends PipelineTestEnvironment {
+ "\n"
+ "pipeline:\n"
+ " schema.change.behavior: unexpected\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
- schemaEvolveDatabase.getDatabaseName());
+ schemaEvolveDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -305,7 +306,7 @@ public class SchemaEvolvingTransformE2eITCase extends PipelineTestEnvironment {
+ "\n"
+ "pipeline:\n"
+ " schema.change.behavior: %s\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
@@ -313,7 +314,8 @@ public class SchemaEvolvingTransformE2eITCase extends PipelineTestEnvironment {
mergeTable ? "(members|new_members)" : "members",
dbName,
dbName,
- behavior);
+ behavior,
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
index c1c8e8109..0a3b0aace 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
@@ -738,12 +738,14 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
+ " projection: ID, 'id -> ' || ID AS UID, PRICEALPHA AS PRICE\n"
+ " filter: ID > 1008\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d\n"
+ + " schema.change.behavior: evolve",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName());
+ transformTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -835,12 +837,14 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
+ " projection: \\*, 'id -> ' || ID AS UID\n"
+ " filter: ID > 1008\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d\n"
+ + " schema.change.behavior: evolve",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName());
+ transformTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -940,12 +944,14 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
+ " projection: ID || ' <- id' AS UID, *\n"
+ " filter: ID > 1008\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d\n"
+ + " schema.change.behavior: evolve",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName());
+ transformTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/PrePartitionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/PrePartitionOperator.java
index 6d66fa878..1171ad07b 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/PrePartitionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/PrePartitionOperator.java
@@ -28,6 +28,7 @@ import org.apache.flink.cdc.common.function.HashFunctionProvider;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.runtime.operators.schema.SchemaOperator;
import org.apache.flink.cdc.runtime.operators.sink.SchemaEvolutionClient;
+import org.apache.flink.cdc.runtime.serializer.event.EventSerializer;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
@@ -108,7 +109,10 @@ public class PrePartitionOperator extends AbstractStreamOperator<PartitioningEve
private void broadcastEvent(Event toBroadcast) {
for (int i = 0; i < downstreamParallelism; i++) {
- output.collect(new StreamRecord<>(new PartitioningEvent(toBroadcast, i)));
+ // Deep-copying each event is required since downstream subTasks might run in the same
+ // JVM
+ Event copiedEvent = EventSerializer.INSTANCE.copy(toBroadcast);
+ output.collect(new StreamRecord<>(new PartitioningEvent(copiedEvent, i)));
}
}
(flink-cdc) 09/12: [FLINK-35056][cdc-connector/sqlserver] Fix scale mapping from SQL Server TIMESTAMP to Flink SQL timestamp
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit daf28ddc99ae5678f50e6f3c3de0d959567d3522
Author: Sergei Morozov <mo...@tut.by>
AuthorDate: Thu Aug 22 10:09:16 2024 -0700
[FLINK-35056][cdc-connector/sqlserver] Fix scale mapping from SQL Server TIMESTAMP to Flink SQL timestamp
This closes #3561.
(cherry picked from commit 52f2019469cdeb63246317b6cb8c6b825233c27c)
---
.../sqlserver/source/utils/SqlServerTypeUtils.java | 4 ++--
.../read/fetch/SqlServerScanFetchTaskTest.java | 26 ++++++++++++++++++++++
.../src/test/resources/ddl/pk.sql | 26 ++++++++++++++++++++++
3 files changed, 54 insertions(+), 2 deletions(-)
diff --git a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-sqlserver-cdc/src/main/java/org/apache/flink/cdc/connectors/sqlserver/source/utils/SqlServerTypeUtils.java b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-sqlserver-cdc/src/main/java/org/apache/flink/cdc/connectors/sqlserver/source/utils/SqlServerTypeUtils.java
index 8f1853837..e5373e36e 100644
--- a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-sqlserver-cdc/src/main/java/org/apache/flink/cdc/connectors/sqlserver/source/utils/SqlServerTypeUtils.java
+++ b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-sqlserver-cdc/src/main/java/org/apache/flink/cdc/connectors/sqlserver/source/utils/SqlServerTypeUtils.java
@@ -71,8 +71,8 @@ public class SqlServerTypeUtils {
return DataTypes.DATE();
case Types.TIMESTAMP:
case Types.TIMESTAMP_WITH_TIMEZONE:
- return column.length() >= 0
- ? DataTypes.TIMESTAMP(column.length())
+ return column.scale().isPresent()
+ ? DataTypes.TIMESTAMP(column.scale().get())
: DataTypes.TIMESTAMP();
case Types.BOOLEAN:
return DataTypes.BOOLEAN();
diff --git a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-sqlserver-cdc/src/test/java/org/apache/flink/cdc/connectors/sqlserver/source/read/fetch/SqlServerScanFetchTaskTest.java b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-sqlserver-cdc/src/test/java/org/apache/flink/cdc/connectors/sqlserver/source/read/fetch/SqlServerScanFetchTaskTest.java
index 73b99ef31..13c7a68e3 100644
--- a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-sqlserver-cdc/src/test/java/org/apache/flink/cdc/connectors/sqlserver/source/read/fetch/SqlServerScanFetchTaskTest.java
+++ b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-sqlserver-cdc/src/test/java/org/apache/flink/cdc/connectors/sqlserver/source/read/fetch/SqlServerScanFetchTaskTest.java
@@ -35,6 +35,7 @@ import org.apache.flink.cdc.connectors.sqlserver.source.reader.fetch.SqlServerSo
import org.apache.flink.cdc.connectors.sqlserver.testutils.RecordsFormatter;
import org.apache.flink.table.api.DataTypes;
import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
import io.debezium.jdbc.JdbcConnection;
import io.debezium.relational.TableId;
@@ -50,6 +51,8 @@ import java.util.List;
import java.util.stream.Collectors;
import static org.apache.flink.cdc.connectors.sqlserver.source.utils.SqlServerConnectionUtils.createSqlServerConnection;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.testcontainers.containers.MSSQLServerContainer.MS_SQL_SERVER_PORT;
@@ -189,6 +192,29 @@ public class SqlServerScanFetchTaskTest extends SqlServerSourceTestBase {
assertEqualsInAnyOrder(Arrays.asList(expected), actual);
}
+ @Test
+ public void testDateTimePrimaryKey() throws Exception {
+ String databaseName = "pk";
+ String tableName = "dbo.dt_pk";
+
+ initializeSqlServerTable(databaseName);
+
+ SqlServerSourceConfigFactory sourceConfigFactory =
+ getConfigFactory(databaseName, new String[] {tableName}, 8096);
+ SqlServerSourceConfig sourceConfig = sourceConfigFactory.create(0);
+ SqlServerDialect sqlServerDialect = new SqlServerDialect(sourceConfig);
+
+ List<SnapshotSplit> snapshotSplits = getSnapshotSplits(sourceConfig, sqlServerDialect);
+ assertFalse(snapshotSplits.isEmpty());
+
+ RowType expectedType =
+ (RowType)
+ DataTypes.ROW(DataTypes.FIELD("dt", DataTypes.TIMESTAMP(3).notNull()))
+ .getLogicalType();
+
+ snapshotSplits.forEach(s -> assertEquals(expectedType, s.getSplitKeyType()));
+ }
+
@Test
public void testDeleteDataInSnapshotScan() throws Exception {
String databaseName = "customer";
diff --git a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-sqlserver-cdc/src/test/resources/ddl/pk.sql b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-sqlserver-cdc/src/test/resources/ddl/pk.sql
new file mode 100644
index 000000000..3986a75be
--- /dev/null
+++ b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-sqlserver-cdc/src/test/resources/ddl/pk.sql
@@ -0,0 +1,26 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+CREATE DATABASE pk;
+
+USE pk;
+EXEC sys.sp_cdc_enable_db;
+
+CREATE TABLE dt_pk (
+ dt datetime NOT NULL PRIMARY KEY,
+ val INT
+);
+
+EXEC sys.sp_cdc_enable_table @source_schema = 'dbo', @source_name = 'dt_pk', @role_name = NULL, @supports_net_changes = 0;
(flink-cdc) 11/12: [hotfix][cdc-runtime] Keep upstream pending requests in order to avoid checkpoint hanging
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 9816848c4b46ed303423411c354b92fcdb10c2a5
Author: yuxiqian <34...@users.noreply.github.com>
AuthorDate: Mon Aug 26 14:07:54 2024 +0800
[hotfix][cdc-runtime] Keep upstream pending requests in order to avoid checkpoint hanging
---
.../cdc/pipeline/tests/SchemaEvolveE2eITCase.java | 1 +
.../tests/SchemaEvolvingTransformE2eITCase.java | 1 +
.../runtime/operators/schema/SchemaOperator.java | 10 +-
.../coordinator/SchemaRegistryRequestHandler.java | 148 +++++++++++++--------
.../schema/event/SchemaChangeRequest.java | 15 ++-
.../runtime/partitioning/PrePartitionOperator.java | 7 +
.../operators/EventOperatorTestHarness.java | 2 +-
7 files changed, 125 insertions(+), 59 deletions(-)
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
index 0a8d7c483..64e3e9c57 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
@@ -338,6 +338,7 @@ public class SchemaEvolveE2eITCase extends PipelineTestEnvironment {
// triggers DropColumnEvent
stmt.execute("ALTER TABLE members DROP COLUMN biological_sex");
stmt.execute("INSERT INTO members VALUES (1013, 'Fiona', 16);");
+ stmt.execute("INSERT INTO members VALUES (1014, 'Gem', 17);");
}
List<String> expectedTmEvents =
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
index 8b306d1e7..edcac1bbe 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java
@@ -349,6 +349,7 @@ public class SchemaEvolvingTransformE2eITCase extends PipelineTestEnvironment {
// triggers DropColumnEvent
stmt.execute("ALTER TABLE members DROP COLUMN biological_sex");
stmt.execute("INSERT INTO members VALUES (1013, 'Fiona', 16);");
+ stmt.execute("INSERT INTO members VALUES (1014, 'Gem', 17);");
}
List<String> expectedTmEvents =
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java
index 1b4f50e89..a1bdd7885 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java
@@ -50,6 +50,7 @@ import org.apache.flink.cdc.runtime.typeutils.BinaryRecordDataGenerator;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.runtime.operators.coordination.CoordinationRequest;
import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
+import org.apache.flink.runtime.state.StateSnapshotContext;
import org.apache.flink.streaming.api.graph.StreamConfig;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
@@ -440,7 +441,8 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
long schemaEvolveTimeOutMillis = System.currentTimeMillis() + rpcTimeOutInMillis;
while (true) {
SchemaChangeResponse response =
- sendRequestToCoordinator(new SchemaChangeRequest(tableId, schemaChangeEvent));
+ sendRequestToCoordinator(
+ new SchemaChangeRequest(tableId, schemaChangeEvent, subTaskId));
if (response.isRegistryBusy()) {
if (System.currentTimeMillis() < schemaEvolveTimeOutMillis) {
LOG.info(
@@ -609,4 +611,10 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
}
}
}
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ // Needless to do anything, since AbstractStreamOperator#snapshotState and #processElement
+ // is guaranteed not to be mixed together.
+ }
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
index 9262a6a90..de5bde4e6 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
@@ -54,7 +54,6 @@ import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
-import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Collectors;
import java.util.stream.Stream;
@@ -77,7 +76,10 @@ public class SchemaRegistryRequestHandler implements Closeable {
/**
* Atomic flag indicating if current RequestHandler could accept more schema changes for now.
*/
- private final AtomicReference<RequestStatus> schemaChangeStatus;
+ private volatile RequestStatus schemaChangeStatus;
+
+ private final List<Integer> pendingSubTaskIds;
+ private final Object schemaChangeRequestLock;
private volatile Throwable currentChangeException;
private volatile List<SchemaChangeEvent> currentDerivedSchemaChangeEvents;
@@ -109,7 +111,10 @@ public class SchemaRegistryRequestHandler implements Closeable {
this.currentDerivedSchemaChangeEvents = new ArrayList<>();
this.currentFinishedSchemaChanges = new ArrayList<>();
this.currentIgnoredSchemaChanges = new ArrayList<>();
- this.schemaChangeStatus = new AtomicReference<>(RequestStatus.IDLE);
+
+ this.schemaChangeStatus = RequestStatus.IDLE;
+ this.pendingSubTaskIds = new ArrayList<>();
+ this.schemaChangeRequestLock = new Object();
}
/**
@@ -119,54 +124,86 @@ public class SchemaRegistryRequestHandler implements Closeable {
*/
public CompletableFuture<CoordinationResponse> handleSchemaChangeRequest(
SchemaChangeRequest request) {
- if (schemaChangeStatus.compareAndSet(RequestStatus.IDLE, RequestStatus.WAITING_FOR_FLUSH)) {
- LOG.info(
- "Received schema change event request {} from table {}. SchemaChangeStatus switched from IDLE to WAITING_FOR_FLUSH, other requests will be blocked.",
- request.getSchemaChangeEvent(),
- request.getTableId().toString());
- SchemaChangeEvent event = request.getSchemaChangeEvent();
-
- // If this schema change event has been requested by another subTask, ignore it.
- if (schemaManager.isOriginalSchemaChangeEventRedundant(event)) {
- LOG.info("Event {} has been addressed before, ignoring it.", event);
- clearCurrentSchemaChangeRequest();
- Preconditions.checkState(
- schemaChangeStatus.compareAndSet(
- RequestStatus.WAITING_FOR_FLUSH, RequestStatus.IDLE),
- "Illegal schemaChangeStatus state: should still in WAITING_FOR_FLUSH state if event was duplicated, not "
- + schemaChangeStatus.get());
+
+ // We use requester subTask ID as the pending ticket, because there will be at most 1 schema
+ // change requests simultaneously from each subTask
+ int requestSubTaskId = request.getSubTaskId();
+
+ synchronized (schemaChangeRequestLock) {
+ // Make sure we handle the first request in the pending list to avoid out-of-order
+ // waiting and blocks checkpointing mechanism.
+ if (schemaChangeStatus == RequestStatus.IDLE) {
+ if (pendingSubTaskIds.isEmpty()) {
+ LOG.info(
+ "Received schema change event request {} from table {} from subTask {}. Pending list is empty, handling this.",
+ request.getSchemaChangeEvent(),
+ request.getTableId().toString(),
+ requestSubTaskId);
+ } else if (pendingSubTaskIds.get(0) == requestSubTaskId) {
+ LOG.info(
+ "Received schema change event request {} from table {} from subTask {}. It is on the first of the pending list, handling this.",
+ request.getSchemaChangeEvent(),
+ request.getTableId().toString(),
+ requestSubTaskId);
+ pendingSubTaskIds.remove(0);
+ } else {
+ LOG.info(
+ "Received schema change event request {} from table {} from subTask {}. It is not the first of the pending list ({}).",
+ request.getSchemaChangeEvent(),
+ request.getTableId().toString(),
+ requestSubTaskId,
+ pendingSubTaskIds);
+ if (!pendingSubTaskIds.contains(requestSubTaskId)) {
+ pendingSubTaskIds.add(requestSubTaskId);
+ }
+ return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.busy()));
+ }
+
+ SchemaChangeEvent event = request.getSchemaChangeEvent();
+
+ // If this schema change event has been requested by another subTask, ignore it.
+ if (schemaManager.isOriginalSchemaChangeEventRedundant(event)) {
+ LOG.info("Event {} has been addressed before, ignoring it.", event);
+ clearCurrentSchemaChangeRequest();
+ LOG.info(
+ "SchemaChangeStatus switched from WAITING_FOR_FLUSH to IDLE for request {} due to duplicated request.",
+ request);
+ return CompletableFuture.completedFuture(
+ wrap(SchemaChangeResponse.duplicate()));
+ }
+ schemaManager.applyOriginalSchemaChange(event);
+ List<SchemaChangeEvent> derivedSchemaChangeEvents =
+ calculateDerivedSchemaChangeEvents(request.getSchemaChangeEvent());
+
+ // If this schema change event is filtered out by LENIENT mode or merging table
+ // route strategies, ignore it.
+ if (derivedSchemaChangeEvents.isEmpty()) {
+ LOG.info("Event {} is omitted from sending to downstream, ignoring it.", event);
+ clearCurrentSchemaChangeRequest();
+ LOG.info(
+ "SchemaChangeStatus switched from WAITING_FOR_FLUSH to IDLE for request {} due to ignored request.",
+ request);
+ return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.ignored()));
+ }
+
LOG.info(
- "SchemaChangeStatus switched from WAITING_FOR_FLUSH to IDLE for request {} due to duplicated request.",
- request);
- return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.duplicate()));
- }
- schemaManager.applyOriginalSchemaChange(event);
- List<SchemaChangeEvent> derivedSchemaChangeEvents =
- calculateDerivedSchemaChangeEvents(request.getSchemaChangeEvent());
-
- // If this schema change event is filtered out by LENIENT mode or merging table route
- // strategies, ignore it.
- if (derivedSchemaChangeEvents.isEmpty()) {
- LOG.info("Event {} is omitted from sending to downstream, ignoring it.", event);
- clearCurrentSchemaChangeRequest();
- Preconditions.checkState(
- schemaChangeStatus.compareAndSet(
- RequestStatus.WAITING_FOR_FLUSH, RequestStatus.IDLE),
- "Illegal schemaChangeStatus state: should still in WAITING_FOR_FLUSH state if event was ignored, not "
- + schemaChangeStatus.get());
+ "SchemaChangeStatus switched from IDLE to WAITING_FOR_FLUSH, other requests will be blocked.");
+ // This request has been accepted.
+ schemaChangeStatus = RequestStatus.WAITING_FOR_FLUSH;
+ currentDerivedSchemaChangeEvents = new ArrayList<>(derivedSchemaChangeEvents);
+ return CompletableFuture.completedFuture(
+ wrap(SchemaChangeResponse.accepted(derivedSchemaChangeEvents)));
+ } else {
LOG.info(
- "SchemaChangeStatus switched from WAITING_FOR_FLUSH to IDLE for request {} due to ignored request.",
- request);
- return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.ignored()));
+ "Schema Registry is busy processing a schema change request, could not handle request {} for now. Added {} to pending list ({}).",
+ request,
+ requestSubTaskId,
+ pendingSubTaskIds);
+ if (!pendingSubTaskIds.contains(requestSubTaskId)) {
+ pendingSubTaskIds.add(requestSubTaskId);
+ }
+ return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.busy()));
}
- currentDerivedSchemaChangeEvents = new ArrayList<>(derivedSchemaChangeEvents);
- return CompletableFuture.completedFuture(
- wrap(SchemaChangeResponse.accepted(derivedSchemaChangeEvents)));
- } else {
- LOG.info(
- "Schema Registry is busy processing a schema change request, could not handle request {} for now.",
- request);
- return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.busy()));
}
}
@@ -213,9 +250,10 @@ public class SchemaRegistryRequestHandler implements Closeable {
}
}
Preconditions.checkState(
- schemaChangeStatus.compareAndSet(RequestStatus.APPLYING, RequestStatus.FINISHED),
+ schemaChangeStatus == RequestStatus.APPLYING,
"Illegal schemaChangeStatus state: should be APPLYING before applySchemaChange finishes, not "
- + schemaChangeStatus.get());
+ + schemaChangeStatus);
+ schemaChangeStatus = RequestStatus.FINISHED;
LOG.info(
"SchemaChangeStatus switched from APPLYING to FINISHED for request {}.",
currentDerivedSchemaChangeEvents);
@@ -248,10 +286,11 @@ public class SchemaRegistryRequestHandler implements Closeable {
}
if (flushedSinkWriters.equals(activeSinkWriters)) {
Preconditions.checkState(
- schemaChangeStatus.compareAndSet(
- RequestStatus.WAITING_FOR_FLUSH, RequestStatus.APPLYING),
+ schemaChangeStatus == RequestStatus.WAITING_FOR_FLUSH,
"Illegal schemaChangeStatus state: should be WAITING_FOR_FLUSH before collecting enough FlushEvents, not "
+ schemaChangeStatus);
+
+ schemaChangeStatus = RequestStatus.APPLYING;
LOG.info(
"All sink subtask have flushed for table {}. Start to apply schema change.",
tableId.toString());
@@ -262,9 +301,10 @@ public class SchemaRegistryRequestHandler implements Closeable {
public CompletableFuture<CoordinationResponse> getSchemaChangeResult() {
Preconditions.checkState(
- !schemaChangeStatus.get().equals(RequestStatus.IDLE),
+ schemaChangeStatus != RequestStatus.IDLE,
"Illegal schemaChangeStatus: should not be IDLE before getting schema change request results.");
- if (schemaChangeStatus.compareAndSet(RequestStatus.FINISHED, RequestStatus.IDLE)) {
+ if (schemaChangeStatus == RequestStatus.FINISHED) {
+ schemaChangeStatus = RequestStatus.IDLE;
LOG.info(
"SchemaChangeStatus switched from FINISHED to IDLE for request {}",
currentDerivedSchemaChangeEvents);
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeRequest.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeRequest.java
index fda6b02f1..fbc5e9c40 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeRequest.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeRequest.java
@@ -36,10 +36,14 @@ public class SchemaChangeRequest implements CoordinationRequest {
private final TableId tableId;
/** The schema changes. */
private final SchemaChangeEvent schemaChangeEvent;
+ /** The ID of subTask that initiated the request. */
+ private final int subTaskId;
- public SchemaChangeRequest(TableId tableId, SchemaChangeEvent schemaChangeEvent) {
+ public SchemaChangeRequest(
+ TableId tableId, SchemaChangeEvent schemaChangeEvent, int subTaskId) {
this.tableId = tableId;
this.schemaChangeEvent = schemaChangeEvent;
+ this.subTaskId = subTaskId;
}
public TableId getTableId() {
@@ -50,6 +54,10 @@ public class SchemaChangeRequest implements CoordinationRequest {
return schemaChangeEvent;
}
+ public int getSubTaskId() {
+ return subTaskId;
+ }
+
@Override
public boolean equals(Object o) {
if (this == o) {
@@ -60,11 +68,12 @@ public class SchemaChangeRequest implements CoordinationRequest {
}
SchemaChangeRequest that = (SchemaChangeRequest) o;
return Objects.equals(tableId, that.tableId)
- && Objects.equals(schemaChangeEvent, that.schemaChangeEvent);
+ && Objects.equals(schemaChangeEvent, that.schemaChangeEvent)
+ && subTaskId == that.subTaskId;
}
@Override
public int hashCode() {
- return Objects.hash(tableId, schemaChangeEvent);
+ return Objects.hash(tableId, schemaChangeEvent, subTaskId);
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/PrePartitionOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/PrePartitionOperator.java
index 1171ad07b..938e6950d 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/PrePartitionOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/partitioning/PrePartitionOperator.java
@@ -31,6 +31,7 @@ import org.apache.flink.cdc.runtime.operators.sink.SchemaEvolutionClient;
import org.apache.flink.cdc.runtime.serializer.event.EventSerializer;
import org.apache.flink.runtime.jobgraph.OperatorID;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
+import org.apache.flink.runtime.state.StateSnapshotContext;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
@@ -147,4 +148,10 @@ public class PrePartitionOperator extends AbstractStreamOperator<PartitioningEve
}
});
}
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ // Needless to do anything, since AbstractStreamOperator#snapshotState and #processElement
+ // is guaranteed not to be mixed together.
+ }
}
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
index 60952b424..a469a3a91 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
@@ -164,7 +164,7 @@ public class EventOperatorTestHarness<OP extends AbstractStreamOperator<E>, E ex
public void registerTableSchema(TableId tableId, Schema schema) {
schemaRegistry.handleCoordinationRequest(
- new SchemaChangeRequest(tableId, new CreateTableEvent(tableId, schema)));
+ new SchemaChangeRequest(tableId, new CreateTableEvent(tableId, schema), 0));
schemaRegistry.handleApplyEvolvedSchemaChangeRequest(new CreateTableEvent(tableId, schema));
}
(flink-cdc) 03/12: [FLINK-36076][minor][cdc-runtime] Set isSchemaChangeApplying as volatile for thread safe consideration
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 9f358ab1d7c2693d475560dd02d62a75ebe659ef
Author: Hongshun Wang <12...@users.noreply.github.com>
AuthorDate: Tue Aug 20 00:53:02 2024 +0800
[FLINK-36076][minor][cdc-runtime] Set isSchemaChangeApplying as volatile for thread safe consideration
This closes #3556.
(cherry picked from commit d3473de4db92229eb09e1e4c698a2db5448c8805)
---
.../org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java | 3 ++-
.../flink/cdc/composer/flink/translator/DataSourceTranslator.java | 7 ++++---
.../operators/schema/coordinator/SchemaRegistryRequestHandler.java | 2 +-
3 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java
index bddd5fc00..b3f17b2e5 100644
--- a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java
+++ b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/FlinkPipelineComposer.java
@@ -97,7 +97,8 @@ public class FlinkPipelineComposer implements PipelineComposer {
// Build Source Operator
DataSourceTranslator sourceTranslator = new DataSourceTranslator();
DataStream<Event> stream =
- sourceTranslator.translate(pipelineDef.getSource(), env, pipelineDef.getConfig());
+ sourceTranslator.translate(
+ pipelineDef.getSource(), env, pipelineDef.getConfig(), parallelism);
// Build PreTransformOperator for processing Schema Event
TransformTranslator transformTranslator = new TransformTranslator();
diff --git a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java
index 7b631c6f1..90bbea73e 100644
--- a/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java
+++ b/flink-cdc-composer/src/main/java/org/apache/flink/cdc/composer/flink/translator/DataSourceTranslator.java
@@ -23,7 +23,6 @@ import org.apache.flink.cdc.common.configuration.Configuration;
import org.apache.flink.cdc.common.event.Event;
import org.apache.flink.cdc.common.factories.DataSourceFactory;
import org.apache.flink.cdc.common.factories.FactoryHelper;
-import org.apache.flink.cdc.common.pipeline.PipelineOptions;
import org.apache.flink.cdc.common.source.DataSource;
import org.apache.flink.cdc.common.source.EventSourceProvider;
import org.apache.flink.cdc.common.source.FlinkSourceFunctionProvider;
@@ -41,12 +40,14 @@ import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
public class DataSourceTranslator {
public DataStreamSource<Event> translate(
- SourceDef sourceDef, StreamExecutionEnvironment env, Configuration pipelineConfig) {
+ SourceDef sourceDef,
+ StreamExecutionEnvironment env,
+ Configuration pipelineConfig,
+ int sourceParallelism) {
// Create data source
DataSource dataSource = createDataSource(sourceDef, env, pipelineConfig);
// Get source provider
- final int sourceParallelism = pipelineConfig.get(PipelineOptions.PIPELINE_PARALLELISM);
EventSourceProvider eventSourceProvider = dataSource.getEventSourceProvider();
if (eventSourceProvider instanceof FlinkSourceProvider) {
// Source
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
index 77360169e..da88753e5 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
@@ -94,7 +94,7 @@ public class SchemaRegistryRequestHandler implements Closeable {
private final Set<Integer> flushedSinkWriters;
/** Status of the execution of current schema change request. */
- private boolean isSchemaChangeApplying;
+ private volatile boolean isSchemaChangeApplying;
/** Executor service to execute schema change. */
private final ExecutorService schemaChangeThreadPool;
(flink-cdc) 06/12: [FLINK-36114][cdc-runtime] Make SchemaRegistryRequestHandler thread safe by blocking subsequent schemaChangeEvent
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 6dd1c11fe13f29fcaf202c35234040f44383e3bf
Author: yuxiqian <34...@users.noreply.github.com>
AuthorDate: Thu Aug 22 19:19:21 2024 +0800
[FLINK-36114][cdc-runtime] Make SchemaRegistryRequestHandler thread safe by blocking subsequent schemaChangeEvent
This closes #3563.
Co-authored-by: Hongshun Wang <lo...@gmail.com>
(cherry picked from commit ee843e2f246c300ecfeaf5cfd529693fadf2625f)
---
.../source/reader/MySqlPipelineRecordEmitter.java | 62 ++--
.../mysql/source/MySqlFullTypesITCase.java | 26 +-
.../mysql/source/MySqlPipelineITCase.java | 30 +-
.../mysql/testutils/MySqSourceTestUtils.java | 19 ++
.../cdc/pipeline/tests/MySqlToDorisE2eITCase.java | 10 +-
.../flink/cdc/pipeline/tests/MysqlE2eITCase.java | 253 ++++++++++-----
.../flink/cdc/pipeline/tests/RouteE2eITCase.java | 35 ++-
.../cdc/pipeline/tests/SchemaEvolveE2eITCase.java | 10 +-
.../cdc/pipeline/tests/TransformE2eITCase.java | 44 ++-
.../flink/cdc/pipeline/tests/UdfE2eITCase.java | 6 +-
.../tests/utils/PipelineTestEnvironment.java | 2 +
.../runtime/operators/schema/SchemaOperator.java | 74 +++--
.../schema/coordinator/SchemaManager.java | 96 ++++++
.../schema/coordinator/SchemaRegistry.java | 10 +-
.../coordinator/SchemaRegistryRequestHandler.java | 342 +++++++++------------
.../schema/event/RefreshPendingListsRequest.java | 27 --
.../schema/event/RefreshPendingListsResponse.java | 26 --
.../schema/event/ReleaseUpstreamRequest.java | 32 --
.../event/SchemaChangeProcessingResponse.java | 4 +-
.../schema/event/SchemaChangeResponse.java | 75 ++++-
...sponse.java => SchemaChangeResultResponse.java} | 8 +-
.../operators/EventOperatorTestHarness.java | 1 -
22 files changed, 711 insertions(+), 481 deletions(-)
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlPipelineRecordEmitter.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlPipelineRecordEmitter.java
index 3f7581347..909ed6c5b 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlPipelineRecordEmitter.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlPipelineRecordEmitter.java
@@ -28,6 +28,7 @@ import org.apache.flink.cdc.connectors.mysql.source.config.MySqlSourceConfig;
import org.apache.flink.cdc.connectors.mysql.source.metrics.MySqlSourceReaderMetrics;
import org.apache.flink.cdc.connectors.mysql.source.split.MySqlSplitState;
import org.apache.flink.cdc.connectors.mysql.table.StartupMode;
+import org.apache.flink.cdc.connectors.mysql.utils.MySqlSchemaUtils;
import org.apache.flink.cdc.connectors.mysql.utils.MySqlTypeUtils;
import org.apache.flink.cdc.debezium.DebeziumDeserializationSchema;
import org.apache.flink.connector.base.source.reader.RecordEmitter;
@@ -68,7 +69,7 @@ public class MySqlPipelineRecordEmitter extends MySqlRecordEmitter<Event> {
// Used when startup mode is not initial
private boolean alreadySendCreateTableForBinlogSplit = false;
- private final List<CreateTableEvent> createTableEventCache;
+ private List<CreateTableEvent> createTableEventCache;
public MySqlPipelineRecordEmitter(
DebeziumDeserializationSchema<Event> debeziumDeserializationSchema,
@@ -80,23 +81,7 @@ public class MySqlPipelineRecordEmitter extends MySqlRecordEmitter<Event> {
sourceConfig.isIncludeSchemaChanges());
this.sourceConfig = sourceConfig;
this.alreadySendCreateTableTables = new HashSet<>();
- this.createTableEventCache = new ArrayList<>();
-
- if (!sourceConfig.getStartupOptions().startupMode.equals(StartupMode.INITIAL)) {
- try (JdbcConnection jdbc = openJdbcConnection(sourceConfig)) {
- List<TableId> capturedTableIds = listTables(jdbc, sourceConfig.getTableFilters());
- for (TableId tableId : capturedTableIds) {
- Schema schema = getSchema(jdbc, tableId);
- createTableEventCache.add(
- new CreateTableEvent(
- org.apache.flink.cdc.common.event.TableId.tableId(
- tableId.catalog(), tableId.table()),
- schema));
- }
- } catch (SQLException e) {
- throw new RuntimeException("Cannot start emitter to fetch table schema.", e);
- }
- }
+ this.createTableEventCache = generateCreateTableEvent(sourceConfig);
}
@Override
@@ -104,6 +89,8 @@ public class MySqlPipelineRecordEmitter extends MySqlRecordEmitter<Event> {
SourceRecord element, SourceOutput<Event> output, MySqlSplitState splitState)
throws Exception {
if (isLowWatermarkEvent(element) && splitState.isSnapshotSplitState()) {
+ // In Snapshot phase of INITIAL startup mode, we lazily send CreateTableEvent to
+ // downstream to avoid checkpoint timeout.
TableId tableId = splitState.asSnapshotSplitState().toMySqlSplit().getTableId();
if (!alreadySendCreateTableTables.contains(tableId)) {
try (JdbcConnection jdbc = openJdbcConnection(sourceConfig)) {
@@ -111,11 +98,24 @@ public class MySqlPipelineRecordEmitter extends MySqlRecordEmitter<Event> {
alreadySendCreateTableTables.add(tableId);
}
}
- } else if (splitState.isBinlogSplitState()
- && !alreadySendCreateTableForBinlogSplit
- && !sourceConfig.getStartupOptions().startupMode.equals(StartupMode.INITIAL)) {
- createTableEventCache.forEach(output::collect);
+ } else if (splitState.isBinlogSplitState() && !alreadySendCreateTableForBinlogSplit) {
alreadySendCreateTableForBinlogSplit = true;
+ if (sourceConfig.getStartupOptions().startupMode.equals(StartupMode.INITIAL)) {
+ // In Snapshot -> Binlog transition of INITIAL startup mode, ensure all table
+ // schemas have been sent to downstream. We use previously cached schema instead of
+ // re-request latest schema because there might be some pending schema change events
+ // in the queue, and that may accidentally emit evolved schema before corresponding
+ // schema change events.
+ createTableEventCache.stream()
+ .filter(
+ event ->
+ !alreadySendCreateTableTables.contains(
+ MySqlSchemaUtils.toDbzTableId(event.tableId())))
+ .forEach(output::collect);
+ } else {
+ // In Binlog only mode, we simply emit all schemas at once.
+ createTableEventCache.forEach(output::collect);
+ }
}
super.processElement(element, output, splitState);
}
@@ -233,4 +233,22 @@ public class MySqlPipelineRecordEmitter extends MySqlRecordEmitter<Event> {
}
return mySqlAntlrDdlParser;
}
+
+ private List<CreateTableEvent> generateCreateTableEvent(MySqlSourceConfig sourceConfig) {
+ try (JdbcConnection jdbc = openJdbcConnection(sourceConfig)) {
+ List<CreateTableEvent> createTableEventCache = new ArrayList<>();
+ List<TableId> capturedTableIds = listTables(jdbc, sourceConfig.getTableFilters());
+ for (TableId tableId : capturedTableIds) {
+ Schema schema = getSchema(jdbc, tableId);
+ createTableEventCache.add(
+ new CreateTableEvent(
+ org.apache.flink.cdc.common.event.TableId.tableId(
+ tableId.catalog(), tableId.table()),
+ schema));
+ }
+ return createTableEventCache;
+ } catch (SQLException e) {
+ throw new RuntimeException("Cannot start emitter to fetch table schema.", e);
+ }
+ }
}
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java
index 91351dabf..2bd1ae689 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java
@@ -295,8 +295,9 @@ public class MySqlFullTypesITCase extends MySqlSourceTestBase {
.executeAndCollect();
// skip CreateTableEvent
- List<Event> snapshotResults = MySqSourceTestUtils.fetchResults(iterator, 2);
- RecordData snapshotRecord = ((DataChangeEvent) snapshotResults.get(1)).after();
+ List<Event> snapshotResults =
+ MySqSourceTestUtils.fetchResultsAndCreateTableEvent(iterator, 1).f0;
+ RecordData snapshotRecord = ((DataChangeEvent) snapshotResults.get(0)).after();
Assertions.assertThat(RecordDataTestUtils.recordFields(snapshotRecord, recordType))
.isEqualTo(expectedSnapshot);
@@ -306,7 +307,8 @@ public class MySqlFullTypesITCase extends MySqlSourceTestBase {
statement.execute("UPDATE precision_types SET time_6_c = null WHERE id = 1;");
}
- List<Event> streamResults = MySqSourceTestUtils.fetchResults(iterator, 1);
+ List<Event> streamResults =
+ MySqSourceTestUtils.fetchResultsAndCreateTableEvent(iterator, 1).f0;
RecordData streamRecord = ((DataChangeEvent) streamResults.get(0)).after();
Assertions.assertThat(RecordDataTestUtils.recordFields(streamRecord, recordType))
.isEqualTo(expectedStreamRecord);
@@ -397,8 +399,9 @@ public class MySqlFullTypesITCase extends MySqlSourceTestBase {
};
// skip CreateTableEvent
- List<Event> snapshotResults = MySqSourceTestUtils.fetchResults(iterator, 2);
- RecordData snapshotRecord = ((DataChangeEvent) snapshotResults.get(1)).after();
+ List<Event> snapshotResults =
+ MySqSourceTestUtils.fetchResultsAndCreateTableEvent(iterator, 1).f0;
+ RecordData snapshotRecord = ((DataChangeEvent) snapshotResults.get(0)).after();
Assertions.assertThat(RecordDataTestUtils.recordFields(snapshotRecord, COMMON_TYPES))
.isEqualTo(expectedSnapshot);
@@ -412,7 +415,8 @@ public class MySqlFullTypesITCase extends MySqlSourceTestBase {
expectedSnapshot[44] = BinaryStringData.fromString("{\"key1\":\"value1\"}");
Object[] expectedStreamRecord = expectedSnapshot;
- List<Event> streamResults = MySqSourceTestUtils.fetchResults(iterator, 1);
+ List<Event> streamResults =
+ MySqSourceTestUtils.fetchResultsAndCreateTableEvent(iterator, 1).f0;
RecordData streamRecord = ((DataChangeEvent) streamResults.get(0)).after();
Assertions.assertThat(RecordDataTestUtils.recordFields(streamRecord, COMMON_TYPES))
.isEqualTo(expectedStreamRecord);
@@ -437,9 +441,10 @@ public class MySqlFullTypesITCase extends MySqlSourceTestBase {
"Event-Source")
.executeAndCollect();
- // skip CreateTableEvent
- List<Event> snapshotResults = MySqSourceTestUtils.fetchResults(iterator, 2);
- RecordData snapshotRecord = ((DataChangeEvent) snapshotResults.get(1)).after();
+ // skip CreateTableEvents
+ List<Event> snapshotResults =
+ MySqSourceTestUtils.fetchResultsAndCreateTableEvent(iterator, 1).f0;
+ RecordData snapshotRecord = ((DataChangeEvent) snapshotResults.get(0)).after();
Assertions.assertThat(RecordDataTestUtils.recordFields(snapshotRecord, recordType))
.isEqualTo(expectedSnapshot);
@@ -450,7 +455,8 @@ public class MySqlFullTypesITCase extends MySqlSourceTestBase {
"UPDATE time_types SET time_6_c = null, timestamp_def_c = default WHERE id = 1;");
}
- List<Event> streamResults = MySqSourceTestUtils.fetchResults(iterator, 1);
+ List<Event> streamResults =
+ MySqSourceTestUtils.fetchResultsAndCreateTableEvent(iterator, 1).f0;
RecordData streamRecord = ((DataChangeEvent) streamResults.get(0)).after();
Assertions.assertThat(RecordDataTestUtils.recordFields(streamRecord, recordType))
.isEqualTo(expectedStreamRecord);
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlPipelineITCase.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlPipelineITCase.java
index bd2c059bb..f019e98d8 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlPipelineITCase.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlPipelineITCase.java
@@ -60,6 +60,7 @@ import java.sql.Statement;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
+import java.util.Iterator;
import java.util.List;
import java.util.stream.Stream;
@@ -223,12 +224,33 @@ public class MySqlPipelineITCase extends MySqlSourceTestBase {
BinaryStringData.fromString("c-21")
})));
}
+ // In this configuration, several subtasks might emit their corresponding CreateTableEvent
+ // to downstream. Since it is not possible to predict how many CreateTableEvents should we
+ // expect, we simply filter them out from expected sets, and assert there's at least one.
List<Event> actual =
- fetchResults(events, 1 + expectedSnapshot.size() + expectedBinlog.size());
- assertThat(actual.get(0)).isEqualTo(createTableEvent);
- assertThat(actual.subList(1, 10))
+ fetchResultsExcept(
+ events, expectedSnapshot.size() + expectedBinlog.size(), createTableEvent);
+ assertThat(actual.subList(0, expectedSnapshot.size()))
.containsExactlyInAnyOrder(expectedSnapshot.toArray(new Event[0]));
- assertThat(actual.subList(10, actual.size())).isEqualTo(expectedBinlog);
+ assertThat(actual.subList(expectedSnapshot.size(), actual.size()))
+ .isEqualTo(expectedBinlog);
+ }
+
+ private static <T> List<T> fetchResultsExcept(Iterator<T> iter, int size, T sideEvent) {
+ List<T> result = new ArrayList<>(size);
+ List<T> sideResults = new ArrayList<>();
+ while (size > 0 && iter.hasNext()) {
+ T event = iter.next();
+ if (!event.equals(sideEvent)) {
+ result.add(event);
+ size--;
+ } else {
+ sideResults.add(sideEvent);
+ }
+ }
+ // Also ensure we've received at least one or many side events.
+ assertThat(sideResults).isNotEmpty();
+ return result;
}
@Test
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/testutils/MySqSourceTestUtils.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/testutils/MySqSourceTestUtils.java
index 0708cd8cd..a76838d66 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/testutils/MySqSourceTestUtils.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/testutils/MySqSourceTestUtils.java
@@ -17,6 +17,9 @@
package org.apache.flink.cdc.connectors.mysql.testutils;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.cdc.common.event.CreateTableEvent;
+
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
@@ -38,6 +41,22 @@ public class MySqSourceTestUtils {
return result;
}
+ public static <T> Tuple2<List<T>, List<CreateTableEvent>> fetchResultsAndCreateTableEvent(
+ Iterator<T> iter, int size) {
+ List<T> result = new ArrayList<>(size);
+ List<CreateTableEvent> createTableEvents = new ArrayList<>();
+ while (size > 0 && iter.hasNext()) {
+ T event = iter.next();
+ if (event instanceof CreateTableEvent) {
+ createTableEvents.add((CreateTableEvent) event);
+ } else {
+ result.add(event);
+ size--;
+ }
+ }
+ return Tuple2.of(result, createTableEvents);
+ }
+
public static String getServerId(int parallelism) {
final Random random = new Random();
int serverId = random.nextInt(100) + 5400;
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToDorisE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToDorisE2eITCase.java
index 0e58c5ed2..508eebdb7 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToDorisE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToDorisE2eITCase.java
@@ -194,12 +194,13 @@ public class MySqlToDorisE2eITCase extends PipelineTestEnvironment {
+ " table.create.properties.replication_num: 1\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
mysqlInventoryDatabase.getDatabaseName(),
DORIS.getUsername(),
- DORIS.getPassword());
+ DORIS.getPassword(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path dorisCdcConnector = TestUtils.getResource("doris-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -324,13 +325,14 @@ public class MySqlToDorisE2eITCase extends PipelineTestEnvironment {
+ " projection: \\*, 'fine' AS FINE\n"
+ " filter: id <> 3 AND id <> 4\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
complexDataTypesDatabase.getDatabaseName(),
DORIS.getUsername(),
DORIS.getPassword(),
- complexDataTypesDatabase.getDatabaseName());
+ complexDataTypesDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path dorisCdcConnector = TestUtils.getResource("doris-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
index 917873bb8..1f730be62 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java
@@ -39,8 +39,6 @@ import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Duration;
-import java.util.Arrays;
-import java.util.List;
import java.util.concurrent.TimeoutException;
/** End-to-end tests for mysql cdc pipeline job. */
@@ -104,11 +102,12 @@ public class MysqlE2eITCase extends PipelineTestEnvironment {
+ " type: values\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
- mysqlInventoryDatabase.getDatabaseName());
+ mysqlInventoryDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -123,54 +122,24 @@ public class MysqlE2eITCase extends PipelineTestEnvironment {
String.format(
"DataChangeEvent{tableId=%s.products, before=[], after=[109, spare tire, 24 inch spare tire, 22.2, null, null, null], op=INSERT, meta=()}",
mysqlInventoryDatabase.getDatabaseName()));
- List<String> expectedEvents =
- Arrays.asList(
- String.format(
- "CreateTableEvent{tableId=%s.customers, schema=columns={`id` INT NOT NULL,`name` VARCHAR(255) NOT NULL 'flink',`address` VARCHAR(1024),`phone_number` VARCHAR(512)}, primaryKeys=id, options=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.customers, before=[], after=[104, user_4, Shanghai, 123567891234], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.customers, before=[], after=[103, user_3, Shanghai, 123567891234], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.customers, before=[], after=[102, user_2, Shanghai, 123567891234], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.customers, before=[], after=[101, user_1, Shanghai, 123567891234], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "CreateTableEvent{tableId=%s.products, schema=columns={`id` INT NOT NULL,`name` VARCHAR(255) NOT NULL 'flink',`description` VARCHAR(512),`weight` FLOAT,`enum_c` STRING 'red',`json_c` STRING,`point_c` STRING}, primaryKeys=id, options=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[], after=[109, spare tire, 24 inch spare tire, 22.2, null, null, null], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[], after=[107, rocks, box of assorted rocks, 5.3, null, null, null], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[], after=[108, jacket, water resistent black wind breaker, 0.1, null, null, null], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[], after=[105, hammer, 14oz carpenter's hammer, 0.875, red, {\"k1\": \"v1\", \"k2\": \"v2\"}, {\"coordinates\":[5,5],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[], after=[106, hammer, 16oz carpenter's hammer, 1.0, null, null, null], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[], after=[103, 12-pack drill bits, 12-pack of drill bits with sizes ranging from #40 to #3, 0.8, red, {\"key3\": \"value3\"}, {\"coordinates\":[3,3],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[], after=[104, hammer, 12oz carpenter's hammer, 0.75, white, {\"key4\": \"value4\"}, {\"coordinates\":[4,4],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[], after=[101, scooter, Small 2-wheel scooter, 3.14, red, {\"key1\": \"value1\"}, {\"coordinates\":[1,1],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[], after=[102, car battery, 12V car battery, 8.1, white, {\"key2\": \"value2\"}, {\"coordinates\":[2,2],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()));
- validateResult(expectedEvents);
+
+ validateResult(
+ "CreateTableEvent{tableId=%s.customers, schema=columns={`id` INT NOT NULL,`name` VARCHAR(255) NOT NULL 'flink',`address` VARCHAR(1024),`phone_number` VARCHAR(512)}, primaryKeys=id, options=()}",
+ "DataChangeEvent{tableId=%s.customers, before=[], after=[104, user_4, Shanghai, 123567891234], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.customers, before=[], after=[103, user_3, Shanghai, 123567891234], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.customers, before=[], after=[102, user_2, Shanghai, 123567891234], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.customers, before=[], after=[101, user_1, Shanghai, 123567891234], op=INSERT, meta=()}",
+ "CreateTableEvent{tableId=%s.products, schema=columns={`id` INT NOT NULL,`name` VARCHAR(255) NOT NULL 'flink',`description` VARCHAR(512),`weight` FLOAT,`enum_c` STRING 'red',`json_c` STRING,`point_c` STRING}, primaryKeys=id, options=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[109, spare tire, 24 inch spare tire, 22.2, null, null, null], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[107, rocks, box of assorted rocks, 5.3, null, null, null], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[108, jacket, water resistent black wind breaker, 0.1, null, null, null], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[105, hammer, 14oz carpenter's hammer, 0.875, red, {\"k1\": \"v1\", \"k2\": \"v2\"}, {\"coordinates\":[5,5],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[106, hammer, 16oz carpenter's hammer, 1.0, null, null, null], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[103, 12-pack drill bits, 12-pack of drill bits with sizes ranging from #40 to #3, 0.8, red, {\"key3\": \"value3\"}, {\"coordinates\":[3,3],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[104, hammer, 12oz carpenter's hammer, 0.75, white, {\"key4\": \"value4\"}, {\"coordinates\":[4,4],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[101, scooter, Small 2-wheel scooter, 3.14, red, {\"key1\": \"value1\"}, {\"coordinates\":[1,1],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[102, car battery, 12V car battery, 8.1, white, {\"key2\": \"value2\"}, {\"coordinates\":[2,2],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}");
+
LOG.info("Begin incremental reading stage.");
// generate binlogs
String mysqlJdbcUrl =
@@ -212,38 +181,160 @@ public class MysqlE2eITCase extends PipelineTestEnvironment {
"DataChangeEvent{tableId=%s.products, before=[111, scooter, Big 2-wheel scooter , 5.17, null, null, null, 1], after=[], op=DELETE, meta=()}",
mysqlInventoryDatabase.getDatabaseName()));
- expectedEvents =
- Arrays.asList(
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[106, hammer, 16oz carpenter's hammer, 1.0, null, null, null], after=[106, hammer, 18oz carpenter hammer, 1.0, null, null, null], op=UPDATE, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[107, rocks, box of assorted rocks, 5.3, null, null, null], after=[107, rocks, box of assorted rocks, 5.1, null, null, null], op=UPDATE, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "AddColumnEvent{tableId=%s.products, addedColumns=[ColumnWithPosition{column=`new_col` INT, position=LAST, existedColumnName=null}]}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[], after=[110, jacket, water resistent white wind breaker, 0.2, null, null, null, 1], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[], after=[111, scooter, Big 2-wheel scooter , 5.18, null, null, null, 1], op=INSERT, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[110, jacket, water resistent white wind breaker, 0.2, null, null, null, 1], after=[110, jacket, new water resistent white wind breaker, 0.5, null, null, null, 1], op=UPDATE, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[111, scooter, Big 2-wheel scooter , 5.18, null, null, null, 1], after=[111, scooter, Big 2-wheel scooter , 5.17, null, null, null, 1], op=UPDATE, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()),
- String.format(
- "DataChangeEvent{tableId=%s.products, before=[111, scooter, Big 2-wheel scooter , 5.17, null, null, null, 1], after=[], op=DELETE, meta=()}",
- mysqlInventoryDatabase.getDatabaseName()));
- validateResult(expectedEvents);
+ validateResult(
+ "DataChangeEvent{tableId=%s.products, before=[106, hammer, 16oz carpenter's hammer, 1.0, null, null, null], after=[106, hammer, 18oz carpenter hammer, 1.0, null, null, null], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[107, rocks, box of assorted rocks, 5.3, null, null, null], after=[107, rocks, box of assorted rocks, 5.1, null, null, null], op=UPDATE, meta=()}",
+ "AddColumnEvent{tableId=%s.products, addedColumns=[ColumnWithPosition{column=`new_col` INT, position=LAST, existedColumnName=null}]}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[110, jacket, water resistent white wind breaker, 0.2, null, null, null, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[111, scooter, Big 2-wheel scooter , 5.18, null, null, null, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[110, jacket, water resistent white wind breaker, 0.2, null, null, null, 1], after=[110, jacket, new water resistent white wind breaker, 0.5, null, null, null, 1], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[111, scooter, Big 2-wheel scooter , 5.18, null, null, null, 1], after=[111, scooter, Big 2-wheel scooter , 5.17, null, null, null, 1], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[111, scooter, Big 2-wheel scooter , 5.17, null, null, null, 1], after=[], op=DELETE, meta=()}");
+ }
+
+ @Test
+ public void testSchemaChangeEvents() throws Exception {
+ String pipelineJob =
+ String.format(
+ "source:\n"
+ + " type: mysql\n"
+ + " hostname: %s\n"
+ + " port: 3306\n"
+ + " username: %s\n"
+ + " password: %s\n"
+ + " tables: %s.\\.*\n"
+ + " server-id: 5400-5404\n"
+ + " server-time-zone: UTC\n"
+ + "\n"
+ + "sink:\n"
+ + " type: values\n"
+ + "\n"
+ + "pipeline:\n"
+ + " parallelism: %d",
+ INTER_CONTAINER_MYSQL_ALIAS,
+ MYSQL_TEST_USER,
+ MYSQL_TEST_PASSWORD,
+ mysqlInventoryDatabase.getDatabaseName(),
+ parallelism);
+ Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
+ Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
+ Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
+ submitPipelineJob(pipelineJob, mysqlCdcJar, valuesCdcJar, mysqlDriverJar);
+ waitUntilJobRunning(Duration.ofSeconds(30));
+ LOG.info("Pipeline job is running");
+ waitUntilSpecificEvent(
+ String.format(
+ "DataChangeEvent{tableId=%s.customers, before=[], after=[104, user_4, Shanghai, 123567891234], op=INSERT, meta=()}",
+ mysqlInventoryDatabase.getDatabaseName()));
+ waitUntilSpecificEvent(
+ String.format(
+ "DataChangeEvent{tableId=%s.products, before=[], after=[109, spare tire, 24 inch spare tire, 22.2, null, null, null], op=INSERT, meta=()}",
+ mysqlInventoryDatabase.getDatabaseName()));
+
+ validateResult(
+ "CreateTableEvent{tableId=%s.customers, schema=columns={`id` INT NOT NULL,`name` VARCHAR(255) NOT NULL 'flink',`address` VARCHAR(1024),`phone_number` VARCHAR(512)}, primaryKeys=id, options=()}",
+ "DataChangeEvent{tableId=%s.customers, before=[], after=[104, user_4, Shanghai, 123567891234], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.customers, before=[], after=[103, user_3, Shanghai, 123567891234], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.customers, before=[], after=[102, user_2, Shanghai, 123567891234], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.customers, before=[], after=[101, user_1, Shanghai, 123567891234], op=INSERT, meta=()}",
+ "CreateTableEvent{tableId=%s.products, schema=columns={`id` INT NOT NULL,`name` VARCHAR(255) NOT NULL 'flink',`description` VARCHAR(512),`weight` FLOAT,`enum_c` STRING 'red',`json_c` STRING,`point_c` STRING}, primaryKeys=id, options=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[109, spare tire, 24 inch spare tire, 22.2, null, null, null], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[107, rocks, box of assorted rocks, 5.3, null, null, null], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[108, jacket, water resistent black wind breaker, 0.1, null, null, null], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[105, hammer, 14oz carpenter's hammer, 0.875, red, {\"k1\": \"v1\", \"k2\": \"v2\"}, {\"coordinates\":[5,5],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[106, hammer, 16oz carpenter's hammer, 1.0, null, null, null], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[103, 12-pack drill bits, 12-pack of drill bits with sizes ranging from #40 to #3, 0.8, red, {\"key3\": \"value3\"}, {\"coordinates\":[3,3],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[104, hammer, 12oz carpenter's hammer, 0.75, white, {\"key4\": \"value4\"}, {\"coordinates\":[4,4],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[101, scooter, Small 2-wheel scooter, 3.14, red, {\"key1\": \"value1\"}, {\"coordinates\":[1,1],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[102, car battery, 12V car battery, 8.1, white, {\"key2\": \"value2\"}, {\"coordinates\":[2,2],\"type\":\"Point\",\"srid\":0}], op=INSERT, meta=()}");
+
+ LOG.info("Begin incremental reading stage.");
+ // generate binlogs
+ String mysqlJdbcUrl =
+ String.format(
+ "jdbc:mysql://%s:%s/%s",
+ MYSQL.getHost(),
+ MYSQL.getDatabasePort(),
+ mysqlInventoryDatabase.getDatabaseName());
+ try (Connection conn =
+ DriverManager.getConnection(
+ mysqlJdbcUrl, MYSQL_TEST_USER, MYSQL_TEST_PASSWORD);
+ Statement stat = conn.createStatement()) {
+ stat.execute("UPDATE products SET description='18oz carpenter hammer' WHERE id=106;");
+ stat.execute("UPDATE products SET weight='5.1' WHERE id=107;");
+
+ // Perform DDL changes after the binlog is generated
+ waitUntilSpecificEvent(
+ String.format(
+ "DataChangeEvent{tableId=%s.products, before=[106, hammer, 16oz carpenter's hammer, 1.0, null, null, null], after=[106, hammer, 18oz carpenter hammer, 1.0, null, null, null], op=UPDATE, meta=()}",
+ mysqlInventoryDatabase.getDatabaseName()));
+
+ LOG.info("Begin schema evolution stage.");
+
+ // Test AddColumnEvent
+ stat.execute("ALTER TABLE products ADD COLUMN new_col INT;");
+ stat.execute(
+ "INSERT INTO products VALUES (default,'jacket','water resistent white wind breaker',0.2, null, null, null, 1);"); // 110
+ stat.execute(
+ "INSERT INTO products VALUES (default,'scooter','Big 2-wheel scooter ',5.18, null, null, null, 1);"); // 111
+ stat.execute(
+ "UPDATE products SET description='new water resistent white wind breaker', weight='0.5' WHERE id=110;");
+ stat.execute("UPDATE products SET weight='5.17' WHERE id=111;");
+ stat.execute("DELETE FROM products WHERE id=111;");
+
+ // Test AlterColumnTypeEvent
+ stat.execute("ALTER TABLE products MODIFY COLUMN new_col BIGINT;");
+ stat.execute(
+ "INSERT INTO products VALUES (default,'derrida','forever 21',2.1728, null, null, null, 2147483649);"); // 112
+
+ // Test RenameColumnEvent
+ stat.execute("ALTER TABLE products RENAME COLUMN new_col TO new_column;");
+ stat.execute(
+ "INSERT INTO products VALUES (default,'dynazenon','SSSS',2.1728, null, null, null, 2147483649);"); // 113
+
+ // Test DropColumnEvent
+ stat.execute("ALTER TABLE products DROP COLUMN new_column;");
+ stat.execute(
+ "INSERT INTO products VALUES (default,'evangelion','Eva',2.1728, null, null, null);"); // 114
+
+ // Test TruncateTableEvent
+ stat.execute("TRUNCATE TABLE products;");
+
+ // Test DropTableEvent. It's all over.
+ stat.execute("DROP TABLE products;");
+ } catch (SQLException e) {
+ LOG.error("Update table for CDC failed.", e);
+ throw e;
+ }
+
+ waitUntilSpecificEvent(
+ String.format(
+ "DropTableEvent{tableId=%s.products}",
+ mysqlInventoryDatabase.getDatabaseName()));
+
+ validateResult(
+ "DataChangeEvent{tableId=%s.products, before=[106, hammer, 16oz carpenter's hammer, 1.0, null, null, null], after=[106, hammer, 18oz carpenter hammer, 1.0, null, null, null], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[107, rocks, box of assorted rocks, 5.3, null, null, null], after=[107, rocks, box of assorted rocks, 5.1, null, null, null], op=UPDATE, meta=()}",
+ "AddColumnEvent{tableId=%s.products, addedColumns=[ColumnWithPosition{column=`new_col` INT, position=LAST, existedColumnName=null}]}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[110, jacket, water resistent white wind breaker, 0.2, null, null, null, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[111, scooter, Big 2-wheel scooter , 5.18, null, null, null, 1], op=INSERT, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[110, jacket, water resistent white wind breaker, 0.2, null, null, null, 1], after=[110, jacket, new water resistent white wind breaker, 0.5, null, null, null, 1], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[111, scooter, Big 2-wheel scooter , 5.18, null, null, null, 1], after=[111, scooter, Big 2-wheel scooter , 5.17, null, null, null, 1], op=UPDATE, meta=()}",
+ "DataChangeEvent{tableId=%s.products, before=[111, scooter, Big 2-wheel scooter , 5.17, null, null, null, 1], after=[], op=DELETE, meta=()}",
+ "AlterColumnTypeEvent{tableId=%s.products, typeMapping={new_col=BIGINT}, oldTypeMapping={new_col=INT}}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[112, derrida, forever 21, 2.1728, null, null, null, 2147483649], op=INSERT, meta=()}",
+ "RenameColumnEvent{tableId=%s.products, nameMapping={new_col=new_column}}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[113, dynazenon, SSSS, 2.1728, null, null, null, 2147483649], op=INSERT, meta=()}",
+ "DropColumnEvent{tableId=%s.products, droppedColumnNames=[new_column]}",
+ "DataChangeEvent{tableId=%s.products, before=[], after=[114, evangelion, Eva, 2.1728, null, null, null], op=INSERT, meta=()}",
+ "TruncateTableEvent{tableId=%s.products}",
+ "DropTableEvent{tableId=%s.products}");
}
- private void validateResult(List<String> expectedEvents) throws Exception {
+ private void validateResult(String... expectedEvents) throws Exception {
+ String dbName = mysqlInventoryDatabase.getDatabaseName();
for (String event : expectedEvents) {
- waitUntilSpecificEvent(event);
+ waitUntilSpecificEvent(String.format(event, dbName, dbName));
}
}
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java
index ab7ed178e..f4ea4d81a 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java
@@ -154,11 +154,12 @@ public class RouteE2eITCase extends PipelineTestEnvironment {
+ " type: values\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
- routeTestDatabase.getDatabaseName());
+ routeTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -248,13 +249,14 @@ public class RouteE2eITCase extends PipelineTestEnvironment {
+ " sink-table: %s.ALL\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
routeTestDatabase.getDatabaseName(),
routeTestDatabase.getDatabaseName(),
- routeTestDatabase.getDatabaseName());
+ routeTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -331,13 +333,14 @@ public class RouteE2eITCase extends PipelineTestEnvironment {
+ " sink-table: NEW_%s.ALPHABET\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
routeTestDatabase.getDatabaseName(),
routeTestDatabase.getDatabaseName(),
- routeTestDatabase.getDatabaseName());
+ routeTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -428,7 +431,7 @@ public class RouteE2eITCase extends PipelineTestEnvironment {
+ " sink-table: NEW_%s.BETAGAMM\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
@@ -436,7 +439,8 @@ public class RouteE2eITCase extends PipelineTestEnvironment {
routeTestDatabase.getDatabaseName(),
routeTestDatabase.getDatabaseName(),
routeTestDatabase.getDatabaseName(),
- routeTestDatabase.getDatabaseName());
+ routeTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -535,7 +539,7 @@ public class RouteE2eITCase extends PipelineTestEnvironment {
+ " sink-table: NEW_%s.TABLEC\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
@@ -545,7 +549,8 @@ public class RouteE2eITCase extends PipelineTestEnvironment {
routeTestDatabase.getDatabaseName(),
routeTestDatabase.getDatabaseName(),
routeTestDatabase.getDatabaseName(),
- routeTestDatabase.getDatabaseName());
+ routeTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -647,14 +652,15 @@ public class RouteE2eITCase extends PipelineTestEnvironment {
+ " sink-table: %s.ALL\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
routeTestDatabase.getDatabaseName(),
routeTestDatabase.getDatabaseName(),
routeTestDatabase.getDatabaseName(),
- routeTestDatabase.getDatabaseName());
+ routeTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -733,13 +739,14 @@ public class RouteE2eITCase extends PipelineTestEnvironment {
+ " replace-symbol: <>\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
routeTestDatabase.getDatabaseName(),
routeTestDatabase.getDatabaseName(),
- routeTestDatabase.getDatabaseName());
+ routeTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
index c84a8824f..0a8d7c483 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java
@@ -226,11 +226,12 @@ public class SchemaEvolveE2eITCase extends PipelineTestEnvironment {
+ "\n"
+ "pipeline:\n"
+ " schema.change.behavior: unexpected\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
- schemaEvolveDatabase.getDatabaseName());
+ schemaEvolveDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -296,13 +297,14 @@ public class SchemaEvolveE2eITCase extends PipelineTestEnvironment {
+ "\n"
+ "pipeline:\n"
+ " schema.change.behavior: %s\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
dbName,
mergeTable ? "(members|new_members)" : "members",
- behavior);
+ behavior,
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
index 6f612ca95..ae516fc06 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java
@@ -118,7 +118,7 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
+ " - source-table: %s.TABLEBETA\n"
+ " projection: ID, VERSION\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
@@ -126,7 +126,8 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
transformTestDatabase.getDatabaseName(),
transformTestDatabase.getDatabaseName(),
transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName());
+ transformTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -200,13 +201,14 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
+ " filter: ID <= 1008\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
transformTestDatabase.getDatabaseName(),
transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName());
+ transformTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -287,7 +289,7 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
+ " - source-table: %s.TABLEBETA\n"
+ " projection: ID, CONCAT('v', VERSION) AS VERSION, LOWER(NAMEBETA) AS NAME\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
@@ -295,7 +297,8 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
transformTestDatabase.getDatabaseName(),
transformTestDatabase.getDatabaseName(),
transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName());
+ transformTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -365,13 +368,14 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
+ " - source-table: %s.TABLEBETA\n"
+ " projection: \\*, CONCAT('v', VERSION) AS VERSION, LOWER(NAMEBETA) AS NAME\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
transformTestDatabase.getDatabaseName(),
transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName());
+ transformTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -446,13 +450,14 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
+ " - source-table: %s.TABLEBETA\n"
+ " projection: \\*, __namespace_name__ || '.' || __schema_name__ || '.' || __table_name__ AS identifier_name, __data_event_type__ AS type\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
transformTestDatabase.getDatabaseName(),
transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName());
+ transformTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -530,12 +535,13 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
+ " - source-table: %s.TABLE\\.*\n"
+ " projection: \\*, ID + 1000 as UID, VERSION AS NEWVERSION\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName());
+ transformTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -613,13 +619,14 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
+ " - source-table: %s.TABLEBETA\n"
+ " projection: ID, CAST(VERSION AS DOUBLE) + 100 AS VERSION, CAST(AGEBETA AS VARCHAR) || ' - ' || NAMEBETA AS IDENTIFIER\n"
+ "pipeline:\n"
- + " parallelism: 1",
+ + " parallelism: %d",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
transformTestDatabase.getDatabaseName(),
transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName());
+ transformTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -708,13 +715,14 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
+ " projection: ID, LOCALTIME as lcl_t, CURRENT_TIME as cur_t, CAST(CURRENT_TIMESTAMP AS TIMESTAMP) as cur_ts, CAST(NOW() AS TIMESTAMP) as now_ts, LOCALTIMESTAMP as lcl_ts, CURRENT_DATE as cur_dt\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1\n"
+ + " parallelism: %d\n"
+ " local-time-zone: America/Los_Angeles",
INTER_CONTAINER_MYSQL_ALIAS,
MYSQL_TEST_USER,
MYSQL_TEST_PASSWORD,
transformTestDatabase.getDatabaseName(),
- transformTestDatabase.getDatabaseName());
+ transformTestDatabase.getDatabaseName(),
+ parallelism);
Path mysqlCdcJar = TestUtils.getResource("mysql-cdc-pipeline-connector.jar");
Path valuesCdcJar = TestUtils.getResource("values-cdc-pipeline-connector.jar");
Path mysqlDriverJar = TestUtils.getResource("mysql-driver.jar");
@@ -781,7 +789,9 @@ public class TransformE2eITCase extends PipelineTestEnvironment {
}
boolean extractDataLines(String line) {
- if (!line.startsWith("DataChangeEvent{")) {
+ // In multiple parallelism mode, a prefix with subTaskId (like '1> ') will be appended.
+ // Should trim it before extracting data fields.
+ if (!line.startsWith("DataChangeEvent{", 3)) {
return false;
}
Stream.of("before", "after")
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java
index 938e2d98e..9f5b98298 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java
@@ -128,7 +128,7 @@ public class UdfE2eITCase extends PipelineTestEnvironment {
+ " projection: ID, VERSION, answer() AS ANS, typeof(ID) AS TYP\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1\n"
+ + " parallelism: %d\n"
+ " user-defined-function:\n"
+ " - name: addone\n"
+ " classpath: org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\n"
@@ -146,6 +146,7 @@ public class UdfE2eITCase extends PipelineTestEnvironment {
transformRenameDatabase.getDatabaseName(),
transformRenameDatabase.getDatabaseName(),
transformRenameDatabase.getDatabaseName(),
+ parallelism,
language,
language,
language,
@@ -267,7 +268,7 @@ public class UdfE2eITCase extends PipelineTestEnvironment {
+ " projection: ID, VERSION, typeof(ID) AS TYP\n"
+ "\n"
+ "pipeline:\n"
- + " parallelism: 1\n"
+ + " parallelism: %d\n"
+ " user-defined-function:\n"
+ " - name: addone\n"
+ " classpath: org.apache.flink.udf.examples.%s.AddOneFunctionClass\n"
@@ -281,6 +282,7 @@ public class UdfE2eITCase extends PipelineTestEnvironment {
transformRenameDatabase.getDatabaseName(),
transformRenameDatabase.getDatabaseName(),
transformRenameDatabase.getDatabaseName(),
+ parallelism,
language,
language,
language);
diff --git a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
index d1c0bb7e7..530b34163 100644
--- a/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
+++ b/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java
@@ -67,6 +67,8 @@ public abstract class PipelineTestEnvironment extends TestLogger {
@Parameterized.Parameter public String flinkVersion;
+ public Integer parallelism = 4;
+
// ------------------------------------------------------------------------------------------
// Flink Variables
// ------------------------------------------------------------------------------------------
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java
index a700fd39c..1b4f50e89 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java
@@ -39,20 +39,17 @@ import org.apache.flink.cdc.common.types.DataTypeRoot;
import org.apache.flink.cdc.common.utils.ChangeEventUtils;
import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry;
import org.apache.flink.cdc.runtime.operators.schema.event.CoordinationResponseUtils;
-import org.apache.flink.cdc.runtime.operators.schema.event.RefreshPendingListsRequest;
-import org.apache.flink.cdc.runtime.operators.schema.event.ReleaseUpstreamRequest;
-import org.apache.flink.cdc.runtime.operators.schema.event.ReleaseUpstreamResponse;
import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeProcessingResponse;
import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeResponse;
import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeResultRequest;
+import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeResultResponse;
import org.apache.flink.cdc.runtime.operators.schema.metrics.SchemaOperatorMetrics;
import org.apache.flink.cdc.runtime.operators.sink.SchemaEvolutionClient;
import org.apache.flink.cdc.runtime.typeutils.BinaryRecordDataGenerator;
import org.apache.flink.runtime.jobgraph.tasks.TaskOperatorEventGateway;
import org.apache.flink.runtime.operators.coordination.CoordinationRequest;
import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
-import org.apache.flink.runtime.state.StateInitializationContext;
import org.apache.flink.streaming.api.graph.StreamConfig;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.ChainingStrategy;
@@ -120,6 +117,7 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
private final SchemaChangeBehavior schemaChangeBehavior;
private transient SchemaOperatorMetrics schemaOperatorMetrics;
+ private transient int subTaskId;
@VisibleForTesting
public SchemaOperator(List<RouteRule> routingRules) {
@@ -153,6 +151,7 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
schemaOperatorMetrics =
new SchemaOperatorMetrics(
getRuntimeContext().getMetricGroup(), schemaChangeBehavior);
+ subTaskId = getRuntimeContext().getIndexOfThisSubtask();
}
@Override
@@ -218,17 +217,6 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
});
}
- @Override
- public void initializeState(StateInitializationContext context) throws Exception {
- if (context.isRestored()) {
- // Multiple operators may appear during a restart process,
- // only clear the pendingSchemaChanges when the first operator starts.
- if (getRuntimeContext().getIndexOfThisSubtask() == 0) {
- sendRequestToCoordinator(new RefreshPendingListsRequest());
- }
- }
- }
-
/**
* This method is guaranteed to not be called concurrently with other methods of the operator.
*/
@@ -248,7 +236,11 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
private void processSchemaChangeEvents(SchemaChangeEvent event)
throws InterruptedException, TimeoutException, ExecutionException {
TableId tableId = event.tableId();
- LOG.info("Table {} received SchemaChangeEvent and start to be blocked.", tableId);
+ LOG.info(
+ "{}> Table {} received SchemaChangeEvent {} and start to be blocked.",
+ subTaskId,
+ tableId,
+ event);
handleSchemaChangeEvent(tableId, event);
// Update caches
originalSchema.put(tableId, getLatestOriginalSchema(tableId));
@@ -412,36 +404,62 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
schemaChangeEvent));
}
- // The request will need to send a FlushEvent or block until flushing finished
+ // The request will block if another schema change event is being handled
SchemaChangeResponse response = requestSchemaChange(tableId, schemaChangeEvent);
- if (!response.getSchemaChangeEvents().isEmpty()) {
- LOG.info(
- "Sending the FlushEvent for table {} in subtask {}.",
- tableId,
- getRuntimeContext().getIndexOfThisSubtask());
+ if (response.isAccepted()) {
+ LOG.info("{}> Sending the FlushEvent for table {}.", subTaskId, tableId);
output.collect(new StreamRecord<>(new FlushEvent(tableId)));
List<SchemaChangeEvent> expectedSchemaChangeEvents = response.getSchemaChangeEvents();
schemaOperatorMetrics.increaseSchemaChangeEvents(expectedSchemaChangeEvents.size());
// The request will block until flushing finished in each sink writer
- ReleaseUpstreamResponse schemaEvolveResponse = requestReleaseUpstream();
+ SchemaChangeResultResponse schemaEvolveResponse = requestSchemaChangeResult();
List<SchemaChangeEvent> finishedSchemaChangeEvents =
schemaEvolveResponse.getFinishedSchemaChangeEvents();
// Update evolved schema changes based on apply results
finishedSchemaChangeEvents.forEach(e -> output.collect(new StreamRecord<>(e)));
+ } else if (response.isDuplicate()) {
+ LOG.info(
+ "{}> Schema change event {} has been handled in another subTask already.",
+ subTaskId,
+ schemaChangeEvent);
+ } else if (response.isIgnored()) {
+ LOG.info(
+ "{}> Schema change event {} has been ignored. No schema evolution needed.",
+ subTaskId,
+ schemaChangeEvent);
+ } else {
+ throw new IllegalStateException("Unexpected response status " + response);
}
}
private SchemaChangeResponse requestSchemaChange(
- TableId tableId, SchemaChangeEvent schemaChangeEvent) {
- return sendRequestToCoordinator(new SchemaChangeRequest(tableId, schemaChangeEvent));
+ TableId tableId, SchemaChangeEvent schemaChangeEvent)
+ throws InterruptedException, TimeoutException {
+ long schemaEvolveTimeOutMillis = System.currentTimeMillis() + rpcTimeOutInMillis;
+ while (true) {
+ SchemaChangeResponse response =
+ sendRequestToCoordinator(new SchemaChangeRequest(tableId, schemaChangeEvent));
+ if (response.isRegistryBusy()) {
+ if (System.currentTimeMillis() < schemaEvolveTimeOutMillis) {
+ LOG.info(
+ "{}> Schema Registry is busy now, waiting for next request...",
+ subTaskId);
+ Thread.sleep(1000);
+ } else {
+ throw new TimeoutException("TimeOut when requesting schema change");
+ }
+ } else {
+ return response;
+ }
+ }
}
- private ReleaseUpstreamResponse requestReleaseUpstream()
+ private SchemaChangeResultResponse requestSchemaChangeResult()
throws InterruptedException, TimeoutException {
CoordinationResponse coordinationResponse =
- sendRequestToCoordinator(new ReleaseUpstreamRequest());
+ sendRequestToCoordinator(new SchemaChangeResultRequest());
long nextRpcTimeOutMillis = System.currentTimeMillis() + rpcTimeOutInMillis;
while (coordinationResponse instanceof SchemaChangeProcessingResponse) {
if (System.currentTimeMillis() < nextRpcTimeOutMillis) {
@@ -451,7 +469,7 @@ public class SchemaOperator extends AbstractStreamOperator<Event>
throw new TimeoutException("TimeOut when requesting release upstream");
}
}
- return ((ReleaseUpstreamResponse) coordinationResponse);
+ return ((SchemaChangeResultResponse) coordinationResponse);
}
private <REQUEST extends CoordinationRequest, RESPONSE extends CoordinationResponse>
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaManager.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaManager.java
index b87ef5b12..60e70b7ed 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaManager.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaManager.java
@@ -18,11 +18,15 @@
package org.apache.flink.cdc.runtime.operators.schema.coordinator;
import org.apache.flink.cdc.common.annotation.Internal;
+import org.apache.flink.cdc.common.event.AddColumnEvent;
import org.apache.flink.cdc.common.event.CreateTableEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEvent;
import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.event.visitor.SchemaChangeEventVisitor;
import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
+import org.apache.flink.cdc.common.schema.Column;
import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.types.DataType;
import org.apache.flink.cdc.common.utils.SchemaUtils;
import org.apache.flink.cdc.runtime.serializer.TableIdSerializer;
import org.apache.flink.cdc.runtime.serializer.schema.SchemaSerializer;
@@ -39,6 +43,7 @@ import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.Objects;
@@ -93,6 +98,97 @@ public class SchemaManager {
return behavior;
}
+ /**
+ * This function checks if the given schema change event has been applied already. If so, it
+ * will be ignored to avoid sending duplicate evolved schema change events to sink metadata
+ * applier.
+ */
+ public final boolean isOriginalSchemaChangeEventRedundant(SchemaChangeEvent event) {
+ TableId tableId = event.tableId();
+ Optional<Schema> latestSchema = getLatestOriginalSchema(tableId);
+ return Boolean.TRUE.equals(
+ SchemaChangeEventVisitor.visit(
+ event,
+ addColumnEvent -> {
+ // It has not been applied if schema does not even exist
+ if (!latestSchema.isPresent()) {
+ return false;
+ }
+ List<Column> existedColumns = latestSchema.get().getColumns();
+
+ // It has been applied only if all columns are present in existedColumns
+ for (AddColumnEvent.ColumnWithPosition column :
+ addColumnEvent.getAddedColumns()) {
+ if (!existedColumns.contains(column.getAddColumn())) {
+ return false;
+ }
+ }
+ return true;
+ },
+ alterColumnTypeEvent -> {
+ // It has not been applied if schema does not even exist
+ if (!latestSchema.isPresent()) {
+ return false;
+ }
+ Schema schema = latestSchema.get();
+
+ // It has been applied only if all column types are set as expected
+ for (Map.Entry<String, DataType> entry :
+ alterColumnTypeEvent.getTypeMapping().entrySet()) {
+ if (!schema.getColumn(entry.getKey()).isPresent()
+ || !schema.getColumn(entry.getKey())
+ .get()
+ .getType()
+ .equals(entry.getValue())) {
+ return false;
+ }
+ }
+ return true;
+ },
+ createTableEvent -> {
+ // It has been applied if such table already exists
+ return latestSchema.isPresent();
+ },
+ dropColumnEvent -> {
+ // It has not been applied if schema does not even exist
+ if (!latestSchema.isPresent()) {
+ return false;
+ }
+ List<String> existedColumnNames = latestSchema.get().getColumnNames();
+
+ // It has been applied only if corresponding column types do not exist
+ return dropColumnEvent.getDroppedColumnNames().stream()
+ .noneMatch(existedColumnNames::contains);
+ },
+ dropTableEvent -> {
+ // It has been applied if such table does not exist
+ return !latestSchema.isPresent();
+ },
+ renameColumnEvent -> {
+ // It has been applied if such table already exists
+ if (!latestSchema.isPresent()) {
+ return false;
+ }
+ List<String> existedColumnNames = latestSchema.get().getColumnNames();
+
+ // It has been applied only if all previous names do not exist, and all
+ // new names already exist
+ for (Map.Entry<String, String> entry :
+ renameColumnEvent.getNameMapping().entrySet()) {
+ if (existedColumnNames.contains(entry.getKey())
+ || !existedColumnNames.contains(entry.getValue())) {
+ return false;
+ }
+ }
+ return true;
+ },
+ truncateTableEvent -> {
+ // We have no way to ensure if a TruncateTableEvent has been applied
+ // before. Just assume it's not.
+ return false;
+ }));
+ }
+
public final boolean schemaExists(
Map<TableId, SortedMap<Integer, Schema>> schemaMap, TableId tableId) {
return schemaMap.containsKey(tableId) && !schemaMap.get(tableId).isEmpty();
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
index 9087ae4b3..8ea3a1f93 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistry.java
@@ -29,8 +29,6 @@ import org.apache.flink.cdc.runtime.operators.schema.event.GetEvolvedSchemaReque
import org.apache.flink.cdc.runtime.operators.schema.event.GetEvolvedSchemaResponse;
import org.apache.flink.cdc.runtime.operators.schema.event.GetOriginalSchemaRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.GetOriginalSchemaResponse;
-import org.apache.flink.cdc.runtime.operators.schema.event.RefreshPendingListsRequest;
-import org.apache.flink.cdc.runtime.operators.schema.event.ReleaseUpstreamRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeResultRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.SinkWriterRegisterEvent;
@@ -201,18 +199,14 @@ public class SchemaRegistry implements OperatorCoordinator, CoordinationRequestH
if (request instanceof SchemaChangeRequest) {
SchemaChangeRequest schemaChangeRequest = (SchemaChangeRequest) request;
return requestHandler.handleSchemaChangeRequest(schemaChangeRequest);
- } else if (request instanceof ReleaseUpstreamRequest) {
- return requestHandler.handleReleaseUpstreamRequest();
+ } else if (request instanceof SchemaChangeResultRequest) {
+ return requestHandler.getSchemaChangeResult();
} else if (request instanceof GetEvolvedSchemaRequest) {
return CompletableFuture.completedFuture(
wrap(handleGetEvolvedSchemaRequest(((GetEvolvedSchemaRequest) request))));
} else if (request instanceof GetOriginalSchemaRequest) {
return CompletableFuture.completedFuture(
wrap(handleGetOriginalSchemaRequest((GetOriginalSchemaRequest) request)));
- } else if (request instanceof SchemaChangeResultRequest) {
- return requestHandler.getSchemaChangeResult();
- } else if (request instanceof RefreshPendingListsRequest) {
- return requestHandler.refreshPendingLists();
} else {
throw new IllegalArgumentException(
"Unrecognized CoordinationRequest type: " + request);
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
index 60280ce48..99019a6b4 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java
@@ -25,6 +25,7 @@ import org.apache.flink.cdc.common.event.DropColumnEvent;
import org.apache.flink.cdc.common.event.RenameColumnEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEvent;
import org.apache.flink.cdc.common.event.SchemaChangeEventType;
+import org.apache.flink.cdc.common.event.SchemaChangeEventWithPreSchema;
import org.apache.flink.cdc.common.event.TableId;
import org.apache.flink.cdc.common.exceptions.UnsupportedSchemaChangeEventException;
import org.apache.flink.cdc.common.pipeline.SchemaChangeBehavior;
@@ -32,41 +33,36 @@ import org.apache.flink.cdc.common.schema.Column;
import org.apache.flink.cdc.common.schema.Schema;
import org.apache.flink.cdc.common.sink.MetadataApplier;
import org.apache.flink.cdc.common.types.DataType;
-import org.apache.flink.cdc.runtime.operators.schema.event.RefreshPendingListsResponse;
-import org.apache.flink.cdc.runtime.operators.schema.event.ReleaseUpstreamRequest;
-import org.apache.flink.cdc.runtime.operators.schema.event.ReleaseUpstreamResponse;
+import org.apache.flink.cdc.common.utils.Preconditions;
import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeProcessingResponse;
import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeRequest;
import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeResponse;
+import org.apache.flink.cdc.runtime.operators.schema.event.SchemaChangeResultResponse;
import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import javax.annotation.concurrent.NotThreadSafe;
-
import java.io.Closeable;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
-import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Collectors;
import java.util.stream.Stream;
-import static org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler.RequestStatus.RECEIVED_RELEASE_REQUEST;
import static org.apache.flink.cdc.runtime.operators.schema.event.CoordinationResponseUtils.wrap;
/** A handler to deal with all requests and events for {@link SchemaRegistry}. */
@Internal
-@NotThreadSafe
public class SchemaRegistryRequestHandler implements Closeable {
private static final Logger LOG = LoggerFactory.getLogger(SchemaRegistryRequestHandler.class);
@@ -80,21 +76,18 @@ public class SchemaRegistryRequestHandler implements Closeable {
private final SchemaDerivation schemaDerivation;
/**
- * Not applied SchemaChangeRequest before receiving all flush success events for its table from
- * sink writers.
+ * Atomic flag indicating if current RequestHandler could accept more schema changes for now.
*/
- private final List<PendingSchemaChange> pendingSchemaChanges;
+ private final AtomicReference<RequestStatus> schemaChangeStatus;
- private final List<SchemaChangeEvent> finishedSchemaChanges;
- private final List<SchemaChangeEvent> ignoredSchemaChanges;
+ private volatile Throwable currentChangeException;
+ private volatile List<SchemaChangeEvent> currentDerivedSchemaChangeEvents;
+ private volatile List<SchemaChangeEvent> currentFinishedSchemaChanges;
+ private volatile List<SchemaChangeEvent> currentIgnoredSchemaChanges;
/** Sink writers which have sent flush success events for the request. */
private final Set<Integer> flushedSinkWriters;
- /** Status of the execution of current schema change request. */
- private volatile boolean isSchemaChangeApplying;
- /** Actual exception if failed to apply schema change. */
- private volatile Throwable schemaChangeException;
/** Executor service to execute schema change. */
private final ExecutorService schemaChangeThreadPool;
@@ -106,16 +99,81 @@ public class SchemaRegistryRequestHandler implements Closeable {
SchemaDerivation schemaDerivation,
SchemaChangeBehavior schemaChangeBehavior) {
this.metadataApplier = metadataApplier;
- this.activeSinkWriters = new HashSet<>();
- this.flushedSinkWriters = new HashSet<>();
- this.pendingSchemaChanges = new LinkedList<>();
- this.finishedSchemaChanges = new LinkedList<>();
- this.ignoredSchemaChanges = new LinkedList<>();
this.schemaManager = schemaManager;
this.schemaDerivation = schemaDerivation;
- this.schemaChangeThreadPool = Executors.newSingleThreadExecutor();
- this.isSchemaChangeApplying = false;
this.schemaChangeBehavior = schemaChangeBehavior;
+
+ this.activeSinkWriters = new HashSet<>();
+ this.flushedSinkWriters = new HashSet<>();
+ this.schemaChangeThreadPool = Executors.newSingleThreadExecutor();
+
+ this.currentDerivedSchemaChangeEvents = new ArrayList<>();
+ this.currentFinishedSchemaChanges = new ArrayList<>();
+ this.currentIgnoredSchemaChanges = new ArrayList<>();
+ this.schemaChangeStatus = new AtomicReference<>(RequestStatus.IDLE);
+ }
+
+ /**
+ * Handle the {@link SchemaChangeRequest} and wait for all sink subtasks flushing.
+ *
+ * @param request the received SchemaChangeRequest
+ */
+ public CompletableFuture<CoordinationResponse> handleSchemaChangeRequest(
+ SchemaChangeRequest request) {
+ if (schemaChangeStatus.compareAndSet(RequestStatus.IDLE, RequestStatus.WAITING_FOR_FLUSH)) {
+ LOG.info(
+ "Received schema change event request {} from table {}. Start to buffer requests for others.",
+ request.getSchemaChangeEvent(),
+ request.getTableId().toString());
+ SchemaChangeEvent event = request.getSchemaChangeEvent();
+
+ // If this schema change event has been requested by another subTask, ignore it.
+ if (schemaManager.isOriginalSchemaChangeEventRedundant(event)) {
+ LOG.info("Event {} has been addressed before, ignoring it.", event);
+ clearCurrentSchemaChangeRequest();
+ Preconditions.checkState(
+ schemaChangeStatus.compareAndSet(
+ RequestStatus.WAITING_FOR_FLUSH, RequestStatus.IDLE),
+ "Illegal schemaChangeStatus state: should still in WAITING_FOR_FLUSH state if event was duplicated.");
+ return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.duplicate()));
+ }
+ schemaManager.applyOriginalSchemaChange(event);
+ List<SchemaChangeEvent> derivedSchemaChangeEvents =
+ calculateDerivedSchemaChangeEvents(request.getSchemaChangeEvent());
+
+ // If this schema change event is filtered out by LENIENT mode or merging table route
+ // strategies, ignore it.
+ if (derivedSchemaChangeEvents.isEmpty()) {
+ LOG.info("Event {} is omitted from sending to downstream, ignoring it.", event);
+ clearCurrentSchemaChangeRequest();
+ Preconditions.checkState(
+ schemaChangeStatus.compareAndSet(
+ RequestStatus.WAITING_FOR_FLUSH, RequestStatus.IDLE),
+ "Illegal schemaChangeStatus state: should still in WAITING_FOR_FLUSH state if event was ignored.");
+ return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.ignored()));
+ }
+
+ // Backfill pre-schema info for sink applying
+ derivedSchemaChangeEvents.forEach(
+ e -> {
+ if (e instanceof SchemaChangeEventWithPreSchema) {
+ SchemaChangeEventWithPreSchema pe = (SchemaChangeEventWithPreSchema) e;
+ if (!pe.hasPreSchema()) {
+ schemaManager
+ .getLatestEvolvedSchema(pe.tableId())
+ .ifPresent(pe::fillPreSchema);
+ }
+ }
+ });
+ currentDerivedSchemaChangeEvents = new ArrayList<>(derivedSchemaChangeEvents);
+ return CompletableFuture.completedFuture(
+ wrap(SchemaChangeResponse.accepted(derivedSchemaChangeEvents)));
+ } else {
+ LOG.info(
+ "Schema Registry is busy processing a schema change request, could not handle request {} for now.",
+ request);
+ return CompletableFuture.completedFuture(wrap(SchemaChangeResponse.busy()));
+ }
}
/**
@@ -126,27 +184,22 @@ public class SchemaRegistryRequestHandler implements Closeable {
*/
private void applySchemaChange(
TableId tableId, List<SchemaChangeEvent> derivedSchemaChangeEvents) {
- isSchemaChangeApplying = true;
- schemaChangeException = null;
- finishedSchemaChanges.clear();
- ignoredSchemaChanges.clear();
-
for (SchemaChangeEvent changeEvent : derivedSchemaChangeEvents) {
if (changeEvent.getType() != SchemaChangeEventType.CREATE_TABLE) {
if (schemaChangeBehavior == SchemaChangeBehavior.IGNORE) {
- ignoredSchemaChanges.add(changeEvent);
+ currentIgnoredSchemaChanges.add(changeEvent);
continue;
}
}
if (!metadataApplier.acceptsSchemaEvolutionType(changeEvent.getType())) {
LOG.info("Ignored schema change {} to table {}.", changeEvent, tableId);
- ignoredSchemaChanges.add(changeEvent);
+ currentIgnoredSchemaChanges.add(changeEvent);
} else {
try {
metadataApplier.applySchemaChange(changeEvent);
- LOG.debug("Applied schema change {} to table {}.", changeEvent, tableId);
+ LOG.info("Applied schema change {} to table {}.", changeEvent, tableId);
schemaManager.applyEvolvedSchemaChange(changeEvent);
- finishedSchemaChanges.add(changeEvent);
+ currentFinishedSchemaChanges.add(changeEvent);
} catch (Throwable t) {
LOG.error(
"Failed to apply schema change {} to table {}. Caused by: {}",
@@ -154,7 +207,7 @@ public class SchemaRegistryRequestHandler implements Closeable {
tableId,
t);
if (!shouldIgnoreException(t)) {
- schemaChangeException = t;
+ currentChangeException = t;
break;
} else {
LOG.warn(
@@ -165,62 +218,9 @@ public class SchemaRegistryRequestHandler implements Closeable {
}
}
}
-
- PendingSchemaChange waitFlushSuccess = pendingSchemaChanges.get(0);
- if (RECEIVED_RELEASE_REQUEST.equals(waitFlushSuccess.getStatus())) {
- startNextSchemaChangeRequest();
- }
- isSchemaChangeApplying = false;
- }
-
- /**
- * Handle the {@link SchemaChangeRequest} and wait for all sink subtasks flushing.
- *
- * @param request the received SchemaChangeRequest
- */
- public CompletableFuture<CoordinationResponse> handleSchemaChangeRequest(
- SchemaChangeRequest request) {
- if (pendingSchemaChanges.isEmpty()) {
- LOG.info(
- "Received schema change event request from table {}. Start to buffer requests for others.",
- request.getTableId().toString());
- if (request.getSchemaChangeEvent() instanceof CreateTableEvent
- && schemaManager.originalSchemaExists(request.getTableId())) {
- return CompletableFuture.completedFuture(
- wrap(new SchemaChangeResponse(Collections.emptyList())));
- }
- schemaManager.applyOriginalSchemaChange(request.getSchemaChangeEvent());
- List<SchemaChangeEvent> derivedSchemaChangeEvents =
- calculateDerivedSchemaChangeEvents(request.getSchemaChangeEvent());
- CompletableFuture<CoordinationResponse> response =
- CompletableFuture.completedFuture(
- wrap(new SchemaChangeResponse(derivedSchemaChangeEvents)));
- if (!derivedSchemaChangeEvents.isEmpty()) {
- PendingSchemaChange pendingSchemaChange =
- new PendingSchemaChange(request, response);
- pendingSchemaChange.derivedSchemaChangeEvents = derivedSchemaChangeEvents;
- pendingSchemaChanges.add(pendingSchemaChange);
- pendingSchemaChanges.get(0).startToWaitForReleaseRequest();
- }
- return response;
- } else {
- LOG.info("There are already processing requests. Wait for processing.");
- CompletableFuture<CoordinationResponse> response = new CompletableFuture<>();
- pendingSchemaChanges.add(new PendingSchemaChange(request, response));
- return response;
- }
- }
-
- /** Handle the {@link ReleaseUpstreamRequest} and wait for all sink subtasks flushing. */
- public CompletableFuture<CoordinationResponse> handleReleaseUpstreamRequest() {
- CompletableFuture<CoordinationResponse> response =
- pendingSchemaChanges.get(0).getResponseFuture();
- if (response.isDone() && !isSchemaChangeApplying) {
- startNextSchemaChangeRequest();
- } else {
- pendingSchemaChanges.get(0).receiveReleaseRequest();
- }
- return response;
+ Preconditions.checkState(
+ schemaChangeStatus.compareAndSet(RequestStatus.APPLYING, RequestStatus.FINISHED),
+ "Illegal schemaChangeStatus state: should be APPLYING before applySchemaChange finishes");
}
/**
@@ -239,76 +239,33 @@ public class SchemaRegistryRequestHandler implements Closeable {
* @param tableId the subtask in SchemaOperator and table that the FlushEvent is about
* @param sinkSubtask the sink subtask succeed flushing
*/
- public void flushSuccess(TableId tableId, int sinkSubtask) throws InterruptedException {
+ public void flushSuccess(TableId tableId, int sinkSubtask) {
flushedSinkWriters.add(sinkSubtask);
if (flushedSinkWriters.equals(activeSinkWriters)) {
+ Preconditions.checkState(
+ schemaChangeStatus.compareAndSet(
+ RequestStatus.WAITING_FOR_FLUSH, RequestStatus.APPLYING),
+ "Illegal schemaChangeStatus state: should be WAITING_FOR_FLUSH before collecting enough FlushEvents");
LOG.info(
"All sink subtask have flushed for table {}. Start to apply schema change.",
tableId.toString());
- PendingSchemaChange waitFlushSuccess = pendingSchemaChanges.get(0);
schemaChangeThreadPool.submit(
- () -> applySchemaChange(tableId, waitFlushSuccess.derivedSchemaChangeEvents));
- Thread.sleep(1000);
-
- if (schemaChangeException != null) {
- throw new RuntimeException("Failed to apply schema change.", schemaChangeException);
- }
-
- if (isSchemaChangeApplying) {
- waitFlushSuccess
- .getResponseFuture()
- .complete(wrap(new SchemaChangeProcessingResponse()));
- } else {
- waitFlushSuccess
- .getResponseFuture()
- .complete(wrap(new ReleaseUpstreamResponse(finishedSchemaChanges)));
- }
+ () -> applySchemaChange(tableId, currentDerivedSchemaChangeEvents));
}
}
- private void startNextSchemaChangeRequest() {
- pendingSchemaChanges.remove(0);
- flushedSinkWriters.clear();
- while (!pendingSchemaChanges.isEmpty()) {
- PendingSchemaChange pendingSchemaChange = pendingSchemaChanges.get(0);
- SchemaChangeRequest request = pendingSchemaChange.changeRequest;
- if (request.getSchemaChangeEvent() instanceof CreateTableEvent
- && schemaManager.evolvedSchemaExists(request.getTableId())) {
- pendingSchemaChange
- .getResponseFuture()
- .complete(wrap(new SchemaChangeResponse(Collections.emptyList())));
- pendingSchemaChanges.remove(0);
- } else {
- List<SchemaChangeEvent> derivedSchemaChangeEvents =
- calculateDerivedSchemaChangeEvents(request.getSchemaChangeEvent());
- pendingSchemaChange
- .getResponseFuture()
- .complete(wrap(new SchemaChangeResponse(derivedSchemaChangeEvents)));
- if (!derivedSchemaChangeEvents.isEmpty()) {
- pendingSchemaChange.derivedSchemaChangeEvents = derivedSchemaChangeEvents;
- pendingSchemaChange.startToWaitForReleaseRequest();
- break;
- }
- }
- }
- }
-
- public CompletableFuture<CoordinationResponse> refreshPendingLists() {
- pendingSchemaChanges.clear();
- flushedSinkWriters.clear();
- return CompletableFuture.completedFuture(wrap(new RefreshPendingListsResponse()));
- }
-
public CompletableFuture<CoordinationResponse> getSchemaChangeResult() {
- if (schemaChangeException != null) {
- throw new RuntimeException("Failed to apply schema change.", schemaChangeException);
- }
-
- if (isSchemaChangeApplying) {
- return CompletableFuture.supplyAsync(() -> wrap(new SchemaChangeProcessingResponse()));
- } else {
+ Preconditions.checkState(
+ !schemaChangeStatus.get().equals(RequestStatus.IDLE),
+ "Illegal schemaChangeStatus: should not be IDLE before getting schema change request results.");
+ if (schemaChangeStatus.compareAndSet(RequestStatus.FINISHED, RequestStatus.IDLE)) {
+ // This request has been finished, return it and prepare for the next request
+ List<SchemaChangeEvent> finishedEvents = clearCurrentSchemaChangeRequest();
return CompletableFuture.supplyAsync(
- () -> wrap(new ReleaseUpstreamResponse(finishedSchemaChanges)));
+ () -> wrap(new SchemaChangeResultResponse(finishedEvents)));
+ } else {
+ // Still working on schema change request, waiting it
+ return CompletableFuture.supplyAsync(() -> wrap(new SchemaChangeProcessingResponse()));
}
}
@@ -422,63 +379,64 @@ public class SchemaRegistryRequestHandler implements Closeable {
}
return events;
}
+ case DROP_TABLE:
+ // We don't drop any tables in Lenient mode.
+ LOG.info("A drop table event {} has been ignored in Lenient mode.", event);
+ return Collections.emptyList();
default:
return Collections.singletonList(event);
}
}
private boolean shouldIgnoreException(Throwable throwable) {
-
// In IGNORE mode, will never try to apply schema change events
- // In EVOLVE and and LENIENT mode, such failure will not be tolerated
+ // In EVOLVE and LENIENT mode, such failure will not be tolerated
// In EXCEPTION mode, an exception will be thrown once captured
return (throwable instanceof UnsupportedSchemaChangeEventException)
&& (schemaChangeBehavior == SchemaChangeBehavior.TRY_EVOLVE);
}
- private static class PendingSchemaChange {
- private final SchemaChangeRequest changeRequest;
- private List<SchemaChangeEvent> derivedSchemaChangeEvents;
- private CompletableFuture<CoordinationResponse> responseFuture;
- private RequestStatus status;
-
- public PendingSchemaChange(
- SchemaChangeRequest changeRequest,
- CompletableFuture<CoordinationResponse> responseFuture) {
- this.changeRequest = changeRequest;
- this.responseFuture = responseFuture;
- this.status = RequestStatus.PENDING;
- }
-
- public SchemaChangeRequest getChangeRequest() {
- return changeRequest;
- }
-
- public CompletableFuture<CoordinationResponse> getResponseFuture() {
- return responseFuture;
- }
-
- public RequestStatus getStatus() {
- return status;
- }
-
- public void startToWaitForReleaseRequest() {
- if (!responseFuture.isDone()) {
- throw new IllegalStateException(
- "Cannot start to wait for flush success before the SchemaChangeRequest is done.");
- }
- this.responseFuture = new CompletableFuture<>();
- this.status = RequestStatus.WAIT_RELEASE_REQUEST;
- }
-
- public void receiveReleaseRequest() {
- this.status = RECEIVED_RELEASE_REQUEST;
+ private List<SchemaChangeEvent> clearCurrentSchemaChangeRequest() {
+ if (currentChangeException != null) {
+ throw new RuntimeException("Failed to apply schema change.", currentChangeException);
}
+ List<SchemaChangeEvent> finishedSchemaChanges =
+ new ArrayList<>(currentFinishedSchemaChanges);
+ flushedSinkWriters.clear();
+ currentDerivedSchemaChangeEvents.clear();
+ currentFinishedSchemaChanges.clear();
+ currentIgnoredSchemaChanges.clear();
+ currentChangeException = null;
+ return finishedSchemaChanges;
}
- enum RequestStatus {
- PENDING,
- WAIT_RELEASE_REQUEST,
- RECEIVED_RELEASE_REQUEST
+ // Schema change event state could transfer in the following way:
+ //
+ // -------- B --------
+ // | |
+ // v |
+ // -------- ---------------------
+ // | IDLE | --- A --> | WAITING_FOR_FLUSH |
+ // -------- ---------------------
+ // ^ |
+ // E C
+ // \ v
+ // ------------ ------------
+ // | FINISHED | <-- D -- | APPLYING |
+ // ------------ ------------
+ //
+ // A: When a request came to an idling request handler.
+ // B: When current request is duplicate or ignored by LENIENT / routed table merging
+ // strategies.
+ // C: When schema registry collected enough flush success events, and actually started to apply
+ // schema changes.
+ // D: When schema change application finishes (successfully or with exceptions)
+ // E: When current schema change request result has been retrieved by SchemaOperator, and ready
+ // for the next request.
+ private enum RequestStatus {
+ IDLE,
+ WAITING_FOR_FLUSH,
+ APPLYING,
+ FINISHED
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/RefreshPendingListsRequest.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/RefreshPendingListsRequest.java
deleted file mode 100644
index a0496c935..000000000
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/RefreshPendingListsRequest.java
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.runtime.operators.schema.event;
-
-import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler;
-import org.apache.flink.runtime.operators.coordination.CoordinationRequest;
-
-/** Request to refresh the pendingSchemaChanges of {@link SchemaRegistryRequestHandler}. */
-public class RefreshPendingListsRequest implements CoordinationRequest {
-
- private static final long serialVersionUID = 1L;
-}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/RefreshPendingListsResponse.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/RefreshPendingListsResponse.java
deleted file mode 100644
index ff0221deb..000000000
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/RefreshPendingListsResponse.java
+++ /dev/null
@@ -1,26 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.runtime.operators.schema.event;
-
-import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
-
-/** Response to refresh the pendingSchemaChanges of {@link RefreshPendingListsRequest}. */
-public class RefreshPendingListsResponse implements CoordinationResponse {
-
- private static final long serialVersionUID = 1L;
-}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ReleaseUpstreamRequest.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ReleaseUpstreamRequest.java
deleted file mode 100644
index 3da85c184..000000000
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ReleaseUpstreamRequest.java
+++ /dev/null
@@ -1,32 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.cdc.runtime.operators.schema.event;
-
-import org.apache.flink.cdc.common.event.FlushEvent;
-import org.apache.flink.cdc.runtime.operators.schema.SchemaOperator;
-import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry;
-import org.apache.flink.runtime.operators.coordination.CoordinationRequest;
-
-/**
- * The request from {@link SchemaOperator} to {@link SchemaRegistry} to request to release upstream
- * after sending {@link FlushEvent}.
- */
-public class ReleaseUpstreamRequest implements CoordinationRequest {
-
- private static final long serialVersionUID = 1L;
-}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeProcessingResponse.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeProcessingResponse.java
index b679ab9e5..bf8d84f35 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeProcessingResponse.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeProcessingResponse.java
@@ -23,8 +23,8 @@ import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry;
import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
/**
- * The response for {@link SchemaChangeResultRequest} or {@link ReleaseUpstreamRequest} from {@link
- * SchemaRegistry} to {@link SchemaOperator} if not apply {@link SchemaChangeEvent} in time.
+ * The response for {@link SchemaChangeResultRequest} from {@link SchemaRegistry} to {@link
+ * SchemaOperator} if not apply {@link SchemaChangeEvent} in time.
*/
public class SchemaChangeProcessingResponse implements CoordinationResponse {
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeResponse.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeResponse.java
index 142de431e..63d57139b 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeResponse.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeResponse.java
@@ -22,6 +22,7 @@ import org.apache.flink.cdc.runtime.operators.schema.SchemaOperator;
import org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry;
import org.apache.flink.runtime.operators.coordination.CoordinationResponse;
+import java.util.Collections;
import java.util.List;
import java.util.Objects;
@@ -38,8 +39,44 @@ public class SchemaChangeResponse implements CoordinationResponse {
*/
private final List<SchemaChangeEvent> schemaChangeEvents;
- public SchemaChangeResponse(List<SchemaChangeEvent> schemaChangeEvents) {
+ private final ResponseCode responseCode;
+
+ public static SchemaChangeResponse accepted(List<SchemaChangeEvent> schemaChangeEvents) {
+ return new SchemaChangeResponse(schemaChangeEvents, ResponseCode.ACCEPTED);
+ }
+
+ public static SchemaChangeResponse busy() {
+ return new SchemaChangeResponse(Collections.emptyList(), ResponseCode.BUSY);
+ }
+
+ public static SchemaChangeResponse duplicate() {
+ return new SchemaChangeResponse(Collections.emptyList(), ResponseCode.DUPLICATE);
+ }
+
+ public static SchemaChangeResponse ignored() {
+ return new SchemaChangeResponse(Collections.emptyList(), ResponseCode.IGNORED);
+ }
+
+ private SchemaChangeResponse(
+ List<SchemaChangeEvent> schemaChangeEvents, ResponseCode responseCode) {
this.schemaChangeEvents = schemaChangeEvents;
+ this.responseCode = responseCode;
+ }
+
+ public boolean isAccepted() {
+ return ResponseCode.ACCEPTED.equals(responseCode);
+ }
+
+ public boolean isRegistryBusy() {
+ return ResponseCode.BUSY.equals(responseCode);
+ }
+
+ public boolean isDuplicate() {
+ return ResponseCode.DUPLICATE.equals(responseCode);
+ }
+
+ public boolean isIgnored() {
+ return ResponseCode.IGNORED.equals(responseCode);
}
public List<SchemaChangeEvent> getSchemaChangeEvents() {
@@ -55,11 +92,43 @@ public class SchemaChangeResponse implements CoordinationResponse {
return false;
}
SchemaChangeResponse response = (SchemaChangeResponse) o;
- return schemaChangeEvents.equals(response.schemaChangeEvents);
+ return Objects.equals(schemaChangeEvents, response.schemaChangeEvents)
+ && responseCode == response.responseCode;
}
@Override
public int hashCode() {
- return Objects.hash(schemaChangeEvents);
+ return Objects.hash(schemaChangeEvents, responseCode);
+ }
+
+ @Override
+ public String toString() {
+ return "SchemaChangeResponse{"
+ + "schemaChangeEvents="
+ + schemaChangeEvents
+ + ", responseCode="
+ + responseCode
+ + '}';
+ }
+
+ /**
+ * Schema Change Response status code.
+ *
+ * <p>- Accepted: Requested schema change request has been accepted exclusively. Any other
+ * schema change requests will be blocked.
+ *
+ * <p>- Busy: Schema registry is currently busy processing another schema change request.
+ *
+ * <p>- Duplicate: This schema change request has been submitted before, possibly by another
+ * paralleled subTask.
+ *
+ * <p>- Ignored: This schema change request has been assessed, but no actual evolution is
+ * required. Possibly caused by LENIENT mode or merging table strategies.
+ */
+ public enum ResponseCode {
+ ACCEPTED,
+ BUSY,
+ DUPLICATE,
+ IGNORED
}
}
diff --git a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ReleaseUpstreamResponse.java b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeResultResponse.java
similarity index 87%
rename from flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ReleaseUpstreamResponse.java
rename to flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeResultResponse.java
index bea880962..7039ef086 100644
--- a/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/ReleaseUpstreamResponse.java
+++ b/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/event/SchemaChangeResultResponse.java
@@ -26,10 +26,10 @@ import java.util.List;
import java.util.Objects;
/**
- * The response for {@link ReleaseUpstreamRequest} from {@link SchemaRegistry} to {@link
+ * The response for {@link SchemaChangeResultRequest} from {@link SchemaRegistry} to {@link
* SchemaOperator}.
*/
-public class ReleaseUpstreamResponse implements CoordinationResponse {
+public class SchemaChangeResultResponse implements CoordinationResponse {
private static final long serialVersionUID = 1L;
@@ -39,7 +39,7 @@ public class ReleaseUpstreamResponse implements CoordinationResponse {
*/
private final List<SchemaChangeEvent> finishedSchemaChangeEvents;
- public ReleaseUpstreamResponse(List<SchemaChangeEvent> finishedSchemaChangeEvents) {
+ public SchemaChangeResultResponse(List<SchemaChangeEvent> finishedSchemaChangeEvents) {
this.finishedSchemaChangeEvents = finishedSchemaChangeEvents;
}
@@ -63,7 +63,7 @@ public class ReleaseUpstreamResponse implements CoordinationResponse {
if (object == null || getClass() != object.getClass()) {
return false;
}
- ReleaseUpstreamResponse that = (ReleaseUpstreamResponse) object;
+ SchemaChangeResultResponse that = (SchemaChangeResultResponse) object;
return Objects.equals(finishedSchemaChangeEvents, that.finishedSchemaChangeEvents);
}
diff --git a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
index 354304599..60952b424 100644
--- a/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
+++ b/flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/testutils/operators/EventOperatorTestHarness.java
@@ -163,7 +163,6 @@ public class EventOperatorTestHarness<OP extends AbstractStreamOperator<E>, E ex
}
public void registerTableSchema(TableId tableId, Schema schema) {
- schemaRegistry.handleApplyOriginalSchemaChangeEvent(new CreateTableEvent(tableId, schema));
schemaRegistry.handleCoordinationRequest(
new SchemaChangeRequest(tableId, new CreateTableEvent(tableId, schema)));
schemaRegistry.handleApplyEvolvedSchemaChangeRequest(new CreateTableEvent(tableId, schema));
(flink-cdc) 01/12: [FLINK-36082][pipeline-connector][kafka] Fix lamda NotSerializableException in KafkaDataSink
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 74f58cd31757970c6d784aa7982275cd18583b55
Author: Kunni <lv...@alibaba-inc.com>
AuthorDate: Mon Aug 19 21:20:01 2024 +0800
[FLINK-36082][pipeline-connector][kafka] Fix lamda NotSerializableException in KafkaDataSink
This closes #3549.
---
.../cdc/connectors/kafka/sink/KafkaDataSink.java | 35 +------------
.../kafka/sink/KafkaMetaDataApplier.java | 57 ++++++++++++++++++++++
2 files changed, 58 insertions(+), 34 deletions(-)
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/main/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSink.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/main/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSink.java
index 2dfc021b3..51e8b180d 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/main/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSink.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/main/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaDataSink.java
@@ -19,9 +19,6 @@ package org.apache.flink.cdc.connectors.kafka.sink;
import org.apache.flink.api.common.serialization.SerializationSchema;
import org.apache.flink.cdc.common.event.Event;
-import org.apache.flink.cdc.common.event.SchemaChangeEvent;
-import org.apache.flink.cdc.common.event.SchemaChangeEventType;
-import org.apache.flink.cdc.common.event.SchemaChangeEventTypeFamily;
import org.apache.flink.cdc.common.sink.DataSink;
import org.apache.flink.cdc.common.sink.EventSinkProvider;
import org.apache.flink.cdc.common.sink.FlinkSinkProvider;
@@ -33,10 +30,7 @@ import org.apache.flink.connector.kafka.sink.KafkaSinkBuilder;
import org.apache.kafka.clients.producer.ProducerConfig;
import java.time.ZoneId;
-import java.util.Arrays;
import java.util.Properties;
-import java.util.Set;
-import java.util.stream.Collectors;
/** A {@link DataSink} for "Kafka" connector. */
public class KafkaDataSink implements DataSink {
@@ -104,33 +98,6 @@ public class KafkaDataSink implements DataSink {
@Override
public MetadataApplier getMetadataApplier() {
- return new MetadataApplier() {
-
- private Set<SchemaChangeEventType> enabledEventTypes =
- Arrays.stream(SchemaChangeEventTypeFamily.ALL).collect(Collectors.toSet());
-
- @Override
- public MetadataApplier setAcceptedSchemaEvolutionTypes(
- Set<SchemaChangeEventType> schemaEvolutionTypes) {
- enabledEventTypes = schemaEvolutionTypes;
- return this;
- }
-
- @Override
- public boolean acceptsSchemaEvolutionType(SchemaChangeEventType schemaChangeEventType) {
- return enabledEventTypes.contains(schemaChangeEventType);
- }
-
- @Override
- public Set<SchemaChangeEventType> getSupportedSchemaEvolutionTypes() {
- // All schema change events are supported.
- return Arrays.stream(SchemaChangeEventTypeFamily.ALL).collect(Collectors.toSet());
- }
-
- @Override
- public void applySchemaChange(SchemaChangeEvent schemaChangeEvent) {
- // simply do nothing here because Kafka do not maintain the schemas.
- }
- };
+ return new KafkaMetaDataApplier();
}
}
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/main/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaMetaDataApplier.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/main/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaMetaDataApplier.java
new file mode 100644
index 000000000..839dd9ea9
--- /dev/null
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-kafka/src/main/java/org/apache/flink/cdc/connectors/kafka/sink/KafkaMetaDataApplier.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.kafka.sink;
+
+import org.apache.flink.cdc.common.event.SchemaChangeEvent;
+import org.apache.flink.cdc.common.event.SchemaChangeEventType;
+import org.apache.flink.cdc.common.event.SchemaChangeEventTypeFamily;
+import org.apache.flink.cdc.common.sink.MetadataApplier;
+
+import java.util.Arrays;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/** Supports {@link KafkaDataSink} to schema evolution. */
+public class KafkaMetaDataApplier implements MetadataApplier {
+
+ private Set<SchemaChangeEventType> enabledEventTypes =
+ Arrays.stream(SchemaChangeEventTypeFamily.ALL).collect(Collectors.toSet());
+
+ @Override
+ public MetadataApplier setAcceptedSchemaEvolutionTypes(
+ Set<SchemaChangeEventType> schemaEvolutionTypes) {
+ enabledEventTypes = schemaEvolutionTypes;
+ return this;
+ }
+
+ @Override
+ public boolean acceptsSchemaEvolutionType(SchemaChangeEventType schemaChangeEventType) {
+ return enabledEventTypes.contains(schemaChangeEventType);
+ }
+
+ @Override
+ public Set<SchemaChangeEventType> getSupportedSchemaEvolutionTypes() {
+ // All schema change events are supported.
+ return Arrays.stream(SchemaChangeEventTypeFamily.ALL).collect(Collectors.toSet());
+ }
+
+ @Override
+ public void applySchemaChange(SchemaChangeEvent schemaChangeEvent) {
+ // simply do nothing here because Kafka do not maintain the schemas.
+ }
+}
(flink-cdc) 02/12: [FLINK-36088][pipeline-connector][paimon] Fix NPE in BucketAssignOperator when job restoration
Posted by le...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch release-3.2
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
commit 568888af7513db3eeb817c7e27e27bd8a9063561
Author: Kunni <lv...@alibaba-inc.com>
AuthorDate: Tue Aug 20 00:50:22 2024 +0800
[FLINK-36088][pipeline-connector][paimon] Fix NPE in BucketAssignOperator when job restoration
This closes #3553
(cherry picked from commit 7f08c6caba082915b9684ea384363f1df85dab8d)
---
.../cdc/connectors/paimon/sink/v2/bucket/BucketAssignOperator.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/v2/bucket/BucketAssignOperator.java b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/v2/bucket/BucketAssignOperator.java
index 07509574c..358d7c36c 100644
--- a/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/v2/bucket/BucketAssignOperator.java
+++ b/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/v2/bucket/BucketAssignOperator.java
@@ -130,7 +130,7 @@ public class BucketAssignOperator extends AbstractStreamOperator<Event>
if (event instanceof DataChangeEvent) {
DataChangeEvent dataChangeEvent = (DataChangeEvent) event;
- if (schemaMaps.containsKey(dataChangeEvent.tableId())) {
+ if (!schemaMaps.containsKey(dataChangeEvent.tableId())) {
Optional<Schema> schema =
schemaEvolutionClient.getLatestEvolvedSchema(dataChangeEvent.tableId());
if (schema.isPresent()) {