[jira] [Created] (FLINK-41107) schema.change.behavior: lenient or exclude.schema.changes is ignored by Hudi and MaxCompute sinks because their MetadataApplier implementations do not honor accepted schema evolution types

["Xin Chen (Jira)" <ji...@apache.org>]

Xin Chen created FLINK-41107:
--------------------------------

             Summary: schema.change.behavior: lenient or  exclude.schema.changes is ignored by Hudi and MaxCompute sinks because their MetadataApplier implementations do not honor accepted schema evolution types
                 Key: FLINK-41107
                 URL: https://issues.apache.org/jira/browse/FLINK-41107
             Project: Flink
          Issue Type: Bug
          Components: Flink CDC
    Affects Versions: cdc-3.7.0, cdc-3.6.0
            Reporter: Xin Chen


HudiMetadataApplier and MaxComputeMetadataApplier do not honor the schema evolution event types configured through include.schema.changes and exclude.schema.changes.

In the Pipeline runtime, the resolved allowed schema change event types are passed to each sink through:

 
{code:java}
MetadataApplier#setAcceptedSchemaEvolutionTypes(...){code}
 

 

Before forwarding an event to the sink, SchemaDerivator checks:

 
{code:java}
metadataApplier.acceptsSchemaEvolutionType(schemaChangeEvent.getType()){code}
 

However, both Hudi and MaxCompute metadata appliers rely on the default methods in MetadataApplier:

 
{code:java}
default MetadataApplier setAcceptedSchemaEvolutionTypes(
        Set<SchemaChangeEventType> schemaEvolutionTypes) {
    return this;
}
default boolean acceptsSchemaEvolutionType(
        SchemaChangeEventType schemaChangeEventType) {
    return true;
}
{code}
 

As a result, the configured accepted event types are discarded and every schema change event is accepted.

This makes configurations such as the following ineffective:

 
{code:java}
sink:
  exclude.schema.changes:
    - truncate.table
    - drop.table{code}
or
{code:java}
schema.change.behavior: lenient{code}
 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)