[jira] [Created] (FLINK-36580) Using debezium.column.exclude.list causes job failure

["MOBIN (Jira)" <ji...@apache.org>]

MOBIN created FLINK-36580:
-----------------------------

             Summary: Using debezium.column.exclude.list causes job failure
                 Key: FLINK-36580
                 URL: https://issues.apache.org/jira/browse/FLINK-36580
             Project: Flink
          Issue Type: Bug
          Components: Flink CDC
    Affects Versions: cdc-3.2.0
            Reporter: MOBIN


When the user uses \* (too many columns) when synchronizing data and wants to ignore some privacy columns, the pipeline yml is as follows:
{code:java}
source:
  type: mysql
  ....
  debezium.column.exclude.list: test_db.test_table.name

transform:
  - source-table: test_db.test_table
  projection: \*
  description: project fields and filter {code}
an error will be reported:
{code:java}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 33370909
at org.apache.flink.cdc.common.data.binary.BinarySegmentUtils.getLongMultiSegments(BinarySegmentUtils.java:731)
at org.apache.flink.cdc.common.data.binary.BinarySegmentUtils.getLong(BinarySegmentUtils.java:721) 
at org.apache.flink.cdc.common.data.binary.BinarySegmentUtils.readTimestampData(BinarySegmentUtils.java:1017) 
at org.apache.flink.cdc.common. data.binary.BinaryRecordData.getTimestamp(BinaryRecordData.java:173) 
at org.apache.flink.cdc.common.data.RecordData.lambda$createFieldGetter$f6fd429a$1(RecordData.java:214) 
at org.apache.flink.cdc.common.data.RecordData.lambda$createFieldGetter$79d0aaaa$1(RecordData.java:245) 
at org.apache.flink.cdc.runtime.operators.transform.PreTransformProcessor.getValueFromBinaryRecordData(PreTransformProcessor.java:111) 
at org.apache.flink.cdc.run time.operators.transform.PreTransformProcessor.processFillDataField(PreTransformProcessor.java:90) 
at org.apache.flink.cdc.runtime.operators.transform.PreTransformOperator.processProjection(PreTransformOperator.java:427) 
at org.apache.flink.cdc.runtime.operators.transform.PreTransformOperator.processDataChangeEvent(PreTransformOperator.java:399) 
at org.apache.flink.cdc.runtime.operators.transform.PreTransformOperator.processElement(PreTransformOperator.java:251) 
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushTo Operator(CopyingChainingOutput.java:75) 
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50) 
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29) 
at org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:310)
at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110) {code}
Cause of error: debezium will ignore the debezium.column.exclude.list field when collecting data, but cdc does not take this into account when obtaining the table schema, resulting in the inconsistency between the number of columns in tableChangeInfo.getSourceSchema() and the number of columns in BinaryRecordData

Temporary solution: projection: \*, cast(null as STRING) as name



--
This message was sent by Atlassian Jira
(v8.20.10#820010)