You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "MOBIN-F (via GitHub)" <gi...@apache.org> on 2025/02/19 06:45:44 UTC
[PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
MOBIN-F opened a new pull request, #3925:
URL: https://github.com/apache/flink-cdc/pull/3925
Repeat step：
1. ALTER TABLE table_name add COLUMN new_first_column bigint first
2. throw ArrayIndexOutOfBoundsException(Because the [add column first] event will be converted into an [add column before] event, ArrayIndexOutOfBoundsException is thrown when getting the existingColumnIndex)
log:
```
2025-02-19 11:32:16,586 INFO org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator [] - All sink subtask have flushed for table dw_app.cdc_sink19. Start to apply schema change request:
SchemaChangeRequest{tableId=dw_app.cdc_sink19, schemaChangeEvent=AddColumnEvent{tableId=dw_app.cdc_sink19, addedColumns=[ColumnWithPosition{column=`new_first_column` BIGINT, position=BEFORE, existedColumnName=first_column}]}, subTaskId=0}
that extracts to:
AddColumnEvent{tableId=rt_ods.cdc_sink19_add_column_EVOLVE4, addedColumns=[ColumnWithPosition{column=`new_first_column` BIGINT, position=BEFORE, existedColumnName=first_column}]}
2025-02-19 11:32:16,716 ERROR org.apache.flink.cdc.runtime.operators.schema.common.SchemaRegistry [] - An exception was triggered from Schema change applying task. Job will fail now.
org.apache.flink.util.FlinkRuntimeException: Failed to apply schema change event.
at org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.lambda$startSchemaChangesEvolve$0(SchemaCoordinator.java:290) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(ArrayList.java:418) ~[?:1.8.0_65]
at java.util.ArrayList.get(ArrayList.java:431) ~[?:1.8.0_65]
at org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumnWithBeforePosition(PaimonMetadataApplier.java:275) ~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumnEventWithPosition(PaimonMetadataApplier.java:237) ~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applyAddColumn(PaimonMetadataApplier.java:206) ~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.lambda$applySchemaChange$0(PaimonMetadataApplier.java:127) ~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at org.apache.flink.cdc.common.event.visitor.SchemaChangeEventVisitor.visit(SchemaChangeEventVisitor.java:47) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
at org.apache.flink.cdc.connectors.paimon.sink.PaimonMetadataApplier.applySchemaChange(PaimonMetadataApplier.java:124) ~[flink-cdc-pipeline-connector-paimon-3.3.0.jar:3.3.0]
at org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.applyAndUpdateEvolvedSchemaChange(SchemaCoordinator.java:434) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
at org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.applySchemaChange(SchemaCoordinator.java:401) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
at org.apache.flink.cdc.runtime.operators.schema.regular.SchemaCoordinator.lambda$startSchemaChangesEvolve$0(SchemaCoordinator.java:288) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
... 5 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
Posted by "MOBIN-F (via GitHub)" <gi...@apache.org>.
MOBIN-F commented on code in PR #3925:
URL: https://github.com/apache/flink-cdc/pull/3925#discussion_r1961119229
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplierTest.java:
##########
@@ -463,12 +463,20 @@ public void testAddColumnWithPosition(String metastore)
"col7_after", org.apache.flink.cdc.common.types.DataTypes.STRING()),
"col2"));
+ addedColumns.add(
+ AddColumnEvent.before(
+ Column.physicalColumn(
+ "col4_first_before",
+ org.apache.flink.cdc.common.types.DataTypes.STRING()),
+ "col1"));
Review Comment:
col4_first already exists, so I added the _before suffix to distinguish it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #3925:
URL: https://github.com/apache/flink-cdc/pull/3925#discussion_r1961112111
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplierTest.java:
##########
@@ -463,12 +463,20 @@ public void testAddColumnWithPosition(String metastore)
"col7_after", org.apache.flink.cdc.common.types.DataTypes.STRING()),
"col2"));
+ addedColumns.add(
+ AddColumnEvent.before(
+ Column.physicalColumn(
+ "col4_first_before",
+ org.apache.flink.cdc.common.types.DataTypes.STRING()),
+ "col1"));
Review Comment:
col4_first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
Posted by "MOBIN-F (via GitHub)" <gi...@apache.org>.
MOBIN-F commented on code in PR #3925:
URL: https://github.com/apache/flink-cdc/pull/3925#discussion_r1961136374
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplierTest.java:
##########
@@ -463,12 +463,20 @@ public void testAddColumnWithPosition(String metastore)
"col7_after", org.apache.flink.cdc.common.types.DataTypes.STRING()),
"col2"));
+ addedColumns.add(
+ AddColumnEvent.before(
+ Column.physicalColumn(
+ "col4_first_before",
+ org.apache.flink.cdc.common.types.DataTypes.STRING()),
+ "col1"));
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on PR #3925:
URL: https://github.com/apache/flink-cdc/pull/3925#issuecomment-2693209226
@leonardBang @ruanhang1993 could you help to check about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
Posted by "MOBIN-F (via GitHub)" <gi...@apache.org>.
MOBIN-F commented on code in PR #3925:
URL: https://github.com/apache/flink-cdc/pull/3925#discussion_r1967096754
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplierTest.java:
##########
@@ -441,6 +441,13 @@ public void testAddColumnWithPosition(String metastore)
tableSchema, catalog.getTable(Identifier.fromString("test.table1")).rowType());
addedColumns.clear();
+
+ addedColumns.add(
+ AddColumnEvent.before(
+ Column.physicalColumn(
+ "col4_first_before",
+ org.apache.flink.cdc.common.types.DataTypes.STRING()),
+ "col1"));
Review Comment:
The before column is actually derived in [schemaUtils](https://github.com/apache/flink-cdc/blob/e7964b160c1df3861571ba2199ba04d91d887ff2/flink-cdc-common/src/main/java/org/apache/flink/cdc/common/utils/SchemaUtils.java#L239). If the transform is defined in the pipeline
```
transform:
- source-table: dw_app.test_table
projection: __table_name__ as table_name,\*
primary-keys: id
```
At this time, when add column first is executed in the mysql table, the [add column first] event will become the [add column before] event.
If no transform is defined, add first first event remains the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #3925:
URL: https://github.com/apache/flink-cdc/pull/3925#discussion_r1967018293
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplierTest.java:
##########
@@ -441,6 +441,13 @@ public void testAddColumnWithPosition(String metastore)
tableSchema, catalog.getTable(Identifier.fromString("test.table1")).rowType());
addedColumns.clear();
+
+ addedColumns.add(
+ AddColumnEvent.before(
+ Column.physicalColumn(
+ "col4_first_before",
+ org.apache.flink.cdc.common.types.DataTypes.STRING()),
+ "col1"));
Review Comment:
So the before column is actually deduced by SchemaRegistry?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
Posted by "MOBIN-F (via GitHub)" <gi...@apache.org>.
MOBIN-F commented on code in PR #3925:
URL: https://github.com/apache/flink-cdc/pull/3925#discussion_r1961136374
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplierTest.java:
##########
@@ -463,12 +463,20 @@ public void testAddColumnWithPosition(String metastore)
"col7_after", org.apache.flink.cdc.common.types.DataTypes.STRING()),
"col2"));
+ addedColumns.add(
+ AddColumnEvent.before(
+ Column.physicalColumn(
+ "col4_first_before",
+ org.apache.flink.cdc.common.types.DataTypes.STRING()),
+ "col1"));
Review Comment:
fix~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #3925:
URL: https://github.com/apache/flink-cdc/pull/3925#discussion_r1961126616
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/test/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplierTest.java:
##########
@@ -463,12 +463,20 @@ public void testAddColumnWithPosition(String metastore)
"col7_after", org.apache.flink.cdc.common.types.DataTypes.STRING()),
"col2"));
+ addedColumns.add(
+ AddColumnEvent.before(
+ Column.physicalColumn(
+ "col4_first_before",
+ org.apache.flink.cdc.common.types.DataTypes.STRING()),
+ "col1"));
Review Comment:
The first column here is already col4_first, right? If col1 is used, then col4_first_fefore should be the second column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37348][pipeline-paimon] fix paimon ArrayIndexOutOfBoundsException when add column first [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang merged PR #3925:
URL: https://github.com/apache/flink-cdc/pull/3925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org