You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "lvyanquan (via GitHub)" <gi...@apache.org> on 2024/08/29 11:13:42 UTC
[PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
lvyanquan opened a new pull request, #3590:
URL: https://github.com/apache/flink-cdc/pull/3590
Support parsing SQL like following:
```
CREATE TABLE `user_log_copy` LIKE `user_log`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590#discussion_r1832247695
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/parser/CustomAlterTableParserListener.java:
##########
@@ -76,6 +77,29 @@ public CustomAlterTableParserListener(
this.changes = changes;
}
+ @Override
+ public void exitCopyCreateTable(MySqlParser.CopyCreateTableContext ctx) {
+ TableId tableId = parser.parseQualifiedTableId(ctx.tableName(0).fullId());
+ TableId originalTableId = parser.parseQualifiedTableId(ctx.tableName(1).fullId());
+ Table original = parser.databaseTables().forTable(originalTableId);
Review Comment:
Do we need to consider some config options that makes the parser only parse captured table list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590#discussion_r2354908980
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlPipelineITCase.java:
##########
Review Comment:
The MySqlDataSource here already used regular expressions for matching newly added tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "ruanhang1993 (via GitHub)" <gi...@apache.org>.
ruanhang1993 commented on code in PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590#discussion_r1833761648
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/parser/CustomAlterTableParserListener.java:
##########
@@ -76,6 +77,29 @@ public CustomAlterTableParserListener(
this.changes = changes;
}
+ @Override
+ public void exitCopyCreateTable(MySqlParser.CopyCreateTableContext ctx) {
+ TableId tableId = parser.parseQualifiedTableId(ctx.tableName(0).fullId());
+ TableId originalTableId = parser.parseQualifiedTableId(ctx.tableName(1).fullId());
+ Table original = parser.databaseTables().forTable(originalTableId);
Review Comment:
I think we need to skip parsing the filtered tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590#issuecomment-2526525052
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590#discussion_r2354216201
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/parser/CustomAlterTableParserListener.java:
##########
@@ -76,6 +77,29 @@ public CustomAlterTableParserListener(
this.changes = changes;
}
+ @Override
+ public void exitCopyCreateTable(MySqlParser.CopyCreateTableContext ctx) {
+ TableId tableId = parser.parseQualifiedTableId(ctx.tableName(0).fullId());
+ TableId originalTableId = parser.parseQualifiedTableId(ctx.tableName(1).fullId());
+ Table original = parser.databaseTables().forTable(originalTableId);
Review Comment:
The original sql was already filtered by table name in https://github.com/apache/flink-cdc/blob/32ec9f57e6be7f8152d5e27cbf2423dc23055b6b/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/io/debezium/connector/mysql/MySqlStreamingChangeEventSource.java#L660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590#issuecomment-2453630337
This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590#issuecomment-2526757300
@ruanhang1993 Would you like to help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590#discussion_r2354519364
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlPipelineITCase.java:
##########
Review Comment:
Can we add similar cases for MySQL CDC pipeline connectors with `scan.binlog.newly-added-table.enabled`?
IIUC if we can't obtain new schema from `CREATE TABLE ... LIKE ...` DDL, triggering failover and re-scanning table schemas should recover the job.
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/parser/CustomAlterTableParserListener.java:
##########
@@ -82,6 +83,29 @@ public CustomAlterTableParserListener(
this.isTableIdCaseInsensitive = isTableIdCaseInsensitive;
}
+ @Override
+ public void exitCopyCreateTable(MySqlParser.CopyCreateTableContext ctx) {
+ TableId tableId = parser.parseQualifiedTableId(ctx.tableName(0).fullId());
+ TableId originalTableId = parser.parseQualifiedTableId(ctx.tableName(1).fullId());
+ Table original = parser.databaseTables().forTable(originalTableId);
+ if (original != null) {
+ parser.databaseTables()
+ .overwriteTable(
+ tableId,
+ original.columns(),
+ original.primaryKeyColumnNames(),
+ original.defaultCharsetName());
+ parser.signalCreateTable(tableId, ctx);
+ Schema.Builder builder = Schema.newBuilder();
+ original.columns().forEach(column -> builder.column(toCdcColumn(column)));
+ if (!original.primaryKeyColumnNames().isEmpty()) {
+ builder.primaryKey(original.primaryKeyColumnNames());
+ }
+ changes.add(new CreateTableEvent(toCdcTableId(tableId), builder.build()));
+ }
+ super.exitCopyCreateTable(ctx);
+ }
Review Comment:
Print a warning log if `CREATE TABLE LIKE` original table isn't presented in the capture list?
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlPipelineITCase.java:
##########
Review Comment:
Also, `MySqlPipelineRecordEmitter` may detect this case by checking if there are "DataChangeEvents" without corresponding "CreateTableEvent" and throw exceptions in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590#issuecomment-2327843421
The test case failure in OceanBase is unrelated with this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3590: [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax.
URL: https://github.com/apache/flink-cdc/pull/3590
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590#issuecomment-3301530227
Hi @yuxiqian @leonardBang, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan merged PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36178][pipeline-connector/mysql] Parse `CREATE TABLE ... LIKE …` syntax. [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #3590:
URL: https://github.com/apache/flink-cdc/pull/3590#discussion_r2354900722
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/parser/CustomAlterTableParserListener.java:
##########
@@ -82,6 +83,29 @@ public CustomAlterTableParserListener(
this.isTableIdCaseInsensitive = isTableIdCaseInsensitive;
}
+ @Override
+ public void exitCopyCreateTable(MySqlParser.CopyCreateTableContext ctx) {
+ TableId tableId = parser.parseQualifiedTableId(ctx.tableName(0).fullId());
+ TableId originalTableId = parser.parseQualifiedTableId(ctx.tableName(1).fullId());
+ Table original = parser.databaseTables().forTable(originalTableId);
+ if (original != null) {
+ parser.databaseTables()
+ .overwriteTable(
+ tableId,
+ original.columns(),
+ original.primaryKeyColumnNames(),
+ original.defaultCharsetName());
+ parser.signalCreateTable(tableId, ctx);
+ Schema.Builder builder = Schema.newBuilder();
+ original.columns().forEach(column -> builder.column(toCdcColumn(column)));
+ if (!original.primaryKeyColumnNames().isEmpty()) {
+ builder.primaryKey(original.primaryKeyColumnNames());
+ }
+ changes.add(new CreateTableEvent(toCdcTableId(tableId), builder.build()));
+ }
+ super.exitCopyCreateTable(ctx);
+ }
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org