You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by "najsword (via GitHub)" <gi...@apache.org> on 2024/12/13 09:25:18 UTC
[I] Failed to apply schema change. [flink-cdc]
najsword opened a new issue, #3799:
URL: https://github.com/apache/flink-cdc/issues/3799
### Please don't check the box below
- [X] I'm aware that bugs and new features should be reported on [Apache Jira](https://issues.apache.org/jira) or Flink mailing list (dev@flink.apache.org) instead of here
### Again, please don't check the box below
- [X] I'm aware that new issues on GitHub will be ignored and automatically closed.
### 请不要勾选以下选项
- [X] 我已知悉缺陷和新功能需要在 [Apache Jira](https://issues.apache.org/jira) 或 Flink 邮件列表（dev@flink.apache.org）中反馈，而不是在这里创建新 issue。
### 也请不要勾选以下选项
- [X] 我已知悉 GitHub 上的新 issue 会被忽略且自动关闭。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to apply schema change. [flink-cdc]
Posted by "najsword (via GitHub)" <gi...@apache.org>.
najsword commented on issue #3799:
URL: https://github.com/apache/flink-cdc/issues/3799#issuecomment-2541044509
> For Doris, the most likely reason is because Doris does not allow any schema change request until the previous one is finished. However, current connector API returns asynchronously, before it finishes. Any chance you're executing frequent schema changes on the same table?
After researching, We dont change shcema. Frequently . (Add information)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to apply schema change. [flink-cdc]
Posted by "najsword (via GitHub)" <gi...@apache.org>.
najsword commented on issue #3799:
URL: https://github.com/apache/flink-cdc/issues/3799#issuecomment-2541012742
> For Doris, the most likely reason is because Doris does not allow any schema change request until the previous one is finished. However, current connector API returns asynchronously, before it finishes. Any chance you're executing frequent schema changes on the same table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to apply schema change. [flink-cdc]
Posted by "najsword (via GitHub)" <gi...@apache.org>.
najsword commented on issue #3799:
URL: https://github.com/apache/flink-cdc/issues/3799#issuecomment-2540971335
Problem: Failed to apply schema change.
Purpose: mysql->doris sync. About Billion data.
Tool: mysql-5.6.46/5.7.23-log doris-3.0.1-rc04-patch1-8c4e5730a1 flink-1.20.0 flick-cdc-3.2.1&yaml
Explain: this table is exist in doris. Usually Happen this Error. How to slove it !!!
Detail:
2024-12-13 17:21:43
org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'Source: Flink CDC Event Source: mysql -> SchemaOperator -> PrePartition' (operator 9899a42c64d67ef3172b7e3be3c1bbb9).
at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:651)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler.clearCurrentSchemaChangeRequest(SchemaRegistryRequestHandler.java:460)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler.getSchemaChangeResult(SchemaRegistryRequestHandler.java:321)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry.lambda$handleCoordinationRequest$3(SchemaRegistry.java:276)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry.lambda$runInEventLoop$2(SchemaRegistry.java:241)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: Failed to apply schema change.
... 7 more
Caused by: SchemaEvolveException{applyingEvent=CreateTableEvent{tableId=jifei_xxxx, schema=columns={`streamnumber` VARCHAR(40) NOT NULL,`servicekey` INT,`callcost` INT,`calledpartynumber` VARCHAR(64),`callingpartynumber` VARCHAR(64),`chargemode` SMALLINT,`specificchargedpar` VARCHAR(24),`translatednumber` VARCHAR(64),`startdateandtime` VARCHAR(24),`stopdateandtime` VARCHAR(24),`duration` INT,`chargeclass` INT,`transparentparamet` VARCHAR(255),`calltype` INT,`callersubgroup` VARCHAR(64),`calleesubgroup` VARCHAR(24),`acrcallid` VARCHAR(40),`oricallednumber` VARCHAR(64),`oricallingnumber` VARCHAR(64),`callerpnp` VARCHAR(24),`calleepnp` VARCHAR(24),`reroute` INT,`groupnumber` VARCHAR(32),`callcategory` INT,`chargetype` INT,`userpin` VARCHAR(8),`acrtype` INT,`videocallflag` INT,`serviceid` VARCHAR(64),`forwardnumber` VARCHAR(64),`extforwardnumber` VARCHAR(64),`srfmsgid` VARCHAR(255),`msserver` VARCHAR(16),`begintime` VARCHAR(24),`releasecause` INT,`releasereason` VARCHAR(128),`areanumb
er` VARCHAR(8),`calledareacode` VARCHAR(20),`localorlong` VARCHAR(20),`id` INT '0',`dtmfkey` VARCHAR(11),`operator` INT '0'}, primaryKeys=streamnumber, options=()}, exceptionMessage='Failed to schemaChange, response: ', cause='null'}
at org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier.applySchemaChange(DorisMetadataApplier.java:117)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler.applySchemaChange(SchemaRegistryRequestHandler.java:238)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler.lambda$flushSuccess$0(SchemaRegistryRequestHandler.java:306)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to apply schema change. [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on issue #3799:
URL: https://github.com/apache/flink-cdc/issues/3799#issuecomment-2540998250
For Doris, the most likely reason is because Doris does not allow any schema change request until the previous one is finished. However, current connector API returns asynchronously, before it finishes. Any chance you're executing frequent schema changes on the same table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to apply schema change. [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #3799: Failed to apply schema change.
URL: https://github.com/apache/flink-cdc/issues/3799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to apply schema change. [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3799:
URL: https://github.com/apache/flink-cdc/issues/3799#issuecomment-2542569873
As required by Apache Flink, please report bugs or new features on Apache Jira under the project Flink using component tag Flink CDC. You must have a JIRA account in order to log cases and issues. If you don’t have an ASF JIRA account, you can request one at the ASF Self-serve portal, account creation requires review by the PMC member of the application project, which normally takes one to two working days to be approved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to apply schema change. [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3799:
URL: https://github.com/apache/flink-cdc/issues/3799#issuecomment-2542569893
This issue has been closed because Flink CDC doesn't use GitHub issue trackers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to apply schema change. [flink-cdc]
Posted by "najsword (via GitHub)" <gi...@apache.org>.
najsword commented on issue #3799:
URL: https://github.com/apache/flink-cdc/issues/3799#issuecomment-2540970062
Problem: Failed to apply schema change.
Purpose: mysql->doris sync. About Billion data.
Tool: mysql-5.6.46/5.7.23-log doris-3.0.1-rc04-patch1-8c4e5730a1 flink-1.20.0 flick-cdc-3.2.1&yaml
Explain: this table is exist in doris. Usually Happen this Error. How to slove it !!!
Detail:
2024-12-13 17:21:43
org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'Source: Flink CDC Event Source: mysql -> SchemaOperator -> PrePartition' (operator 9899a42c64d67ef3172b7e3be3c1bbb9).
at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:651)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler.clearCurrentSchemaChangeRequest(SchemaRegistryRequestHandler.java:460)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler.getSchemaChangeResult(SchemaRegistryRequestHandler.java:321)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry.lambda$handleCoordinationRequest$3(SchemaRegistry.java:276)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistry.lambda$runInEventLoop$2(SchemaRegistry.java:241)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: Failed to apply schema change.
... 7 more
Caused by: SchemaEvolveException{applyingEvent=CreateTableEvent{tableId=jifei_gdnjhjzx_smp.acr_record_160325_202306_transfer, schema=columns={`streamnumber` VARCHAR(40) NOT NULL,`servicekey` INT,`callcost` INT,`calledpartynumber` VARCHAR(64),`callingpartynumber` VARCHAR(64),`chargemode` SMALLINT,`specificchargedpar` VARCHAR(24),`translatednumber` VARCHAR(64),`startdateandtime` VARCHAR(24),`stopdateandtime` VARCHAR(24),`duration` INT,`chargeclass` INT,`transparentparamet` VARCHAR(255),`calltype` INT,`callersubgroup` VARCHAR(64),`calleesubgroup` VARCHAR(24),`acrcallid` VARCHAR(40),`oricallednumber` VARCHAR(64),`oricallingnumber` VARCHAR(64),`callerpnp` VARCHAR(24),`calleepnp` VARCHAR(24),`reroute` INT,`groupnumber` VARCHAR(32),`callcategory` INT,`chargetype` INT,`userpin` VARCHAR(8),`acrtype` INT,`videocallflag` INT,`serviceid` VARCHAR(64),`forwardnumber` VARCHAR(64),`extforwardnumber` VARCHAR(64),`srfmsgid` VARCHAR(255),`msserver` VARCHAR(16),`begintime` VARCHAR(24),`releasecause`
INT,`releasereason` VARCHAR(128),`areanumber` VARCHAR(8),`calledareacode` VARCHAR(20),`localorlong` VARCHAR(20),`id` INT '0',`dtmfkey` VARCHAR(11),`operator` INT '0'}, primaryKeys=streamnumber, options=()}, exceptionMessage='Failed to schemaChange, response: ', cause='null'}
at org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier.applySchemaChange(DorisMetadataApplier.java:117)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler.applySchemaChange(SchemaRegistryRequestHandler.java:238)
at org.apache.flink.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler.lambda$flushSuccess$0(SchemaRegistryRequestHandler.java:306)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to apply schema change. [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on issue #3799:
URL: https://github.com/apache/flink-cdc/issues/3799#issuecomment-2541060600
To ensure your problems got tracked, could you please create a [JIRA](https://issues.apache.org/jira/projects/FLINK/summary) ticket and provide more information like full logs, schema and YAML configuration? We don't use GitHub issues to collect defect reports.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to apply schema change. [flink-cdc]
Posted by "najsword (via GitHub)" <gi...@apache.org>.
najsword commented on issue #3799:
URL: https://github.com/apache/flink-cdc/issues/3799#issuecomment-2541016190
> For Doris, the most likely reason is because Doris does not allow any schema change request until the previous one is finished. However, current connector API returns asynchronously, before it finishes. Any chance you're executing frequent schema changes on the same table?
Maybe lot of change in binlog. It is also necessary to synchronize quickly. How to slove ? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org