You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "vinlee19 (via GitHub)" <gi...@apache.org> on 2024/01/17 09:39:52 UTC
[PR] [fix](cdc) fix datetime precision and regular matching format errors after turning on single-sink [doris-flink-connector]
vinlee19 opened a new pull request, #298:
URL: https://github.com/apache/doris-flink-connector/pull/298
when synchronizing the entire database use `single-sink`, the following two problems will occur:
1. MySQL DATETIME/TIMESTAMP precision mapping error.
3. When including_tables is null, it will cause a regular expression format error，for example:
```java
org.apache.flink.util.FlinkRuntimeException: Failed to discovery tables to capture
at com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.discoveryCaptureTables(MySqlSnapshotSplitAssigner.java:197)
at com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.open(MySqlSnapshotSplitAssigner.java:182)
at com.ververica.cdc.connectors.mysql.source.assigners.MySqlHybridSplitAssigner.open(MySqlHybridSplitAssigner.java:93)
at com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator.start(MySqlSourceEnumerator.java:92)
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$start$1(SourceCoordinator.java:233)
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$runInEventLoop$10(SourceCoordinator.java:469)
at org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalArgumentException: Can't find any matched tables, please check your configured database-name: [test_cdc] and table-name: [(test_cdc)\.(null)]
at com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.discoverCapturedTables(DebeziumUtils.java:196)
at com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.discoveryCaptureTables(MySqlSnapshotSplitAssigner.java:193)
... 13 more
```
Thus, when `including_tables` is null, you can set `including-tables= '.*'` to solve this problem.
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
4. Has unit tests been added: (Yes/No/No Need)
5. Has document been added or modified: (Yes/No/No Need)
6. Does it need to update dependencies: (Yes/No)
7. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cdc) fix datetime precision and regular matching format errors after turning on single-sink [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #298:
URL: https://github.com/apache/doris-flink-connector/pull/298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cdc) fix datetime precision and regular matching format errors after turning on single-sink [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #298:
URL: https://github.com/apache/doris-flink-connector/pull/298#discussion_r1455112021
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/jsondebezium/JsonDebeziumSchemaChangeImplV2.java:
##########
@@ -337,6 +337,9 @@ public String buildDorisTypeName(JsonNode column) {
String dorisTypeName;
switch (sourceConnector) {
case MYSQL:
+ if (sourceTypeName.equals("DATETIME") || sourceTypeName.equals("TIMESTAMP")) {
+ length = 20 + length;
+ }
dorisTypeName = MysqlType.toDorisType(sourceTypeName, length, scale);
Review Comment:
Can it be added directly to this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cdc) fix datetime precision and regular matching format errors after turning on single-sink [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #298:
URL: https://github.com/apache/doris-flink-connector/pull/298#discussion_r1456741673
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/MysqlType.java:
##########
@@ -144,8 +144,17 @@ public static String toDorisType(String type, Integer length, Integer scale) {
return DorisType.DATE_V2;
case DATETIME:
case TIMESTAMP:
- int dtScale = length > 19 ? length - 20 : 0;
- return String.format("%s(%s)", DorisType.DATETIME_V2, Math.min(dtScale, 6));
+ // default precision is 0
+ if (length == null || length <= 0) {
+ return DorisType.DATETIME_V2;
+ // In JsonDebeziumSchemaSerializer record,the length of timestamp/datetime is 0
+ // to 6.
+ } else if (length <= 6) {
+ return String.format("%s(%s)", DorisType.DATETIME_V2, length);
+ } else {
+ int dtScale = length > 19 ? length - 20 : 0;
+ return String.format("%s(%s)", DorisType.DATETIME_V2, Math.min(dtScale, 6));
Review Comment:
what about length=7, Can we just write it like this : `Math.min(length > 19 ? length - 20 : length, 6)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cdc) fix datetime precision and regular matching format errors after turning on single-sink [doris-flink-connector]
Posted by "vinlee19 (via GitHub)" <gi...@apache.org>.
vinlee19 commented on code in PR #298:
URL: https://github.com/apache/doris-flink-connector/pull/298#discussion_r1455354222
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/jsondebezium/JsonDebeziumSchemaChangeImplV2.java:
##########
@@ -337,6 +337,9 @@ public String buildDorisTypeName(JsonNode column) {
String dorisTypeName;
switch (sourceConnector) {
case MYSQL:
+ if (sourceTypeName.equals("DATETIME") || sourceTypeName.equals("TIMESTAMP")) {
+ length = 20 + length;
+ }
dorisTypeName = MysqlType.toDorisType(sourceTypeName, length, scale);
Review Comment:
I have added it to MysqlType.toDorisType()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org