You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/10/10 11:10:28 UTC
[GitHub] [inlong] EMsnap opened a new pull request, #6128: [INLONG-6113][Sort] Mysql cdc connector support read table schema when using debezium function
EMsnap opened a new pull request, #6128:
URL: https://github.com/apache/inlong/pull/6128
- Fixes #6113
### Motivation
The Mysql Connector for extracting data now doesn't support reading table schema when there is no primary key specified.
Canal Json requires field names called mysqlType and sqlType which contains information in table schema
The pr modifies the implementation of debezium function to get table schema from debezium databaseHistory
### Modifications
1 Add a new meta field called DATA_BYTES to represent the canal presentation of a record
2 make the table schema contained in databasehistory static
3 fill canal json with info from table schema
### Verifying this change
run AllMigrateTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #6128: [INLONG-6113][Sort] Mysql cdc connector support read table schema when using debezium function
Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #6128:
URL: https://github.com/apache/inlong/pull/6128#discussion_r991195949
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/SqlServerExtractNode.java:
##########
@@ -139,6 +139,6 @@ public boolean isVirtual(MetaField metaField) {
@Override
public Set<MetaField> supportedMetaFields() {
return EnumSet.of(MetaField.PROCESS_TIME, MetaField.TABLE_NAME, MetaField.DATABASE_NAME,
- MetaField.SCHEMA_NAME, MetaField.OP_TS);
+ MetaField.SCHEMA_NAME, MetaField.OP_TS, MetaField.DATA_BYTES);
Review Comment:
Maybe, we can add this field when it support this metafield
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6128: [INLONG-6113][Sort] Mysql cdc connector support read table schema when using debezium function
Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #6128:
URL: https://github.com/apache/inlong/pull/6128#discussion_r991904447
##########
inlong-sort/sort-connectors/mysql-cdc/src/main/java/org/apache/inlong/sort/cdc/mysql/table/MySqlReadableMetadata.java:
##########
@@ -432,7 +427,27 @@ public static Map<String, String> getMysqlType(@Nullable TableChanges.TableChang
return mysqlType;
}
- private static String getMetaData(SourceRecord record, String tableNameKey) {
+ /**
+ * get sql type from table schema, represents the jdbc data type
+ * @param tableSchema
+ */
+ public static Map<String, Integer> getSqlType(@Nullable TableChanges.TableChange tableSchema) {
+ if (tableSchema == null) {
+ return null;
+ }
+ Map<String, Integer> sqlType = new HashMap<>();
+ final Table table = tableSchema.getTable();
+ table.columns()
+ .forEach(
+ column -> {
+ sqlType.put(
+ column.name(),
+ column.jdbcType());
+ });
+ return sqlType;
+ }
Review Comment:
```suggestion
/**
* get sql type from table schema, represents the jdbc data type
*
* @param tableSchema table schema
*/
public static Map<String, Integer> getSqlType(@Nullable TableChanges.TableChange tableSchema) {
if (tableSchema == null) {
return null;
}
Map<String, Integer> sqlType = new HashMap<>();
final Table table = tableSchema.getTable();
table.columns().forEach(
column -> sqlType.put(column.name(), column.jdbcType())
);
return sqlType;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6128: [INLONG-6113][Sort] Mysql cdc connector support read table schema when using debezium function
Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6128:
URL: https://github.com/apache/inlong/pull/6128#discussion_r991199803
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/SqlServerExtractNode.java:
##########
@@ -139,6 +139,6 @@ public boolean isVirtual(MetaField metaField) {
@Override
public Set<MetaField> supportedMetaFields() {
return EnumSet.of(MetaField.PROCESS_TIME, MetaField.TABLE_NAME, MetaField.DATABASE_NAME,
- MetaField.SCHEMA_NAME, MetaField.OP_TS);
+ MetaField.SCHEMA_NAME, MetaField.OP_TS, MetaField.DATA_BYTES);
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] yunqingmoswu merged pull request #6128: [INLONG-6113][Sort] Mysql cdc connector support read table schema when using debezium function
Posted by GitBox <gi...@apache.org>.
yunqingmoswu merged PR #6128:
URL: https://github.com/apache/inlong/pull/6128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org