You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/10/18 09:31:01 UTC
[GitHub] [inlong] EMsnap opened a new pull request, #6210: [INLONG-6174][Sort] Mysql connector support meta data with debezium format
EMsnap opened a new pull request, #6210:
URL: https://github.com/apache/inlong/pull/6210
- Fixes #6174
### Motivation
[INLONG-6174][Sort] Mysql connector support meta data with debezium format
### Modifications
[INLONG-6174][Sort] Mysql connector support meta data with debezium format
### Verifying this change
run allmigrateUt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6210: [INLONG-6174][Sort] Mysql connector support meta data with debezium format
Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6210:
URL: https://github.com/apache/inlong/pull/6210#discussion_r1000527832
##########
inlong-sort/sort-connectors/mysql-cdc/src/main/java/org/apache/inlong/sort/cdc/mysql/table/MySqlReadableMetadata.java:
##########
@@ -98,7 +100,7 @@ public Object read(SourceRecord record) {
}),
DATA(
- "meta.data",
Review Comment:
Please keep in mind code compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6210: [INLONG-6174][Sort] Mysql connector support meta data with debezium format
Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6210:
URL: https://github.com/apache/inlong/pull/6210#discussion_r999170494
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/KafkaLoadNode.java:
##########
@@ -226,8 +226,8 @@ public String getMetadataKey(MetaField metaField) {
case OP_TYPE:
metadataKey = "value.op-type";
break;
- case DATA:
- metadataKey = "value.data";
+ case DATA_CANAL:
Review Comment:
I guess using two seperated meta data is fair enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6210: [INLONG-6174][Sort] Mysql connector support meta data with debezium format
Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6210:
URL: https://github.com/apache/inlong/pull/6210#discussion_r999363750
##########
inlong-common/src/main/java/org/apache/inlong/common/enums/MetaField.java:
##########
@@ -62,12 +62,22 @@ public enum MetaField {
/**
* Represents a canal json of a record in database (in string format)
*/
- DATA,
Review Comment:
Please keep in mind code compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6210: [INLONG-6174][Sort] Mysql connector support meta data with debezium format
Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6210:
URL: https://github.com/apache/inlong/pull/6210#discussion_r1000628573
##########
inlong-sort/sort-connectors/mysql-cdc/src/main/java/org/apache/inlong/sort/cdc/mysql/table/MySqlReadableMetadata.java:
##########
@@ -98,7 +100,7 @@ public Object read(SourceRecord record) {
}),
DATA(
- "meta.data",
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6210: [INLONG-6174][Sort] Mysql connector support meta data with debezium format
Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6210:
URL: https://github.com/apache/inlong/pull/6210#discussion_r999170494
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/KafkaLoadNode.java:
##########
@@ -226,8 +226,8 @@ public String getMetadataKey(MetaField metaField) {
case OP_TYPE:
metadataKey = "value.op-type";
break;
- case DATA:
- metadataKey = "value.data";
+ case DATA_CANAL:
Review Comment:
I guess using two seperated meta data is fare enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #6210: [INLONG-6174][Sort] MySql connector support meta data with debezium format
Posted by GitBox <gi...@apache.org>.
dockerzhang merged PR #6210:
URL: https://github.com/apache/inlong/pull/6210
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6210: [INLONG-6174][Sort] Mysql connector support meta data with debezium format
Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6210:
URL: https://github.com/apache/inlong/pull/6210#discussion_r1000490064
##########
inlong-common/src/main/java/org/apache/inlong/common/enums/MetaField.java:
##########
@@ -62,12 +62,22 @@ public enum MetaField {
/**
* Represents a canal json of a record in database (in string format)
*/
- DATA,
Review Comment:
done thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6210: [INLONG-6174][Sort] MySql connector support meta data with debezium format
Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6210:
URL: https://github.com/apache/inlong/pull/6210#discussion_r1001439770
##########
inlong-sort/sort-connectors/mysql-cdc/src/main/java/org/apache/inlong/sort/cdc/mysql/table/MySqlReadableMetadata.java:
##########
@@ -379,6 +415,38 @@ public Object read(SourceRecord record) {
}
});
+ private static StringData getCanalData(SourceRecord record, GenericRowData rowData,
+ TableChange tableSchema) {
+ Struct messageStruct = (Struct) record.value();
+ Struct sourceStruct = messageStruct.getStruct(FieldName.SOURCE);
+ // tableName
+ String tableName = getMetaData(record, AbstractSourceInfo.TABLE_NAME_KEY);
+ // databaseName
+ String databaseName = getMetaData(record, AbstractSourceInfo.DATABASE_NAME_KEY);
+ // opTs
+ long opTs = (Long) sourceStruct.get(AbstractSourceInfo.TIMESTAMP_KEY);
+ // ts
+ long ts = (Long) messageStruct.get(FieldName.TIMESTAMP);
+ // actual data
+ GenericRowData data = rowData;
+ Map<String, Object> field = (Map<String, Object>) data.getField(0);
+ List<Map<String, Object>> dataList = new ArrayList<>();
+ dataList.add(field);
+
+ CanalJson canalJson = CanalJson.builder()
+ .data(dataList).database(databaseName)
+ .sql("").es(opTs).isDdl(false).pkNames(getPkNames(tableSchema))
+ .mysqlType(getMysqlType(tableSchema)).table(tableName).ts(ts)
+ .type(getOpType(record)).sqlType(getSqlType(tableSchema)).build();
+
+ try {
+ ObjectMapper objectMapper = new ObjectMapper();
Review Comment:
done thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6210: [INLONG-6174][Sort] Mysql connector support meta data with debezium format
Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #6210:
URL: https://github.com/apache/inlong/pull/6210#discussion_r1001294560
##########
inlong-sort/sort-connectors/mysql-cdc/src/main/java/org/apache/inlong/sort/cdc/mysql/table/MySqlReadableMetadata.java:
##########
@@ -379,6 +415,38 @@ public Object read(SourceRecord record) {
}
});
+ private static StringData getCanalData(SourceRecord record, GenericRowData rowData,
+ TableChange tableSchema) {
+ Struct messageStruct = (Struct) record.value();
+ Struct sourceStruct = messageStruct.getStruct(FieldName.SOURCE);
+ // tableName
+ String tableName = getMetaData(record, AbstractSourceInfo.TABLE_NAME_KEY);
+ // databaseName
+ String databaseName = getMetaData(record, AbstractSourceInfo.DATABASE_NAME_KEY);
+ // opTs
+ long opTs = (Long) sourceStruct.get(AbstractSourceInfo.TIMESTAMP_KEY);
+ // ts
+ long ts = (Long) messageStruct.get(FieldName.TIMESTAMP);
+ // actual data
+ GenericRowData data = rowData;
+ Map<String, Object> field = (Map<String, Object>) data.getField(0);
+ List<Map<String, Object>> dataList = new ArrayList<>();
+ dataList.add(field);
+
+ CanalJson canalJson = CanalJson.builder()
+ .data(dataList).database(databaseName)
+ .sql("").es(opTs).isDdl(false).pkNames(getPkNames(tableSchema))
+ .mysqlType(getMysqlType(tableSchema)).table(tableName).ts(ts)
+ .type(getOpType(record)).sqlType(getSqlType(tableSchema)).build();
+
+ try {
+ ObjectMapper objectMapper = new ObjectMapper();
Review Comment:
Suggest using a global static instance of ObjectMapper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6210: [INLONG-6174][Sort] Mysql connector support meta data with debezium format
Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6210:
URL: https://github.com/apache/inlong/pull/6210#discussion_r998292054
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/KafkaLoadNode.java:
##########
@@ -226,8 +226,8 @@ public String getMetadataKey(MetaField metaField) {
case OP_TYPE:
metadataKey = "value.op-type";
break;
- case DATA:
- metadataKey = "value.data";
+ case DATA_CANAL:
Review Comment:
Maybe it is better to add 'source.multiple.format' to decide which serialization to use？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org