You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "lordcheng10 (via GitHub)" <gi...@apache.org> on 2023/02/28 05:20:52 UTC
[GitHub] [inlong] lordcheng10 opened a new pull request, #7456: [INLONG-7455][Sort]
lordcheng10 opened a new pull request, #7456:
URL: https://github.com/apache/inlong/pull/7456
…, the dirty data type should be converted to RowData for archiving.
### Prepare a Pull Request
- Fixes https://github.com/apache/inlong/issues/7455
### Motivation
Start a mysql synchronization task for Iceberg and insert a type mismatch of dirty data, the archived dirty data should contain data content,eg: [time] [database] [table] [data]
But the test found that it does not contain data:

### Modifications
When the types do not match, the dirty data type passed to the handleDirtyData method should not be in json format, but should be converted into a RowData object:
https://github.com/apache/inlong/blob/08887b1dc240c015775cfc3c16b5ec8666833a78/inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/DynamicSchemaHandleOperator.java#L314-L316
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #7456: [INLONG-7455][Sort] Fixing Dirty Data Archival Format Issues in Iceberg Connector
Posted by "Yizhou-Yang (via GitHub)" <gi...@apache.org>.
Yizhou-Yang commented on code in PR #7456:
URL: https://github.com/apache/inlong/pull/7456#discussion_r1119658935
##########
inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/DynamicSchemaHandleOperator.java:
##########
@@ -314,7 +314,11 @@ private void handleSchemaInfoEvent(TableIdentifier tableId, Schema schema) {
LOG.warn("Ignore table {} schema change, old: {} new: {}.",
tableId, dataSchema, latestSchema, e);
blacklist.add(tableId);
- handleDirtyData(jsonNode, jsonNode, DirtyType.EXTRACT_ROWDATA_ERROR, e, tableId);
+ List<RowData> rowDataForDataSchemaList = dynamicSchemaFormat.extractRowData(jsonNode,
Review Comment:
is there a possibility that dynamicSchemaFormat.extractRowData() will throw an exception? If so, is it handled properly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] Yizhou-Yang commented on a diff in pull request #7456: [INLONG-7455][Sort] Fixing Dirty Data Archival Format Issues in Iceberg Connector
Posted by "Yizhou-Yang (via GitHub)" <gi...@apache.org>.
Yizhou-Yang commented on code in PR #7456:
URL: https://github.com/apache/inlong/pull/7456#discussion_r1119658935
##########
inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/DynamicSchemaHandleOperator.java:
##########
@@ -314,7 +314,11 @@ private void handleSchemaInfoEvent(TableIdentifier tableId, Schema schema) {
LOG.warn("Ignore table {} schema change, old: {} new: {}.",
tableId, dataSchema, latestSchema, e);
blacklist.add(tableId);
- handleDirtyData(jsonNode, jsonNode, DirtyType.EXTRACT_ROWDATA_ERROR, e, tableId);
+ List<RowData> rowDataForDataSchemaList = dynamicSchemaFormat.extractRowData(jsonNode,
Review Comment:
is there a possibility that this line will throw an exception? If so, is it handled properly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] lordcheng10 commented on a diff in pull request #7456: [INLONG-7455][Sort] Fixing Dirty Data Archival Format Issues in Iceberg Connector
Posted by "lordcheng10 (via GitHub)" <gi...@apache.org>.
lordcheng10 commented on code in PR #7456:
URL: https://github.com/apache/inlong/pull/7456#discussion_r1119720991
##########
inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/DynamicSchemaHandleOperator.java:
##########
@@ -314,7 +314,11 @@ private void handleSchemaInfoEvent(TableIdentifier tableId, Schema schema) {
LOG.warn("Ignore table {} schema change, old: {} new: {}.",
tableId, dataSchema, latestSchema, e);
blacklist.add(tableId);
- handleDirtyData(jsonNode, jsonNode, DirtyType.EXTRACT_ROWDATA_ERROR, e, tableId);
+ List<RowData> rowDataForDataSchemaList = dynamicSchemaFormat.extractRowData(jsonNode,
Review Comment:
1. dynamicSchemaFormat.extractRowData() usually throws an exception when the schema does not match.
2. Use dataSchema for conversion, usually no exception will be thrown;
3. We can also use try catch to output the error log when an exception is thrown? @Yizhou-Yang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap merged pull request #7456: [INLONG-7455][Sort] Fix dirty data archival format issues in Iceberg connector
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap merged PR #7456:
URL: https://github.com/apache/inlong/pull/7456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org