You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/10/28 02:43:27 UTC

[GitHub] [inlong] thesumery opened a new pull request, #6312: [INLONG-6311][Sort] Bugfix:Multiple sink scene canal json old data missing field cause job fail

thesumery opened a new pull request, #6312:
URL: https://github.com/apache/inlong/pull/6312

   [INLONG-6311][Sort] Bugfix:Multiple sink scene canal json old data missing field cause job fail
   
   ### Prepare a Pull Request
   
   - [INLONG-6311][Sort] Bugfix:Multiple sink scene canal json old data missing field cause job fail
   - Fixes #6311 
   
   ### Motivation
   
   *Bugfix:Multiple sink scene canal json old data missing field cause job fail*
   
   ### Modifications
   
   *Bugfix:Multiple sink scene canal json old data missing field cause job fail*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] thesumery commented on a diff in pull request #6312: [INLONG-6311][Sort] Bugfix:Multiple sink scene canal json old data missing field cause job fail

Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6312:
URL: https://github.com/apache/inlong/pull/6312#discussion_r1007558564


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/format/JsonDynamicSchemaFormat.java:
##########
@@ -99,7 +99,7 @@ public abstract class JsonDynamicSchemaFormat extends AbstractDynamicSchemaForma
 
     protected JsonDynamicSchemaFormat() {
         this.rowDataConverters =
-                new JsonToRowDataConverters(true, false, TimestampFormat.ISO_8601);
+                new JsonToRowDataConverters(false, false, TimestampFormat.ISO_8601);

Review Comment:
   it's recommanded to config `TimestampFormat` and `ignoreParseErrors` as a configurable parameter.But `failOnMissingField` should not be exported. Here are the reasons.
   1. `failOnMissingField` should always be false beacuse old data always fill modified field, it means it always miss some fields.So if we want to extract old data in canal json, we should always set this parameters to `false`.
   
   I will submit an issue to follow it. #6313 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] thesumery commented on a diff in pull request #6312: [INLONG-6311][Sort] Bugfix:Multiple sink scene canal json old data missing field cause job fail

Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6312:
URL: https://github.com/apache/inlong/pull/6312#discussion_r1007558564


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/format/JsonDynamicSchemaFormat.java:
##########
@@ -99,7 +99,7 @@ public abstract class JsonDynamicSchemaFormat extends AbstractDynamicSchemaForma
 
     protected JsonDynamicSchemaFormat() {
         this.rowDataConverters =
-                new JsonToRowDataConverters(true, false, TimestampFormat.ISO_8601);
+                new JsonToRowDataConverters(false, false, TimestampFormat.ISO_8601);

Review Comment:
   it's recommanded to config TimestampFormat as a configurable parameter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] dockerzhang merged pull request #6312: [INLONG-6311][Sort] Fix missing fields error for the old Canal JSON data in multiple sink scenes

Posted by GitBox <gi...@apache.org>.
dockerzhang merged PR #6312:
URL: https://github.com/apache/inlong/pull/6312


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] thesumery commented on a diff in pull request #6312: [INLONG-6311][Sort] Bugfix:Multiple sink scene canal json old data missing field cause job fail

Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6312:
URL: https://github.com/apache/inlong/pull/6312#discussion_r1007558564


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/format/JsonDynamicSchemaFormat.java:
##########
@@ -99,7 +99,7 @@ public abstract class JsonDynamicSchemaFormat extends AbstractDynamicSchemaForma
 
     protected JsonDynamicSchemaFormat() {
         this.rowDataConverters =
-                new JsonToRowDataConverters(true, false, TimestampFormat.ISO_8601);
+                new JsonToRowDataConverters(false, false, TimestampFormat.ISO_8601);

Review Comment:
   it's recommanded to config `TimestampFormat` and `ignoreParseErrors` as a configurable parameter.But `failOnMissingField` should not be exported. Here are the reasons.
   1. `failOnMissingField` should always be false beacuse old data always fill modified field, it means it always miss some fields.So if we want to extract old data in canal json, we should always set this parameters to `false`.
   
   I will submit an issue to follow it. [](https://github.com/apache/inlong/issues/6313)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] thesumery commented on pull request #6312: [INLONG-6311][Sort] Bugfix:Multiple sink scene canal json old data missing field cause job fail

Posted by GitBox <gi...@apache.org>.
thesumery commented on PR #6312:
URL: https://github.com/apache/inlong/pull/6312#issuecomment-1294412113

   it's recommanded to config TimestampFormat and ignoreParseErrors as a configurable parameter.But failOnMissingField should not be exported. Here are the reasons.
   
   failOnMissingField should always be false beacuse old data always fill modified field, it means it always miss some fields.So if we want to extract old data in canal json, we should always set this parameters to false.
   I will submit an issue to follow it. https://github.com/apache/inlong/issues/6313


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] thesumery commented on a diff in pull request #6312: [INLONG-6311][Sort] Bugfix:Multiple sink scene canal json old data missing field cause job fail

Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6312:
URL: https://github.com/apache/inlong/pull/6312#discussion_r1007558564


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/format/JsonDynamicSchemaFormat.java:
##########
@@ -99,7 +99,7 @@ public abstract class JsonDynamicSchemaFormat extends AbstractDynamicSchemaForma
 
     protected JsonDynamicSchemaFormat() {
         this.rowDataConverters =
-                new JsonToRowDataConverters(true, false, TimestampFormat.ISO_8601);
+                new JsonToRowDataConverters(false, false, TimestampFormat.ISO_8601);

Review Comment:
   it's recommanded to config `TimestampFormat` and `ignoreParseErrors` as a configurable parameter.But `failOnMissingField` should not be exported. Here are the reasons.
   1. `failOnMissingField` should always be false beacuse old data always fill modified field, it means it always miss some fields.So if we want to extract old data in canal json, we should always set this parameters to `false`.
   
   I will submit an issue to follow it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] EMsnap commented on a diff in pull request #6312: [INLONG-6311][Sort] Bugfix:Multiple sink scene canal json old data missing field cause job fail

Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6312:
URL: https://github.com/apache/inlong/pull/6312#discussion_r1007556081


##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/format/JsonDynamicSchemaFormat.java:
##########
@@ -99,7 +99,7 @@ public abstract class JsonDynamicSchemaFormat extends AbstractDynamicSchemaForma
 
     protected JsonDynamicSchemaFormat() {
         this.rowDataConverters =
-                new JsonToRowDataConverters(true, false, TimestampFormat.ISO_8601);
+                new JsonToRowDataConverters(false, false, TimestampFormat.ISO_8601);

Review Comment:
   suggested a configurable parameter



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org