You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/11/02 12:55:31 UTC

[GitHub] [inlong] thesumery opened a new pull request, #6383: [INLONG-6382][Sort] Bugfix:Iceberg miss data when source table do not have primary key in multiple sink scences

thesumery opened a new pull request, #6383:
URL: https://github.com/apache/inlong/pull/6383

   [INLONG-6382][Sort] Bugfix:Iceberg miss data when source table do not have primary key in multiple sink scences
   
   ### Prepare a Pull Request
   - [INLONG-6382][Sort] Bugfix:Iceberg miss data when source table do not have primary key in multiple sink scences
   - Fixes #6382 
   
   ### Motivation
   
   *Bugfix:Iceberg miss data when source table do not have primary key in multiple sink scences*
   
   ### Modifications
   
   *Bugfix:Iceberg miss data when source table do not have primary key in multiple sink scences*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] yunqingmoswu merged pull request #6383: [INLONG-6382][Sort] Iceberg data is messed up when the source table has no primary key in multiple sink scenes

Posted by GitBox <gi...@apache.org>.
yunqingmoswu merged PR #6383:
URL: https://github.com/apache/inlong/pull/6383


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] EMsnap commented on pull request #6383: [INLONG-6382][Sort] Iceberg misses data when the source table has no primary key in multiple sink scenes

Posted by GitBox <gi...@apache.org>.
EMsnap commented on PR #6383:
URL: https://github.com/apache/inlong/pull/6383#issuecomment-1303148604

   I think the Pr title doesn't match with the code 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6383: [INLONG-6382][Sort] Iceberg misses data when the source table has no primary key in multiple sink scenes

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6383:
URL: https://github.com/apache/inlong/pull/6383#discussion_r1013660379


##########
inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/IcebergMultipleStreamWriter.java:
##########
@@ -130,12 +129,6 @@ public void processElement(RecordWithSchema recordWithSchema) throws Exception {
             List<Integer> equalityFieldIds = recordWithSchema.getPrimaryKeys().stream()
                     .map(pk -> recordWithSchema.getSchema().findField(pk).fieldId())
                     .collect(Collectors.toList());
-            // if physical primary key not exist, put all field to logical primary key
-            if (equalityFieldIds.isEmpty()) {

Review Comment:
   It is recommend to add a option 'sink.multiple.handle-strategy-without-pk'(The value maybe ['append'|'all-columns']) to decide what to do, if it is 'append',  all the records will be write use append mode, else use upsert mode by all columns.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] thesumery commented on a diff in pull request #6383: [INLONG-6382][Sort] Iceberg misses data when the source table has no primary key in multiple sink scenes

Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6383:
URL: https://github.com/apache/inlong/pull/6383#discussion_r1013781469


##########
inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/sink/multiple/IcebergMultipleStreamWriter.java:
##########
@@ -130,12 +129,6 @@ public void processElement(RecordWithSchema recordWithSchema) throws Exception {
             List<Integer> equalityFieldIds = recordWithSchema.getPrimaryKeys().stream()
                     .map(pk -> recordWithSchema.getSchema().findField(pk).fieldId())
                     .collect(Collectors.toList());
-            // if physical primary key not exist, put all field to logical primary key
-            if (equalityFieldIds.isEmpty()) {

Review Comment:
   good proposal



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org