You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/21 20:13:38 UTC

[GitHub] [hudi] kazdy opened a new issue, #7266: [SUPPORT] Insert into ovewrites existing record with the same key

kazdy opened a new issue, #7266:
URL: https://github.com/apache/hudi/issues/7266

   **Describe the problem you faced**
   
   spark sql insert updates the whole record if the record with same PK already exists in hudi table
   
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   ```
   create table hudi_cow_nonpcf_tbl (
     uuid int,
     name string,
     price double
   ) using hudi;
   
   set hoodie.sql.insert.mode=strict;
   
   # first insert
   insert into hudi_cow_nonpcf_tbl select 1, ‘a1’, 20;
   
   select * from hudi_cow_nonpcf_tbl;
   
   # returns
   1	a1	20.0
   
   # another insert with the same key, different values:
   insert into hudi_cow_nonpcf_tbl select 1, ‘a2’, 30;
   
   select * from hudi_cow_nonpcf_tbl;
   # returns
   1	a2	30.0
   ```
   
   **Expected behavior**
   
   There's a difference in behavior when precombine field is specified and Hudi throws an error.
   I would the second insert fail if a record with the same key already exists when precombine field is not specified and strict insert mode is enabled.
   
   **Environment Description**
   
   * Hudi version : 0.12.1 oss
   
   * Spark version : 3.2.1 oss
   
   * Hive version : none
   
   * Hadoop version : unknown
   
   * Storage (HDFS/S3/GCS..) : local filesystem
   
   * Running on Docker? (yes/no) : no
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] codope commented on issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key in strict insert mode when no precombine field is specified.

Posted by GitBox <gi...@apache.org>.
codope commented on issue #7266:
URL: https://github.com/apache/hudi/issues/7266#issuecomment-1325003477

   For COW table, irrespective of whether precombine field is provided or not, PK uniqueness should be guaranteed in case of strict insert mode. Thanks for raising the PR. I'll review it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] kazdy commented on issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key in strict insert mode when no precombine field is specified.

Posted by GitBox <gi...@apache.org>.
kazdy commented on issue #7266:
URL: https://github.com/apache/hudi/issues/7266#issuecomment-1322719011

   I think the issue is here:
   https://github.com/apache/hudi/blob/b662cf6789cbefb4af3c10dc544c1040ff57cbb3/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala#L140-L161
   
   In this case, `isNonStrictMode` is `false` and `hasPrecombineColumn` is `false` 
   so it matches `INSERT_OPERATION_OPT_VAL `
   and Hudi uses `OverwriteWithLatestAvroPayload` and does not validate for duplicate keys even though `strict` insert mode is being used.
   https://github.com/apache/hudi/blob/b662cf6789cbefb4af3c10dc544c1040ff57cbb3/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala#L163-L170
   
   This behavior is inconsistent with the documentation which tells nothing about preCombineField:
   
   > For strict mode, insert statement will keep the primary key uniqueness constraint which do not allow duplicate record.
   
   https://hudi.apache.org/docs/configurations#hoodiesqlinsertmode
   
   cc @jonvex since we talked about this on slack
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] codope closed issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key in strict insert mode when no precombine field is specified.

Posted by GitBox <gi...@apache.org>.
codope closed issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key in strict insert mode when no precombine field is specified.
URL: https://github.com/apache/hudi/issues/7266


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] kazdy closed issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key (expected?)

Posted by GitBox <gi...@apache.org>.
kazdy closed issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key (expected?)
URL: https://github.com/apache/hudi/issues/7266


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] kazdy commented on issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key (expected?)

Posted by GitBox <gi...@apache.org>.
kazdy commented on issue #7266:
URL: https://github.com/apache/hudi/issues/7266#issuecomment-1322511752

   expected, see
   https://hudi.apache.org/docs/configurations#hoodiesqlinsertmode
   to change behaviour
   
   `set hoodie.sql.insert.mode=strict;`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key in strict insert mode when no precombine field is specified.

Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #7266:
URL: https://github.com/apache/hudi/issues/7266#issuecomment-1323195377

   Nice analitics, can you follow up here @nsivabalan ? Actually Flink has fixed this long time ago.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org