You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/21 20:13:38 UTC
[GitHub] [hudi] kazdy opened a new issue, #7266: [SUPPORT] Insert into ovewrites existing record with the same key
kazdy opened a new issue, #7266:
URL: https://github.com/apache/hudi/issues/7266
**Describe the problem you faced**
spark sql insert updates the whole record if the record with same PK already exists in hudi table
**To Reproduce**
Steps to reproduce the behavior:
```
create table hudi_cow_nonpcf_tbl (
uuid int,
name string,
price double
) using hudi;
set hoodie.sql.insert.mode=strict;
# first insert
insert into hudi_cow_nonpcf_tbl select 1, ‘a1’, 20;
select * from hudi_cow_nonpcf_tbl;
# returns
1 a1 20.0
# another insert with the same key, different values:
insert into hudi_cow_nonpcf_tbl select 1, ‘a2’, 30;
select * from hudi_cow_nonpcf_tbl;
# returns
1 a2 30.0
```
**Expected behavior**
There's a difference in behavior when precombine field is specified and Hudi throws an error.
I would the second insert fail if a record with the same key already exists when precombine field is not specified and strict insert mode is enabled.
**Environment Description**
* Hudi version : 0.12.1 oss
* Spark version : 3.2.1 oss
* Hive version : none
* Hadoop version : unknown
* Storage (HDFS/S3/GCS..) : local filesystem
* Running on Docker? (yes/no) : no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key in strict insert mode when no precombine field is specified.
Posted by GitBox <gi...@apache.org>.
codope commented on issue #7266:
URL: https://github.com/apache/hudi/issues/7266#issuecomment-1325003477
For COW table, irrespective of whether precombine field is provided or not, PK uniqueness should be guaranteed in case of strict insert mode. Thanks for raising the PR. I'll review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key in strict insert mode when no precombine field is specified.
Posted by GitBox <gi...@apache.org>.
kazdy commented on issue #7266:
URL: https://github.com/apache/hudi/issues/7266#issuecomment-1322719011
I think the issue is here:
https://github.com/apache/hudi/blob/b662cf6789cbefb4af3c10dc544c1040ff57cbb3/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala#L140-L161
In this case, `isNonStrictMode` is `false` and `hasPrecombineColumn` is `false`
so it matches `INSERT_OPERATION_OPT_VAL `
and Hudi uses `OverwriteWithLatestAvroPayload` and does not validate for duplicate keys even though `strict` insert mode is being used.
https://github.com/apache/hudi/blob/b662cf6789cbefb4af3c10dc544c1040ff57cbb3/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala#L163-L170
This behavior is inconsistent with the documentation which tells nothing about preCombineField:
> For strict mode, insert statement will keep the primary key uniqueness constraint which do not allow duplicate record.
https://hudi.apache.org/docs/configurations#hoodiesqlinsertmode
cc @jonvex since we talked about this on slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key in strict insert mode when no precombine field is specified.
Posted by GitBox <gi...@apache.org>.
codope closed issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key in strict insert mode when no precombine field is specified.
URL: https://github.com/apache/hudi/issues/7266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy closed issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key (expected?)
Posted by GitBox <gi...@apache.org>.
kazdy closed issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key (expected?)
URL: https://github.com/apache/hudi/issues/7266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key (expected?)
Posted by GitBox <gi...@apache.org>.
kazdy commented on issue #7266:
URL: https://github.com/apache/hudi/issues/7266#issuecomment-1322511752
expected, see
https://hudi.apache.org/docs/configurations#hoodiesqlinsertmode
to change behaviour
`set hoodie.sql.insert.mode=strict;`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #7266: [SUPPORT] Insert into ovewrites existing record with the same key in strict insert mode when no precombine field is specified.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #7266:
URL: https://github.com/apache/hudi/issues/7266#issuecomment-1323195377
Nice analitics, can you follow up here @nsivabalan ? Actually Flink has fixed this long time ago.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org