You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lvyanquan (via GitHub)" <gi...@apache.org> on 2023/08/27 02:46:54 UTC
[GitHub] [hudi] lvyanquan opened a new issue, #9549: [SUPPORT] Insert operation in Spark will cause inconsistent
lvyanquan opened a new issue, #9549:
URL: https://github.com/apache/hudi/issues/9549
**Describe the problem you faced**
Insert operation in spark don‘t perform tagLocation method to find oldLocation, then it may lead to inconsistent.
For example, I have 2 files A and B in a table which primaryKey is id, and A contains a record with id = 1, then I insert a new record which id = 1 too.
1) if file A is small than target size for parquet files, then this new record will be inserted to A, and I got one record with id = 1.
2) if file B is small than target size for parquet files, then this new record will be inserted to B, and I got two records with id = 1.
**To Reproduce**
Steps to reproduce the behavior:
```
create table test (id int, age int) using hudi tblproperties(primaryKey='id');
```
1)
insert twice, and only one record with id = 1 will be remained.
```
insert into test values(1, 1);
insert into test values(1, 2);
```
2)
need to insert into table test with more records.
table tmp has records range of id from 1 to 1000000.
```
insert into tests as select * from tmp;
```
then insert record with id = 1
```
insert into test values(1, 2);
```
I got two records with id = 1.
**Expected behavior**
it's reasonable?it will cause result of count(*) to be inconsistent too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9549: [SUPPORT] Insert operation in Spark will cause inconsistent
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9549:
URL: https://github.com/apache/hudi/issues/9549#issuecomment-1694665618
@lvyanquan Yes, This is expected and by design as you are not providing the precombine field in your table. We normally do that for out immutable datasets. So it runs the operation type insert which dont use tagging and you may saw duplicates.
We also do small file handling so in use case 1, you don't see duplicates due to this. You can turn on the flag allow duplicates(hoodie.merge.allow.duplicate.on.inserts) for that. I understand this may cause confusion that is why in new release we are auto enabling the flag so all duplicates will flow in.
You can provide some precombineField, then it will behave as upsert as no duplicates will be created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #9549: [SUPPORT] Insert operation in Spark will cause inconsistent
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #9549: [SUPPORT] Insert operation in Spark will cause inconsistent
URL: https://github.com/apache/hudi/issues/9549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9549: [SUPPORT] Insert operation in Spark will cause inconsistent
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9549:
URL: https://github.com/apache/hudi/issues/9549#issuecomment-1697741598
@lvyanquan Closing out this issue. Please reopen in case you have any more queries on this. Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org