You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bithw1 (via GitHub)" <gi...@apache.org> on 2024/09/24 11:09:25 UTC
[I] [SUPPORT] two insert into operation works like upsert [hudi]
bithw1 opened a new issue, #11996:
URL: https://github.com/apache/hudi/issues/11996
Hi,
I am using Hudi 0.15.0,
In the spark-sql cli, I do the following things. I insert two records with two insert clause, I think there will be two records(with same id),but only one is left there, looks Hudi does upsert instead of insert here,
The default behavior of insert into is insert, so I don't understand how upsert happens here
```
CREATE TABLE IF NOT EXISTS hudi_cow_19 (
a INT,
b INT,
c INT
)
USING hudi
tblproperties(
type='cow',
primaryKey='a',
hoodie.datasource.write.precombine.field='c',
hoodie.merge.allow.duplicate.on.inserts='true'
);
insert into hudi_cow_19 select 1,1,1; --- insert the first record
select * from hudi_cow_19;
insert into hudi_cow_19 select 1,10,10; --- insert the second record with same key as the first one
select * from hudi_cow_19;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] two insert into operation works like upsert [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11996:
URL: https://github.com/apache/hudi/issues/11996#issuecomment-2372635150
Eliminate the primary key definition, that is what we call a pk-less table,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] two insert into operation works like upsert [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on issue #11996:
URL: https://github.com/apache/hudi/issues/11996#issuecomment-2373025599
@bithw1 the default value will be modified when running job if not specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] two insert into operation works like upsert [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 closed issue #11996: [SUPPORT] two insert into operation works like upsert
URL: https://github.com/apache/hudi/issues/11996
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] two insert into operation works like upsert [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 commented on issue #11996:
URL: https://github.com/apache/hudi/issues/11996#issuecomment-2373077115
> @bithw1 the default value will be modified when running job if not specified.
@KnightChess I have set this option explicitly per your guide,but I still see the same result(one updated record instead of two records)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] two insert into operation works like upsert [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 commented on issue #11996:
URL: https://github.com/apache/hudi/issues/11996#issuecomment-2373076226
> set hoodie.combine.before.insert = false;
I have set this option explicitly per your guide,but I still see the same result(one updated record instead of two records)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] two insert into operation works like upsert [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 commented on issue #11996:
URL: https://github.com/apache/hudi/issues/11996#issuecomment-2372784452
> Eliminate the primary key definition, that is what we call a pk-less table,
Thanks, I tried and it works for me!
So, can i conclude that with pk definition and precombine field, insert operation will work like upsert?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] two insert into operation works like upsert [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on issue #11996:
URL: https://github.com/apache/hudi/issues/11996#issuecomment-2375529288
@bithw1 my mistake, the `hoodie.combine.before.insert` is controll the imcoming records, your insert is two sql. you can set `set hoodie.spark.sql.insert.into.operation = insert` for pk table too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] two insert into operation works like upsert [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on issue #11996:
URL: https://github.com/apache/hudi/issues/11996#issuecomment-2371316934
@bithw1 because you set precombine field, you can set
`set hoodie.combine.before.insert = false;`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] two insert into operation works like upsert [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 commented on issue #11996:
URL: https://github.com/apache/hudi/issues/11996#issuecomment-2372782842
> @bithw1 because you set precombine field, you can set `set hoodie.combine.before.insert = false;`
Thanks, but it doesn't work for me..actually, `hoodie.combine.before.insert` is false by default.
```
public static final ConfigProperty<String> COMBINE_BEFORE_INSERT = ConfigProperty
.key("hoodie.combine.before.insert")
.defaultValue("false")
.markAdvanced()
.withDocumentation("When inserted records share same key, controls whether they should be first combined (i.e de-duplicated) before"
+ " writing to storage.");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] two insert into operation works like upsert [hudi]
Posted by "bithw1 (via GitHub)" <gi...@apache.org>.
bithw1 commented on issue #11996:
URL: https://github.com/apache/hudi/issues/11996#issuecomment-2375605864
Thanks @KnightChess
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org