You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vamshikrishnakyatham (via GitHub)" <gi...@apache.org> on 2025/10/03 04:13:42 UTC
[I] INSERT INTO should preserve duplication with hoodie.spark.sql.insert.into.operation=insert [hudi]
vamshikrishnakyatham opened a new issue, #14049:
URL: https://github.com/apache/hudi/issues/14049
### Bug Description
**What happened:**
INSERT INTO operations silently deduplicate records when tables have preCombineField defined, even when explicitly setting `hoodie.spark.sql.insert.into.operation=insert`. This violates standard SQL INSERT semantics where duplicates should be preserved.
**What you expected:**
INSERT operations should preserve all records including duplicates, matching standard SQL behavior. Only UPSERT operations should perform deduplication.
**Steps to reproduce:**
1. Create a Hudi table with primaryKey and preCombineField properties
2. Insert initial data (e.x, 5 records)
3. Set `hoodie.spark.sql.insert.into.operation=insert`
4. For ex, Execute INSERT INTO table SELECT * FROM table
5. Observe that there are duplicates
### Environment
**Hudi version:** 1.1
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] INSERT INTO should preserve duplication with hoodie.spark.sql.insert.into.operation=insert [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #14049: INSERT INTO should preserve duplication with hoodie.spark.sql.insert.into.operation=insert
URL: https://github.com/apache/hudi/issues/14049
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org