You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "mzheng-plaid (via GitHub)" <gi...@apache.org> on 2024/04/17 19:06:59 UTC
[I] [SUPPORT] hoodie.combine.before.insert silently broken for bulk_insert if meta fields disabled (causes duplicates) [hudi]
mzheng-plaid opened a new issue, #11044:
URL: https://github.com/apache/hudi/issues/11044
**Describe the problem you faced**
`hoodie.combine.before.insert` works with `bulk_insert` if the meta fields are enabled but silently does not work and causes duplicates if disabled (ie. `"hoodie.populate.meta.fields": "false"`).
**To Reproduce**
I provided a trivial reproduction below (`hoodie.populate.meta.fields` seems like only option that matters on whether bug happens):
```
# Generate dummy data
from pyspark.sql import Row
input_data = [
Row(
id=4,
value="foo",
ts=0,
),
Row(
id=4,
value="bar",
ts=1,
),
]
df = spark.createDataFrame(input_data)
# Example Hudi configs
hudi_options = {
"hoodie.table.name": "fake_name",
"hoodie.datasource.write.table.name": "fake_name",
"hoodie.datasource.write.table.type": "COPY_ON_WRITE",
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.metadata.enable": "false",
"hoodie.bootstrap.index.enable": "false",
"hoodie.datasource.write.partitionpath.field": "",
"hoodie.datasource.write.recordkey.field": "id",
"hoodie.datasource.write.precombine.field": "ts",
# Testing out bulk insert
"hoodie.combine.before.insert": "true",
"hoodie.datasource.write.operation": "bulk_insert",
}
PATH = "hdfs:///example_github
df.write.format("hudi").options(**hudi_options).mode("overwrite").save(PATH)
# Should be 1 but prints 2
print(spark.read.format("hudi").load(PATH).count())
# Both rows exist
print(spark.read.format("hudi").load(PATH).collect())
```
**Expected behavior**
The following is surprising:
- `bulk_insert` deduplicates properly if `hoodie.populate.meta.fields` is enabled
- `bulk_insert` does not deduplicate if `hoodie.populate.meta.fields` is disabled
- `insert` deduplicates properly regardless of `hoodie.populate.meta.fields`
I think the user expectation is that `bulk_insert` has consistent/documented behavior regardless of `hoodie.populate.meta.fields`. Ideally we would like `hoodie.combine.before.insert` to work.
**Environment Description**
This runs on https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-6101-release.html
* Hudi version : 0.12.2
* Spark version : 3.3.1
* Hive version : 3.1.3
* Hadoop version : 3.3.3
* Storage (HDFS/S3/GCS..) : HDFS/S3 both seem affected
* Running on Docker? (yes/no) : Yes
**Additional context**
We would love to update to a new version of Hudi but there are serious blocking bugs with key generators that are still open:
- https://github.com/apache/hudi/issues/10508 ("The ComplexKeyGenerator does not produce the same result for 0.14.1 than previous versions.")
- https://github.com/apache/hudi/issues/8372 (CustomKeyGenerator does not work with delete partitions operation)
Is there a way to workaround this on our current version of Hudi?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hoodie.combine.before.insert silently broken for bulk_insert if meta fields disabled (causes duplicates) [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11044:
URL: https://github.com/apache/hudi/issues/11044#issuecomment-2095323096
@mzheng-plaid This issue still exists in 0.14.1. I confirmed that this issue is only there for bulk_insert and hoodie.populate.meta.fields as false. (as mentioned by @mzheng-plaid )
Created a jira ticket for tracking too - https://issues.apache.org/jira/browse/HUDI-7717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hoodie.combine.before.insert silently broken for bulk_insert if meta fields disabled (causes duplicates) [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #11044: [SUPPORT] hoodie.combine.before.insert silently broken for bulk_insert if meta fields disabled (causes duplicates)
URL: https://github.com/apache/hudi/issues/11044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hoodie.combine.before.insert silently broken for bulk_insert if meta fields disabled (causes duplicates) [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on issue #11044:
URL: https://github.com/apache/hudi/issues/11044#issuecomment-2091400299
@codope are we targeting this fix for 0.15.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hoodie.combine.before.insert silently broken for bulk_insert if meta fields disabled (causes duplicates) [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on issue #11044:
URL: https://github.com/apache/hudi/issues/11044#issuecomment-2115676824
@mzheng-plaid @ad1happy2go The issue could be closed.
Fixed in master branch: 7fc5adad7aa9787e961c36536a08622f62fabe49
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hoodie.combine.before.insert silently broken for bulk_insert if meta fields disabled (causes duplicates) [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11044:
URL: https://github.com/apache/hudi/issues/11044#issuecomment-2115692210
Thanks a lot @geserdugarov
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org