You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "liujinhui1994 (via GitHub)" <gi...@apache.org> on 2023/09/14 09:19:16 UTC
[GitHub] [hudi] liujinhui1994 opened a new issue, #9714: [SUPPORT]
liujinhui1994 opened a new issue, #9714:
URL: https://github.com/apache/hudi/issues/9714
**Describe the problem you faced**
After turning on metadata for my task, I found that the data was duplicated.
;
**To Reproduce**
Steps to reproduce the behavior:
dataFrame.write().format("org.apache.hudi")
.option("hoodie.datasource.write.recordkey.field", "uuid")
.option("hoodie.datasource.write.partitionpath.field", “ds”)
.option("hoodie.parquet.small.file.limit", 128 * 1024 * 1024)
.option("hoodie.datasource.hive_sync.partition_fields", DS_)
.option("hoodie.deltastreamer.keygen.timebased.timestamp.type", "DATE_STRING")
.option("hoodie.upsert.shuffle.parallelism", 200)
.option("hoodie.datasource.write.operation", "upsert")
.option("hoodie.datasource.write.precombine.field", "time")
.option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.ComplexKeyGenerator")
.option("hoodie.datasource.write.table.type", "MERGE_ON_READ")
.option("hoodie.embed.timeline.server", true)
.option("hoodie.datasource.write.streaming.ignore.failed.batch", false)
.option("hoodie.cleaner.commits.retained", "15")
.option("hoodie.datasource.hive_sync.table_properties", "spark.sql.partitionProvider=catalog")
.option("hoodie.keep.min.commits", 25)
.option("hoodie.keep.max.commits", 30)
.option("hoodie.clean.async", false)
.option("hoodie.table.name", getIncHoodieTableName(cfg.dataBase, cfg.tableName, cfg.model))
.option("hoodie.datasource.write.payload.class","org.apache.hudi.common.model.DefaultHoodieRecordPayload")
.option("hoodie.payload.event.time.field", “ts”)
.option("hoodie.payload.ordering.field", “ts”)
.option("hoodie.write.markers.type", "DIRECT")
.option("hoodie.finalize.write.parallelism", cfg.parallelism)
.option("hoodie.rollback.parallelism", cfg.parallelism)
.option("hoodie.metadata.enable", "true")
.option("hoodie.metadata.index.bloom.filter.enable", "true")
.option("hoodie.metadata.index.column.stats.enable", "true")
.option("hoodie.metadata.index.column.stats.column.list", "uuid")
.option("hoodie.bloom.index.use.metadata", "true")
.options(map)
.mode(Append)
.save("G:/LJH111/datalake/data/dls/mysql/test/rt/" + cfg.dataBase);
the above is the hudi config fo the task.
There was no problem before when I added the following configuration.
map.put("hoodie.metadata.enable", "true");
map.put("hoodie.metadata.index.bloom.filter.enable", "true");
map.put("hoodie.metadata.index.column.stats.enable", "true");
map.put("hoodie.metadata.index.column.stats.column.list", cfg.pk);
map.put("hoodie.bloom.index.use.metadata", "true").
1.
2.
3.
4.
**Expected behavior**
The same uuid should not appear in different filegroups.
**Environment Description**
* Hudi version : 0.11.1
* Spark version : 3.2.1
* Hive version : 3.1.0
* Hadoop version : 3.2.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Enable metadata table, Spark write mor table duplicate data [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1768475807
@liuzhuang2017 We were able to reproduce this issue , related github issue here with all the details - https://github.com/apache/hudi/issues/9870
Tracking JIRA -
So, as both issues are same. We will track as part of that issue and JIRA and closing this one. Thanks a lot. Please reopen in case you feel otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "liujinhui1994 (via GitHub)" <gi...@apache.org>.
liujinhui1994 commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1720377835
> @liujinhui1994 Can you try using the later version of hudi i.e. 0.12.3 or 0.13.1.
Is it because of a certain PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1719165416
@liujinhui1994 Could you also please try turning off the property hoodie.bloom.index.use.metadata to identify the root cause of the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "liujinhui1994 (via GitHub)" <gi...@apache.org>.
liujinhui1994 commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1722660429
For some reason, the version we use to read Hudi is still stuck at 0.10.0. So I am cautious every time I upgrade.
I also want to reproduce it locally. It may be because the data samples are relatively small and the reproduction is not successful.
@ad1happy2go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Enable metadata table, Spark write mor table duplicate data [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
URL: https://github.com/apache/hudi/issues/9714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "liujinhui1994 (via GitHub)" <gi...@apache.org>.
liujinhui1994 commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1724873741
I discovered a phenomenon and found it statistically through SQL;
select count(*) as counts,uuid from xxx where ds_='2023-09-18' group by uuid having COUNT(*)>1
The maximum number of counts is 2, which means that only one piece of UUID data is repeated. @ad1happy2go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Enable metadata table, Spark write mor table duplicate data [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1754673726
@liujinhui1994 I kept the job running for long but couldn't got any duplicates. Any useful insights which you can share which could help us to triage it better. Do you see stage failures or rollbacks in the timeline. Did you tried to reproduce this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Enable metadata table, Spark write mor table duplicate data [hudi]
Posted by "liujinhui1994 (via GitHub)" <gi...@apache.org>.
liujinhui1994 commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1756668935
Yes, this problem persists in our production environment @ad1happy2go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1722784717
@liujinhui1994 Yes I tried to reproduce too with smaller dataset but that worked. Running it for a larger dataset and trying to reproduce. Please let us know if you able to reproduce. Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1737686880
@liujinhui1994 Is this UUID generated as part of code or it was part of input dataset itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "liujinhui1994 (via GitHub)" <gi...@apache.org>.
liujinhui1994 commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1738360075
it was part of input dataset itself. @ad1happy2go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Enable metadata table, Spark write mor table duplicate data [hudi]
Posted by "liujinhui1994 (via GitHub)" <gi...@apache.org>.
liujinhui1994 commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1756669374
Can you show me the configuration written by your hudi?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1723692201
@liujinhui1994 Just a question, By any chance are you generating your record key ie. UUID as part of code or it is part of your input dataset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1719161853
@liujinhui1994 Can you try using the later version of hudi i.e. 0.12.3 or 0.13.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Enable metadata table, Spark write mor table duplicate data [hudi]
Posted by "liujinhui1994 (via GitHub)" <gi...@apache.org>.
liujinhui1994 commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1759014656
@ad1happy2go That's really strange
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Enable metadata table, Spark write mor table duplicate data [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1757232992
```
inserts.write.format("org.apache.hudi")
.option("hoodie.datasource.write.recordkey.field", "uuid")
.option("hoodie.datasource.write.partitionpath.field", "city")
.option("hoodie.parquet.small.file.limit", 128)
.option("hoodie.datasource.hive_sync.partition_fields", "city")
.option("hoodie.upsert.shuffle.parallelism", 200)
.option("hoodie.datasource.write.operation", "upsert")
.option("hoodie.datasource.write.precombine.field", "ts")
.option("hoodie.datasource.write.table.type", "MERGE_ON_READ")
.option("hoodie.embed.timeline.server", true)
.option("hoodie.datasource.write.streaming.ignore.failed.batch", false)
.option("hoodie.cleaner.commits.retained", "15")
.option("hoodie.datasource.hive_sync.table_properties", "spark.sql.partitionProvider=catalog")
.option("hoodie.keep.min.commits", 25)
.option("hoodie.keep.max.commits", 30)
.option("hoodie.clean.async", false)
.option("hoodie.table.name", tableName)
.option("hoodie.datasource.write.payload.class", "org.apache.hudi.common.model.DefaultHoodieRecordPayload")
.option("hoodie.payload.event.time.field", "ts")
.option("hoodie.payload.ordering.field", "ts")
.option("hoodie.write.markers.type", "DIRECT")
.option("hoodie.metadata.enable", "true")
.option("hoodie.metadata.index.bloom.filter.enable", "true")
.option("hoodie.metadata.index.column.stats.enable", "true")
.option("hoodie.metadata.index.column.stats.column.list", "uuid")
.option("hoodie.bloom.index.use.metadata", "true")
.mode(Append)
.save(basePath);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1720715959
@liujinhui1994 There were few known metadata table related issues in 0.11. So can you please try the later versions. Not fully sure if this issue will be related to those but better to upgrade. Meanwhile, I am working on this issue and trying to reproduce in my local setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "liujinhui1994 (via GitHub)" <gi...@apache.org>.
liujinhui1994 commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1720377633
> @liujinhui1994 Could you also please try turning off the property hoodie.bloom.index.use.metadata to identify the root cause of the issue?
hoodie.bloom.index.use.metadata If I change this to false, my writing will be very slow. The amount of data in this table is relatively large, and the table still uses bloom index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on issue #9714: [SUPPORT] Enable metadata table, Spark write mor table duplicate data
Posted by "liujinhui1994 (via GitHub)" <gi...@apache.org>.
liujinhui1994 commented on issue #9714:
URL: https://github.com/apache/hudi/issues/9714#issuecomment-1719082136
@codope @yihua please help look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org