You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ArpitAdhikari (via GitHub)" <gi...@apache.org> on 2024/08/27 10:12:33 UTC
[I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
ArpitAdhikari opened a new issue, #11836:
URL: https://github.com/apache/hudi/issues/11836
**Describe the problem you faced**
We are using hudi in our cdc pipeline which consumes from MSK topics. We are getting duplicate entries in our tables.
To brief the scenario, we are using Hudi v0.12.2 on emr-6.10.1. Our data is captured from MySQL events, which can be of insert or upsert type, also the spark application looks fine. The tables that we are seeing duplicates in are mostly MOR ones with BLOOM index.
**Expected behavior**
Any update to coming in to the pipeline should update the already existing record.
**Environment Description
The attachment here contains:
1. Query being used to detect the duplicates,
2. Metafields,
3. Sample output of the query showing duplicates
4. Hudi configs
[Attachment.txt](https://github.com/user-attachments/files/16760403/Attachment.txt)
**
* Hudi version : 0.12.2
* Spark version : Spark 3.3.1
* Hive version : Hive 3.1.3
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : Yes
**Additional context**
* EMR Version: emr-6.10.1
We have also tried removing the duplicates manually with a python script and reran the job, but after running for few days it again started inserting duplicate records.
Screenshot of the latest timeline :
<img width="1347" alt="Screenshot 2024-08-27 at 3 33 43 PM" src="https://github.com/user-attachments/assets/67e8c1d0-e82a-4467-9718-fdee6f90377f">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ArpitAdhikari (via GitHub)" <gi...@apache.org>.
ArpitAdhikari commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2331321525
@ad1happy2go Can we have the call at around 5:30 IST, my colleague would also want to join?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2331400862
@ArpitAdhikari Sure, lets conrdinate on slack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2312180740
@ArpitAdhikari Can you check your timeline if 20240807151109878 and 20240820153145763 exists in timeline? Can you please provide your zipped .hoodie directory if possible ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ArpitAdhikari (via GitHub)" <gi...@apache.org>.
ArpitAdhikari commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2314498406
Hey @ad1happy2go, I have gotten the same issue occuring for another table. I am attaching the query result for the query below:
SELECT * FROM table_a where _hoodie_record_key = '11111';
Output:
[queryOutput.csv](https://github.com/user-attachments/files/16777643/queryOutput.csv)
Timeline Zipped:
[hoodie_timeline.zip](https://github.com/user-attachments/files/16777805/hoodie_timeline.zip)
Also, I check the timeline and both the files are present in timeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2312271092
Sure. @ArpitAdhikari Also look for any rollbacks or spark retries. Can you use hudi 0.12.3 as that have some more bug fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy closed issue #11836: [SUPPORT] Duplicate records getting inserted at Hudi S3 sink
URL: https://github.com/apache/hudi/issues/11836
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "somanath-goudar (via GitHub)" <gi...@apache.org>.
somanath-goudar commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-3516432250
@rangareddy - We are facing the same issue. Would you be able to help us?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2331218177
Can you try with 0.14.1 or 0.15.0 to confirm if you still see the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ArpitAdhikari (via GitHub)" <gi...@apache.org>.
ArpitAdhikari commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2331301561
We recently moved to Hudi 0.14.1 to avoid this issue but just today got to see the duplicates again in few of the tableTo answer your question, yes we use aysnc cleaning with below properties:
"hoodie.write.concurrency.mode": "OPTIMISTIC_CONCURRENCY_CONTROL"
"hoodie.write.lock.provider": "org.apache.hudi.client.transaction.lock.InProcessLockProvider"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2331216694
@ArpitAdhikari There was no open issue related to this with BLOOM. There were couple issue one related to GLOBAL_BLOOM and another related hoodie.bloom.index.use.metadata as true.
I reviewed the timeline and i see you are using async cleaning. Are you setting lock provider for the same? I see that the dups got created by subsequent compaction commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2331308756
@ArpitAdhikari Thanks , just reviewed your configs too. It all looks good to me. So this issue might still be there. I tried reproducing but not able to do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2318283666
@ArpitAdhikari We had a known issue with bloom index - https://github.com/apache/hudi/pull/9886
which was fixed as part of https://github.com/apache/hudi/pull/9886
Can you try with latest Hudi version or try this fix if you still see dups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-3466712406
Hi @ArpitAdhikari
I hope this issue is now resolved. If you are still encountering problems, please let me know so we can schedule a quick call to troubleshoot. I'll keep this ticket open for one more week and close it if there's no further response. Feel free to reopen it if the issue becomes reproducible again in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-3491169014
Closing this issue because the user don't have any follow-up questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ArpitAdhikari (via GitHub)" <gi...@apache.org>.
ArpitAdhikari commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2320477100
Hey @ad1happy2go , can you confirm if this issue has been solved in Hudi v0.14.1?
We are running on emr-7.1.0, Hive 3.1.3, Spark 3.5.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Duplicate records getting inserted at Hudi S3 sink [hudi]
Posted by "ArpitAdhikari (via GitHub)" <gi...@apache.org>.
ArpitAdhikari commented on issue #11836:
URL: https://github.com/apache/hudi/issues/11836#issuecomment-2312244849
Since this was a production application, I had to remove the duplicates and rerun the job. I am trying to reproduce the scenario, once done will update here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org