You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "fujianhua168 (via GitHub)" <gi...@apache.org> on 2023/07/20 12:52:00 UTC
[GitHub] [hudi] fujianhua168 opened a new issue, #9244: [SUPPORT] After a Hadoop datanode failure, Spark3.2 encountered duplicate data on the same primary key while reading the hudi bucket more table data
fujianhua168 opened a new issue, #9244:
URL: https://github.com/apache/hudi/issues/9244
**Describe the problem you faced**
A few days ago in the production environment, a datanode in the Hadoop cluster malfunctioned, causing the flink streaming write task( for hudi bucket mor table) failed. After restarting the Flink task, when we used Spark3.2 or Presto333 to read data from the table, we found duplicate data under the same primary key ，yet the duplicate records have the same Hudi system field values (_hoodie_commit_time, _hoodie_commit_seqno, _hoodie_filename) .
Note: This Flink write task has been running normally for several days.
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark3.2 encountered duplicate data while reading the hudi bucket MOR table [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #9244:
URL: https://github.com/apache/hudi/issues/9244#issuecomment-4845368525
Closing this out — it's a Flink write-failure-recovery scenario, and the reader-side gap it exposed is addressed in 1.x.
What happened: the datanode downtime killed the Flink write mid-flight and left an uncommitted / un-rolled-back log block in the bucket file group; after restart, that leftover block was read alongside the retried write, producing duplicates that carry the failed write's hoodie meta fields.
In 1.x this is handled:
- the log reader now skips uncommitted log files/blocks (HUDI-6952, [#9879](https://github.com/apache/hudi/pull/9879), in 1.0.0+), so a failed write's leftover block is no longer read;
- the completion-time-based timeline (1.0) means readers only surface data from completed instants;
- Flink failover handling was hardened (partial failover [#7208], plus fixes to re-trigger failover after pending instants are recommitted).
The original report was on 0.13.0, which predates these. Since the timeline for the failed run was never shared and the relevant recovery/reader code has been substantially reworked, I'm closing this. If you can still reproduce duplicates after a write failure on a recent 1.x release (Flink + bucket index + MOR), please reopen with the .hoodie/ timeline listing around the failure and we'll dig in. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark3.2 encountered duplicate data while reading the hudi bucket MOR table [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #9244: [SUPPORT] Spark3.2 encountered duplicate data while reading the hudi bucket MOR table
URL: https://github.com/apache/hudi/issues/9244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9244: [SUPPORT] After a Hadoop datanode failure, Spark3.2 encountered duplicate data on the same primary key while reading the hudi bucket more table data
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9244:
URL: https://github.com/apache/hudi/issues/9244#issuecomment-1649147339
@fujianhua168 Is it possible to provide us the timeline to triage this better when datanode has downtimed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org