You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2024/07/09 16:12:04 UTC
[PR] [HUDI-7969] Fix data loss caused by concurrent write and clean [hudi]
Zouxxyy opened a new pull request, #11600:
URL: https://github.com/apache/hudi/pull/11600
### Change Logs
Suppose we have two concurrent jobs, write and clean, here is the problem flow:
1. insert job: write partition p1, but has not started writing files
2. clean job: through list partitions, get p1, and finds that there are no files in p1, then adds p1 to the partitions list to be deleted
3. insert job, starts writing files to partition p1 and commits
4. clean job, deletes partition p1, data is lost
### Impact
fix it
### Risk level (write none, low medium or high below)
medium
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7969] Fix data loss caused by concurrent write and clean [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11600:
URL: https://github.com/apache/hudi/pull/11600#issuecomment-2218217394
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "88a24e8f622541b989a1e25c67090ce0b9326a12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88a24e8f622541b989a1e25c67090ce0b9326a12",
"triggerType" : "PUSH"
}, {
"hash" : "3f20dfb03ff5df3345c410b06495780b438a3e6d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f20dfb03ff5df3345c410b06495780b438a3e6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 88a24e8f622541b989a1e25c67090ce0b9326a12 UNKNOWN
* 3f20dfb03ff5df3345c410b06495780b438a3e6d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7969] Fix data loss caused by concurrent write and clean [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11600:
URL: https://github.com/apache/hudi/pull/11600#issuecomment-2218186484
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "88a24e8f622541b989a1e25c67090ce0b9326a12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88a24e8f622541b989a1e25c67090ce0b9326a12",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 88a24e8f622541b989a1e25c67090ce0b9326a12 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7969] Fix data loss caused by concurrent write and clean [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11600:
URL: https://github.com/apache/hudi/pull/11600#discussion_r1671438127
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java:
##########
@@ -451,8 +451,12 @@ && noSubsequentReplaceCommit(earliestInstant.getTimestamp(), partitionPath)) {
* <p>IMPORTANT: {@code fsView.getAllFileGroups} does not return pending file groups for metadata table,
* file listing must be used instead.
*/
- private boolean hasPendingFiles(String partitionPath) {
+ private boolean mayHavePendingFiles(String partitionPath) {
try {
+ // As long as there are pending commits never delete empty partitions, because they may write files to any partitions.
+ if (!hoodieTable.getMetaClient().getCommitsTimeline().filterInflightsAndRequested().empty()) {
+ return true;
Review Comment:
For streaming ingestion, there should always be an pending instant on the timeline, and the following up logic may never reach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7969] Fix data loss caused by concurrent write and clean [hudi]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy closed pull request #11600: [HUDI-7969] Fix data loss caused by concurrent write and clean
URL: https://github.com/apache/hudi/pull/11600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7969] Fix data loss caused by concurrent write and clean [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11600:
URL: https://github.com/apache/hudi/pull/11600#issuecomment-2218379680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "88a24e8f622541b989a1e25c67090ce0b9326a12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88a24e8f622541b989a1e25c67090ce0b9326a12",
"triggerType" : "PUSH"
}, {
"hash" : "3f20dfb03ff5df3345c410b06495780b438a3e6d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24790",
"triggerID" : "3f20dfb03ff5df3345c410b06495780b438a3e6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 88a24e8f622541b989a1e25c67090ce0b9326a12 UNKNOWN
* 3f20dfb03ff5df3345c410b06495780b438a3e6d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24790)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7969] Fix data loss caused by concurrent write and clean [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11600:
URL: https://github.com/apache/hudi/pull/11600#issuecomment-2218292886
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "88a24e8f622541b989a1e25c67090ce0b9326a12",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88a24e8f622541b989a1e25c67090ce0b9326a12",
"triggerType" : "PUSH"
}, {
"hash" : "3f20dfb03ff5df3345c410b06495780b438a3e6d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24790",
"triggerID" : "3f20dfb03ff5df3345c410b06495780b438a3e6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 88a24e8f622541b989a1e25c67090ce0b9326a12 UNKNOWN
* 3f20dfb03ff5df3345c410b06495780b438a3e6d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24790)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org