You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 03:47:23 UTC
[I] Relax compaction in metadata being fenced based on inflight requests in data table [hudi]
hudi-bot opened a new issue, #14864:
URL: https://github.com/apache/hudi/issues/14864
Relax compaction in metadata being fenced based on inflight requests in data table.
Compaction in metadata is triggered only if there are no inflight requests in data table. This might cause liveness problem since for very large deployments, we could either have compaction or clustering always in progress. So, we should try to see how we can relax this constraint.
Proposal to remove this dependency:
With recent addition of spurious deletes config, we can actually get away with this.
As of now, we have 3 inter linked nuances.
- Compaction in metadata may not kick in, if there are any inflight operations in data table.
- Rollback when being applied to metadata table has a dependency on last compaction instant in metadata table. We might even throw exception if instant being rolledback is < latest metadata compaction instant time.
- Archival in data table is fenced by latest compaction in metadata table.
So, just incase data timeline has any dangling inflght operation (lets say someone tried clustering, and killed midway and did not ever attempt again), metadata compaction will never kick in at all for good. I need to check what does archival do for such inflight operations in data table though when it tries to archive near by commits.
So, with spurious deletes support which we added recently, all these can be much simplified.
Whenever we want to apply a rollback commit, we don't need to take different actions based on whether the commit being rolled back is already committed to metadata table or not. Just go ahead and apply the rollback. Merging of metadata payload records will take care of this. If the commit was already synced, final merged payload may not have spurious deletes. If the commit being rolledback was never committed to metadata, final merged payload may have some spurious deletes which we can ignore.
With this, compaction in metadata does not need to have any dependency on inflight operations in data table.
And we can loosen up the dependency of archival in data table on metadata table compaction as well.
So, in summary, all the 3 dependencies quoted above will be moot if we go with this approach. Archival in data table does not have any dependency on metadata table compaction. Rollback when being applied to metadata table does not care about last metadata table compaction. Compaction in metadata table can proceed even if there are inflight operations in data table.
Especially our logic to apply rollback metadata to metadata table will become a lot simpler and is easy to reason about.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-2458
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1292
- Fix version(s):
- 1.1.0
---
## Comments
29/Nov/21 23:12;pwason;>> Archival in data table does not have any dependency on metadata table compaction.
With the sync design,[ we validate the deltacommits to read|[https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java#L293]] for metadata table based on which completed commits are present in the dataset. So dataset should never be archiving the instants before compaction on metadata table.
;;;
---
29/Nov/21 23:20;pwason;Another way to solve this issue is to think in terms of instant completion times. Currently, HUDI does not record time for completion of instants. But if that was available then all of these conditions may be met.
Assume the following dataset timeline with long running clustering op (Metadata table is not compacted due to the t2.replacecommit inflight):
t1.commit t2.replacecommit.inflight t3.commit
When t2 completes (say at time t4), it created t2.replacecommit. But what if there was a way to track that t2 completed at t4? Then metadata table could be tracking the completion times and compaction can proceed.
One way is to change the instant format to <start_time>-<end_time>.<action>
Another way is to check modifcation times of instant files (may not work across all cloud stores).;;;
---
22/Dec/21 02:27;shivnarayan;agreed. Archival in datatable has to have dependency on compaction in metadata table. can't do much there.
But the compaction reliance on inflght operations in data table can be relaxed and in turn rollback dependency on compaction can also be relaxed?
wrt your proposal on start and end time, yeah that would make this easier. [~alexey.kudinkin] might be looking into the start and end time.
;;;
---
29/Mar/23 19:33;shivnarayan;Documenting all intricacies we have b/w DT and MDT
1. Compaction in metadata may not kick in, if there are any inflight operations in data table.
2. Rollback when being applied to metadata table has a dependency on last compaction instant in metadata table. The commit being rolled back, if its already synced to MDT, we apply -F records. if not, we don't. this is to ensure we only do -F only if we have ever done +F before (no spurious deletes).
3. Archival in data table is fenced by latest compaction in metadata table.
- Not sure if this is really an issue. any failed commit in DT, will anyway stop the archival in DT.
4. MDT archival depends on earliest pending/inflight commit in DT.
5. rollback from DT when applied to MDT has a dep on archival in MDT: if the commit being rolled back is already archived in MDT, we throw an exception.
- But I can't think of a reason why would we archive. bcoz, MDT archival has a dependency on DT earliest inflight commit. and MDT compaction has a dependency on earliest inflight commit in DT. and so this may not be valid use-case or dependency(MDT archival fix was added later though)
6. HoodieMetadataLogRecordReader#getValidInstantTimestamps relies on the active instants on DT timeline for validity filtering. if for some reason, we make DT archival proceed beyond any inflight commits in DT, it could result in invalid data being served from MDT.
1 & 2 : bcoz, we have a constraint in MDT, that any instant which is < latest compaction is fully synced up w/ MDT(i.e. either completed or rolledback). In other words, there can't be a situation where, we can't have a partially failed commits in data table which is not yet synced to MDT and MDT compaction should not kick in. (esply if the commit is completed in MDT, but not in DT). If MDT compaction kicks in, incomplete data could be committed now.;;;
---
30/Mar/23 08:27;danny0405;With the restriction aboove, here is the illustration:
*Proposal*
* relax MDT pending instants deduction to compact to ignore compaction and clustering from DT;
* Ensure compaction of MDT ignores uncommitted data;
* this might work only for spurious deletes ignored flow.
*The new workflow*
1. t5: assumes there is requested compaction at t5 on DT timeline:
DT: t5.compaction.req
2. t10: the MDT compaction triggers (ignore the pending t5 on DT)
MDT: t10_002.commit
3. t20: DT triggers a ROLLBACK for t5
DT: t20.rb
MDT: t20.dc
We may not know if t5 was already committed to MDT or not.
And hence had to go for -F records. (only works for spurious deletes ignore flow).
4. t30: t5 is retried and succeed in MDT but failed in DT (attempt1)
MDT: t5.dc with files f10, f11.
DT: crash
5. t31: MDT compaction triggered
MDT: t31_002.commit
The rollback payload for t5 in MDT decides that t5 is already synced. We do -F.
6. t40: t5 is retried and filed to commit to MDT (attempt2)
DT: t5.compaction.req._2nd with files f20, f21.
crash
The rollback payload for t5 in MDT decides that t5 is not synced. We do -F with tombstone messages (only works for spurious deletes ignore flow));;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Relax compaction in metadata being fenced based on inflight requests in data table [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #14864: Relax compaction in metadata being fenced based on inflight requests in data table
URL: https://github.com/apache/hudi/issues/14864
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Relax compaction in metadata being fenced based on inflight requests in data table [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #14864:
URL: https://github.com/apache/hudi/issues/14864#issuecomment-3630210899
### Linked PR(s)
- https://github.com/apache/hudi/pull/4143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Relax compaction in metadata being fenced based on inflight requests in data table [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #14864:
URL: https://github.com/apache/hudi/issues/14864#issuecomment-3593855068
This issue belongs to a closed JIRA Epic: https://issues.apache.org/jira/browse/HUDI-1292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Relax compaction in metadata being fenced based on inflight requests in data table [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14864:
URL: https://github.com/apache/hudi/issues/14864#issuecomment-4795272585
This appears to have been addressed by [HUDI-7510](https://issues.apache.org/jira/browse/HUDI-7510) in commit `9b094e628d` (PR #10874, Apr 2024).
In current `master`, `HoodieBackedTableMetadataWriter#compactIfNecessary` no longer blocks MDT compaction when DT has inflight instants. Instead, it picks a compaction instant time that is `instantTimeMinusMillis(earliestPendingDtInstant, 1)` (see `HoodieBackedTableMetadataWriter.java` lines 2191-2202), letting the compaction proceed against log files that completed before that point. `validateCompactionScheduling` (lines 2276-2291) only checks for pending MDT compactions/log-compactions now — not DT inflight ops.
The legacy strict gate is preserved only in `HoodieBackedTableMetadataWriterTableVersionSix#validateCompactionScheduling` for table-version-6 compatibility.
If the goal in HUDI-2458 was specifically the liveness fix for the modern (v8+) path, this can likely be closed. If you intended additional simplifications around rollback / archival fencing, please clarify and we can reopen scope. cc folks on the JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org