You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 08:05:45 UTC

[I] retain the commit before inflignt compaction to support schema Evolution [hudi]

hudi-bot opened a new issue, #16019:
URL: https://github.com/apache/hudi/issues/16019

   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-6361
   - Type: Improvement
   - Fix version(s):
     - 1.1.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] retain the commit before inflignt compaction to support schema Evolution [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on issue #16019:
URL: https://github.com/apache/hudi/issues/16019#issuecomment-3630247564

   ### Linked PR(s)
   
   - https://github.com/apache/hudi/pull/8938


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] retain the commit before inflignt compaction to support schema Evolution [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.

nsivabalan closed issue #16019: retain the commit before inflignt compaction to support schema Evolution
URL: https://github.com/apache/hudi/issues/16019


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] retain the commit before inflignt compaction to support schema Evolution [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.

nsivabalan commented on issue #16019:
URL: https://github.com/apache/hudi/issues/16019#issuecomment-4795518011

This appears to have been addressed in current master, though through a different change than the originally linked PR #8938 (which was closed unmerged).

- `CleanerUtils.getEarliestCommitToRetain` (`hudi-common/src/main/java/org/apache/hudi/common/util/CleanerUtils.java`, lines 135-146) caps the earliest-commit-to-retain at the last completed instant **before** the earliest pending instant, so the cleaner will not remove the commit immediately preceding an inflight compaction.
- `TimelineArchiverV2.getCommitInstantsToArchive` (`hudi-client/hudi-client-common/.../v2/TimelineArchiverV2.java`, lines 195-214) and `TimelineArchiverV1` apply the same rule on the archival side.

The cleaner-side guard was added via HUDI-6370 / PR #8972 (commit `ee5bb7c2e369`, 'Automatically decide archival boundary based on cleaning configs'). Schema/checkpoint resolution across compaction has been further hardened by PR #18576 (schema lookup falls back to non-ingestion commits) and PR #18590 (clean instants carry rolling metadata). Could this issue and HUDI-6361 be closed? If there's still a specific repro where the schema chain breaks across an inflight compaction in master, please share details.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org