[I] [RFC-112] Preserve the Hudi timeline on full refresh and isolate checkpoint generations [hudi]

["danny0405 (via GitHub)" <gi...@apache.org>]

danny0405 opened a new issue, #20404:
URL: https://github.com/apache/hudi/issues/20404

   ### Task Description
   
   Implement full-table TRUNCATE through an empty insert_overwrite_table replace commit, preserving timeline/savepoints and metadata-table consistency. Empty input must still replace all existing file groups. Coordinate this with streaming checkpoint-generation isolation: derive a stable identifier from target/flow/checkpoint generation, use it for both dedup checks and commit metadata, retain it on normal restart, and change it when SDP creates a fresh checkpoint directory.
   
   These changes must ship together: retaining old sink metadata while restarting batch IDs at zero under the old identifier can silently skip data. Define legacy-checkpoint migration and explicit physical-purge compatibility behavior. Manual checkpoint deletion in place is unsupported; checkpoint relocation requires an explicit rebuild/migration.
   
   Keep the truncate–alter–append non-atomic window explicit. Related #16342 covers truncate concurrency more broadly; coordinate implementation, but this task specifically couples timeline preservation to SDP checkpoint reset correctness.
   
   ### Acceptance criteria
   
   - [ ] Successful truncate publishes a completed replace commit, including for empty input; history/savepoints and metadata-table consistency are preserved subject to retention.
   - [ ] Failure before truncate commit preserves the previous snapshot; failure afterward leaves the documented empty snapshot.
   - [ ] Ordinary restart skips committed batches; full refresh replays batch 0..N under a new identifier without skipping old high-water marks.
   - [ ] Tests cover failure after Hudi commit but before Spark checkpoint completion, empty refresh, checkpoint-path changes, and legacy identifiers.
   - [ ] Ordinary SQL truncate regression tests and migration/retention documentation are included.
   
   ### Task Type
   
   Code improvement/refactoring
   
   ### Related Issues
   
   Parent feature issue: #20401
   RFC: #19997 (under review; implementation must follow the agreed contract).
   This task can start independently; coordinate integration with the sibling tasks under #20401.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org