Re: [I] Tuning merge small archive files [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #14993:
URL: https://github.com/apache/hudi/issues/14993#issuecomment-4795298367

   This issue tracks follow-up work for PR #4078, which implemented small archive file merging on the legacy log-based archived timeline (`HoodieTimelineArchiveLog`).
   
   That architecture was replaced by the LSM-tree archived timeline in [HUDI-6539 / PR #9209](https://github.com/apache/hudi/pull/9209) (commit d924f181633a, Aug 2023). The new design directly addresses the two items in the description:
   
   1. **Tuning merge strategy** — `LSMTimelineWriter.compactAndClean()` uses a layered LSM compaction with `hoodie.timeline.compaction.batch.size` and a per-file size limit, replacing the old single-level merge plan. See `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/versioning/v2/LSMTimelineWriter.java`.
   2. **Multi-writer** — A single JSON manifest version pointer provides read/write snapshot isolation across concurrent writers; see `LSMTimeline` and the `TimelineArchiverV2` flow.
   
   Subsequent corner-case fixes have landed since then (e.g. #18408 for leftover merge files, #17784 for large archive files blocking compaction). The original concern is no longer applicable to current master. Closing as obsolete — please reopen with a concrete repro on the LSM timeline if the issue still surfaces.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org