[I] [Improvement] Avoid buffering unused record positions in native log writes [hudi]

["cshuo (via GitHub)" <gi...@apache.org>]

cshuo opened a new issue, #20116:
URL: https://github.com/apache/hudi/issues/20116

   ### Task Description
   
   **What needs to be done:**
   
   Avoid buffering record positions in `HoodieNativeLogFormatWriter` when position metadata is not available or enabled for the append. Gate collection in both `appendRecord()` and `appendDeleteRecord()` on `baseFileInstantTimeOfPositions.isPresent()`.
   
   **Why this task is needed:**
   
   The native log writer streams records into the underlying file writers, but currently maintains two additional `ArrayList<Long>` buffers:
   
   - `appendRecord()` unconditionally adds `record.getCurrentPosition()` to `dataRecordPositions`.
   - `appendDeleteRecord()` adds an entry to `deleteRecordPositions` for every deletion, using `-1L` when there is no base-file instant for positions.
   - At flush time, `addRecordPositionsIfRequired()` returns without using either buffer when the header lacks `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS`.
   
   Consequently, appends without position metadata retain O(N) unused list entries, where N is the total number of data and delete records accumulated before a flush. Clearing the lists after a flush releases their elements but retains the backing array capacity. Even when boxed values such as `-1L` are cached, the list reference arrays still grow with the record count.
   
   This is an avoidable heap-memory cost; no data correctness failure has been observed.
   
   **Reproduction:**
   
   1. Construct a native log writer with `baseFileInstantTimeOfPositions = Option.empty()`, as can occur when writing record positions is disabled or no base file is available.
   2. Append data records and/or delete records.
   3. Inspect the corresponding position lists before flushing: they contain one entry per appended record even though the footer will not use them.
   
   **Expected behavior and validation:**
   
   - With no base-file instant for positions, both position buffers remain empty throughout the append and the footer contains no position metadata.
   - With position metadata enabled, preserve the existing encoding behavior, including data-record order, duplicate positions, and invalid-position handling.
   - Regression tests should inspect the buffers before flush, in addition to checking the resulting footer. Footer-only assertions do not detect this unnecessary allocation.
   
   A local fix with this coverage passed all 13 tests in `TestHoodieNativeLogFormatWriter` using:
   
   ```bash
   mvn -pl hudi-client/hudi-client-common -am \
     -Dtest=TestHoodieNativeLogFormatWriter \
     -Dsurefire.failIfNoSpecifiedTests=false \
     -DskipITs -DskipSparkTests -DskipScalaTests test
   ```
   
   ### Task Type
   
   Performance optimization
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement] Avoid buffering unused record positions in native log writes [hudi]

["wombatu-kun (via GitHub)" <gi...@apache.org>]

wombatu-kun closed issue #20116: [Improvement] Avoid buffering unused record positions in native log writes
URL: https://github.com/apache/hudi/issues/20116


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org