[I] Optimize scanInternalv2 to single pass [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot opened a new issue, #16243:
URL: https://github.com/apache/hudi/issues/16243

   The current algorithm take two passes over the log blocks: # First pass to collect all the valid blocks alongwith block instant times including rollback block's target instant time.
    # Second pass, in rever order of block instant time, to track final compacted instant times for each block.
   
   Now that we have removed appending to the same log file for multiple deltacommits, we can probably scan in single pass by keeping an active list or hash map of block times to their corresponding block, updating as we go. Should be tested for:
    # Out of order merged blocks: Log compaction is scheduled and by the time it appended a block, another block is added by another writer.
    # Log compaction operation failed, so a rollback is issued for this block. Here the rollback can be next block or can come at a later point of time.
    # Log compaction is executing and, before committing, compaction starts running on the same file group.
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-6888
   - Type: Task
   - Epic: https://issues.apache.org/jira/browse/HUDI-3580
   - Fix version(s):
     - 1.1.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Optimize scanInternalv2 to single pass [hudi]

["rangareddy (via GitHub)" <gi...@apache.org>]

rangareddy commented on issue #16243:
URL: https://github.com/apache/hudi/issues/16243#issuecomment-5434650221

   This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-6888).
   
   **Findings: the linked PR has not merged; keeping this open.**
   
   PR #9762 (`[HUDI-6888] [WIP] Optimize scanInternalV2 in AbstractHoodieLogRecordReader`) is still **open and unmerged**, so nothing has landed. No commit on `master` references HUDI-6888 either.
   
   Keeping this open until it merges.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Optimize scanInternalv2 to single pass [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot commented on issue #16243:
URL: https://github.com/apache/hudi/issues/16243#issuecomment-3630177035

   ### Linked PR(s)
   
   - https://github.com/apache/hudi/pull/9762


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org