Re: [I] HoodieFileIndex file listing perf issue [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #14625:
URL: https://github.com/apache/hudi/issues/14625#issuecomment-4795212871

   This appears to be already fixed on master by PR #12125 (HUDI-8390), commit 7c52728dbfc5 (Oct 2024). The nested-loop hot spot you flagged in `HoodieFileIndex#filterFileSlices` was rewritten so the inner loop iterates the small per-slice file list (base + log files) and does O(1) `Set.contains` lookups against `candidateFilesNamesOpt`, instead of scanning the full ~55.8k candidate-files list for every slice. Since you were running on a 0.14.1-based branch, you would not have had this fix. Could you retry on master (or a recent 1.x release) and confirm the driver-side stall is gone? If so, we can close this out. cc @daviszhang


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org