[GitHub] [hudi] ad1happy2go commented on issue #8824: [SUPPORT] Performance and Data Integrity Issues with Hudi for Long-Term Data Retention

["ad1happy2go (via GitHub)" <gi...@apache.org>]

ad1happy2go commented on issue #8824:
URL: https://github.com/apache/hudi/issues/8824#issuecomment-1571419634

   @machadoluiz After updating min and max commits , we expect latency to increase significantly as while writing Hudi performs a index lookup stage in which it loads all the active partitions and perform the same.
   
   When your min.commit and max.commit were small, it only loads last 30 commits and do index lookup for upsets. That's why you may have saw the duplicates coming if the update is updating a row which is ingested 30 commits ago.
   
   When your min.commit and max.commit were increased now it has to load those many commits to do index lookup which will involve shuffle.  You can look into tuning guide to understand if you are improve on performance by tweaking spark configuration. https://hudi.apache.org/docs/tuning-guide/
   
   May be know when you say latency increased from 00m:25s to 02m:30s, How many commits were there in the later case?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org