[GitHub] [iceberg] aokolnychyi edited a comment on pull request #1211: Use Spark for Scanning of Manifest Files during Snapshot Expiration

[GitBox <gi...@apache.org>]

aokolnychyi edited a comment on pull request #1211:
URL: https://github.com/apache/iceberg/pull/1211#issuecomment-659490881


   Thanks, @RussellSpitzer! Let me take a look at this today.
   
   To give a bit of background, we had a case where people compacted their table that reduced the number of files from 5+ million to under 1 million in a short period of time. That processed generated a huge number of new snapshots as different parts were optimized separately (one leaf partition at a time). In addition, there was a huge number of manifests to scan and a lot of expired data files. As a result, the expiry of snapshots was really slow.
   
   This is an attempt to fix such edge cases.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org