[GitHub] [iceberg] xloya opened a new pull request #4427: [Spark][Core]: Release memory when rewriting RewriteFileGroup finished

[GitBox <gi...@apache.org>]

xloya opened a new pull request #4427:
URL: https://github.com/apache/iceberg/pull/4427


   Please see this for the initial patch: #4359, this patch makes some optimizations.  
   When rewriting, `Tasks` do not need to retain successful tasks for example `RewriteFileGroup` to revert. However, the references to these RewriteFileGroups are currently retained, and the memory will not be released after the rewriting finished. When rewriting the v2 table, a `RewriteFileGroup` may contain a large number of data files and delete files. After the `partial commit` is enabled, the memory overhead of the driver will also increase as the rewrite proceeds. So it makes sense to release memory here.  
   @RussellSpitzer Could you help to review again when you have time? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org