[GitHub] [iceberg] RussellSpitzer commented on issue #3703: DeleteOrphanFiles or ExpireSnapshots outofmemory

[GitBox <gi...@apache.org>]

RussellSpitzer commented on issue #3703:
URL: https://github.com/apache/iceberg/issues/3703#issuecomment-991101986


   We have considered this but we were worried about overwhelming distributed file systems and losing control over the parallelism of the job. That said have you tried just increasing driver memory, we have folks doing some rather large remove orphan and expire snapshot jobs and they just use larger drivers when required (they also usually increase the parallelism of the driver delete process as well.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org