[GitHub] [arrow] StuartHadfield commented on issue #36303: PyArrow Write Dataset Memory Consumption - How does it work?

["StuartHadfield (via GitHub)" <gi...@apache.org>]

StuartHadfield commented on issue #36303:
URL: https://github.com/apache/arrow/issues/36303#issuecomment-1607666771

   Hmmm so after being basically stable at 1GB usage until 20m rows, the 20m-30m part has increased memory consumption to about 1.6GB. 
   
   It seems likely that using a 500k row limit will OOM my container. In this case, is my only option for making writes more memory efficient to use a lower row limit per file? Or perhaps if I lower the max number of open files, that might help too?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org