[GitHub] [arrow] ssirovica commented on issue #14007: [Go] Writing records to parquet with arrow.BinaryTypes.String leaks memory

[GitBox <gi...@apache.org>]

ssirovica commented on issue #14007:
URL: https://github.com/apache/arrow/issues/14007#issuecomment-1232167498

   I confirmed your behavior.
   
   Sorry, I set the test loop a little low. 50,000,000 is not enouch to surface the issue. If you bump it up to 5,000,000,000 (100x) as much you should see the memory grow large.
   
   The machine I'm on is an Intel based macbook. My reported memory from top is over 10G about 40seconds into the program running.
   
   > Also, since you're using the default allocator (which is just Go's allocator) it's also possible that the memory you're building up is from the reallocations of continuously expanding the string builder's buffers (since you're not reserving the memory) and then the Garbage collector hasn't run to clean it up yet depending on your options.
   
   I thought maybe it was something related here. So I have tried forcing the garbage collector to run after every right. That doesn't collect the memory. My suspicion is somewhere in the write path the pointer to the buffers are retained so they don't get GC'd. 
   
   Happy to share some more findings.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org