You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/06 09:23:51 UTC

[GitHub] [arrow] LukasBec commented on issue #11043: [C++] Memory Usage of Parquet Stream Writer

LukasBec commented on issue #11043:
URL: https://github.com/apache/arrow/issues/11043#issuecomment-913493475


   Thanks for the answer!
   
   > One thing to note is that you are creating 400 columns even if you are only using 100 of them. This makes the row group metadata about 4x larger than it needs to be.
   
   I am actually  also using all 400 columns but i see that my example is a little confusing.
   
   > This is probably just an artifact of the way you are measuring memory usage. The VSZ/RSS numbers tend to grow in chunks because malloc will try and reduce the # of allocations it needs to make.
   
   Ah yes i forgot about that.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org