You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by "tustvold (via GitHub)" <gi...@apache.org> on 2024/06/01 10:27:52 UTC

Re: [I] Improve memory overhead of parquet encoder [arrow-rs]

tustvold commented on issue #5828:
URL: https://github.com/apache/arrow-rs/issues/5828#issuecomment-2143397164

   One could definitely envisage the dictionary encoder buffering the encoded indices.
   
   In the short-term you could likely reduce the memory consumption by limiting the maximum number of values per page, e.g. to the recommended 20,000. This should reduce the memory consumption significantly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org