You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/10/28 09:10:07 UTC

[GitHub] [hadoop] ibobak commented on pull request #2723: HADOOP-13126 Add BrotliCodec based on Brotli4j library

ibobak commented on PR #2723:
URL: https://github.com/apache/hadoop/pull/2723#issuecomment-1294738611

   Colleagues, 
   
   I've taken the source code from this commit  https://github.com/apache/hadoop/pull/2723/commits/47f05930c2f5c576a6c25238c187bdf3409b8f23 
   
   made a jar of it, plugged it into my Spark cluster, launched a huge job with many transformations and actions, and found that there is a serious memory leak: executors consume RAM more and more (no matter that there is a limitation of 20GB, they consumed 40GB).
   
   I've made my own version of Brotli codec (also based on brotli4j) by looking at how Snappy and others are made, and it works with no memory leaks.  Soon I'll post my PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org