[GitHub] [incubator-celeborn] pan3793 commented on pull request #1707: [CELEBORN-789] Increase default value of flushBuffer's max components

["pan3793 (via GitHub)" <gi...@apache.org>]

pan3793 commented on PR #1707:
URL: https://github.com/apache/incubator-celeborn/pull/1707#issuecomment-1630631305

   My test shows that 128~256 works better in low memory cases (2g off-heap memory per worker)
   
   When lower than 128, the CPU cost is very high on CompositeBuffer compaction, and there is a high probability that "direct memory usage" < "disk buffer size", since pause is judged by "direct memory usage", OOM happens frequently.
   
   Even with 128, sometimes "direct memory usage" < "disk buffer size"
   
   ```
   Direct memory usage: 1476.0 MiB/2.0 GiB, disk buffer size: 1850.7 MiB
   ```
   
   On the other hand, with the growth of "maxComponents", memory fragmentation grows rapidly too.
   
   My final choice is 192. But I think setting 128 in default should be fine considering most users will give it a larger resource than me.
   
   <img width="1661" alt="image" src="https://github.com/apache/incubator-celeborn/assets/26535726/49d0522d-1739-4e90-b6c6-1a6ec223c491">
   
   <img width="1274" alt="image" src="https://github.com/apache/incubator-celeborn/assets/26535726/c4c829b6-5983-4f77-90c9-3a2a5f97e1f0">
   
   <img width="1317" alt="image" src="https://github.com/apache/incubator-celeborn/assets/26535726/db9b7c53-bac8-47fb-90e6-1594aafe8ea5">
   
   <img width="1251" alt="image" src="https://github.com/apache/incubator-celeborn/assets/26535726/60a33054-f18e-4ad7-8692-c3242a299534">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org