[GitHub] NicoK commented on issue #7185: [FLINK-10884] [yarn/mesos] adjust container memory param to set a safe margin from offheap memory

[GitBox <gi...@apache.org>]

NicoK commented on issue #7185: [FLINK-10884] [yarn/mesos]  adjust  container memory param  to set a safe margin from offheap memory
URL: https://github.com/apache/flink/pull/7185#issuecomment-447890879
 
 
   Let me drop some general notes: the `cutoff` was meant to include anything but the heap and thus cover for classes (metaspace), off-heap, etc. and when I touched this code, I kept that.
   If you look into https://github.com/apache/flink/blob/master/flink-dist/src/main/flink-bin/bin/taskmanager.sh#L65, you'll also see that we practically set it to infinity there. Having an actual limit (as is used through `ContaineredTaskManagerParameters`) will only result in a different error in case you are hitting it: `java.lang.OutOfMemoryError` instead of being killed by YARN/Mesos/etc. That may be better for diagnosing it but doesn't solve the root cause of something using up too much memory.
   
   We also need to touch all memory calculations for https://issues.apache.org/jira/browse/FLINK-10317 again - maybe splitting this off is already enough for you? So having `totalContainerMem = heap + offHeap + Metaspace`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services