[GitHub] wg1026688210 commented on a change in pull request #7185: [FLINK-10884] [yarn/mesos] adjust container memory param to set a safe margin from offheap memory

[GitBox <gi...@apache.org>]

wg1026688210 commented on a change in pull request #7185: [FLINK-10884] [yarn/mesos]  adjust  container memory param  to set a safe margin from offheap memory
URL: https://github.com/apache/flink/pull/7185#discussion_r237810314
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/ContaineredTaskManagerParameters.java
 ##########
 @@ -158,8 +158,10 @@ public static ContaineredTaskManagerParameters create(
 
 		// (2) split the remaining Java memory between heap and off-heap
 		final long heapSizeMB = TaskManagerServices.calculateHeapSizeMB(containerMemoryMB - cutoffMB, config);
-		// use the cut-off memory for off-heap (that was its intention)
-		final long offHeapSizeMB = containerMemoryMB - heapSizeMB;
+		// (3) try to compute the offHeapMemory from a safe margin
+		final long restMemoryMB = containerMemoryMB - heapSizeMB;
+		final long offHeapCutoffMemory = calculateOffHeapCutoffMB(config, restMemoryMB);
 
 Review comment:
   Thank you for your attention. I do add a containerized.offheap-cutoff-ratio for two reason
    1.  it will be unfriendly for  a small memory container such as 1gb memory container  if we only use containerized.heap-cutoff-ratio and containerized.heap-cutoff-min to cutoff the memory 0f container physical memory .  
    2. native memory  not only  contains memory which used by rockdb state backend but also   memory pools are called arenas which will cost enormous memory in multi-core cpu machine
   as time goes on . You can use 'pmap -x {TM container pid}' command to validate it 
    So I want to  add a  option to  adjust memory  flexiblely  for someone who wana adjust both heap meomory and offheap memory to adapt the three kinds of memory.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services