You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by 徐涛 <ha...@gmail.com> on 2019/06/13 02:22:52 UTC

Source code question - about the logic of calculating network buffer

Hi Experts,
	I am debugging the WordCount Flink streaming program in local mode. Flink version is 1.7.2
	I saw the following calculation logic about network buffer in class TaskManagerServices.	
	jvmHeapNoNet is equal to -xmx amount in Java.
	why the networkBufBytes = xmx / (1-networkBufFraction) * networkBufFraction, it is a bit strange.
	why it is not calculated by xmx * networkBufFraction?
 
final long networkBufBytes = Math.min(networkBufMax, Math.max(networkBufMin,
   (long) (jvmHeapNoNet / (1.0 - networkBufFraction) * networkBufFraction)));

Best 
Henry

Re: Source code question - about the logic of calculating network buffer

Posted by 徐涛 <ha...@gmail.com>.
Hi Yun,
	Thanks a lot for the detailed and clear explanation, that is very helpful.

Best
Henry

> 在 2019年6月13日，上午10:32，Yun Gao <yu...@aliyun.com> 写道：
> 
> Hi tao,
> 
>       As a whole, `networkBufBytes` is not part of the heap. In fact, it is allocated from the direct memory. The rough relationship (ignores min/max and assumes managed memory is allocated on heap) between the variables are:
>         
>               Total memory of TM (configured by taskmanager.heap.mb, the name may be a little misleading) = heap memory (-XmX) + networkBufBytes.
>               networkBufBytes = Total memory of TM * networkBufFraction.
> 
>      Therefore, the networkBufBytes = xmx / (1 - networkBufFraction) * networkBufFraction.
> 
> Best,
> Yun
> ------------------------------------------------------------------
> From:徐涛 <ha...@gmail.com>
> Send Time:2019 Jun. 13 (Thu.) 10:23
> To:user <us...@flink.apache.org>
> Subject:Source code question - about the logic of calculating network buffer
> 
> Hi Experts,
> I am debugging the WordCount Flink streaming program in local mode. Flink version is 1.7.2
> I saw the following calculation logic about network buffer in class TaskManagerServices.
> jvmHeapNoNet is equal to -xmx amount in Java.
> why the networkBufBytes = xmx / (1-networkBufFraction) * networkBufFraction, it is a bit strange.
> why it is not calculated by xmx * networkBufFraction?
>  
> final long networkBufBytes = Math.min(networkBufMax, Math.max(networkBufMin,
>    (long) (jvmHeapNoNet / (1.0 - networkBufFraction) * networkBufFraction)));
> 
> Best 
> Henry
>

Re: Source code question - about the logic of calculating network buffer

Posted by Yun Gao <yu...@aliyun.com>.
Hi tao,

      As a whole, `networkBufBytes` is not part of the heap. In fact, it is allocated from the direct memory. The rough relationship (ignores min/max and assumes managed memory is allocated on heap) between the variables are:
              Total memory of TM (configured by taskmanager.heap.mb, the name may be a little misleading) = heap memory (-XmX) + networkBufBytes.
              networkBufBytes = Total memory of TM * networkBufFraction.

     Therefore, the networkBufBytes = xmx / (1 - networkBufFraction) * networkBufFraction.

Best,
Yun
------------------------------------------------------------------
From:徐涛 <ha...@gmail.com>
Send Time:2019 Jun. 13 (Thu.) 10:23
To:user <us...@flink.apache.org>
Subject:Source code question - about the logic of calculating network buffer

Hi Experts,
 I am debugging the WordCount Flink streaming program in local mode. Flink version is 1.7.2
 I saw the following calculation logic about network buffer in class TaskManagerServices.
 jvmHeapNoNet is equal to -xmx amount in Java.
 why the networkBufBytes = xmx / (1-networkBufFraction) * networkBufFraction, it is a bit strange.
 why it is not calculated by xmx * networkBufFraction?
final long networkBufBytes = Math.min(networkBufMax, Math.max(networkBufMin,
   (long) (jvmHeapNoNet / (1.0 - networkBufFraction) * networkBufFraction)));

Best 
Henry