You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by XiangWei Huang <xw...@gmail.com> on 2019/05/20 08:46:40 UTC
questions about yarn container（taskmanager）memory allocation
Hi all,
Currently i am running my flink application in yarn session mode and
using below commnad :
*bin/yarn-session.sh -d -s 3 -jm 1024 -tm 4096*
when taskmanager complete to started,i found the container launching
command is :
* bin/java -Xms2765m -Xmx2765m -XX:MaxDirectMemorySize=1331m ......*
from flink source code,i'v learned how direct memory size and heap memory
size calculated :
container total memory: 4096M
network memory fraction: 0.1
cut off memory fraction: 0.25
networkMemory = 4096*0.1 = 409.6
cut off memory = (4096 - 409.6) * 0.25 = 921.6
*directMemorySize* = 409.6 + 921.6 = 1331.2
*heapsize *= 4096 - 1331.2 = 2764.8
Below is the environment i'm using to run yarn and flink:
* jdk version: 1.8*
* flink version: 1.6.1*
*OS: centos7*
there are two questions about memory allocation i want to ask:
1. Since the jdk version i an using is 1.8,is it necessary to consider
the *metaspace
memory *into calculation.According to the current way to calculate memory
size（without metaspace memory） may cause
container running beyond physical memory limit.
2. Is the native memory that rocksdb using part of direct memory（limit by
jvm parameter *MaxDirectMemorySize*）,if not how do i control the size it
used.
Re: questions about yarn container（taskmanager）memory allocation
Posted by Xintong Song <to...@gmail.com>.
Hi XiangWei,
Thank you for the inputs. I agree with you that it is possible that
containers may use extra memory in 1.8. As for native memory, it is memory
used by JVM and other processes outside JVM. So it's not limited by
MaxDirectMemorySize.
The community is working on a refactoring plan about resource management of
Flink. AFAIK, the native memory is on the table, but I'm not sure whether
the metaspace memory is considered. I think we should create a jira issue
on this.
Thank you~
Xintong Song
On Mon, May 20, 2019 at 4:47 PM XiangWei Huang <xw...@gmail.com>
wrote:
> Hi all,
> Currently i am running my flink application in yarn session mode and
> using below commnad :
> *bin/yarn-session.sh -d -s 3 -jm 1024 -tm 4096*
> when taskmanager complete to started,i found the container launching
> command is :
> * bin/java -Xms2765m -Xmx2765m -XX:MaxDirectMemorySize=1331m ......*
> from flink source code,i'v learned how direct memory size and heap memory
> size calculated :
> container total memory: 4096M
> network memory fraction: 0.1
> cut off memory fraction: 0.25
> networkMemory = 4096*0.1 = 409.6
> cut off memory = (4096 - 409.6) * 0.25 = 921.6
> *directMemorySize* = 409.6 + 921.6 = 1331.2
> *heapsize *= 4096 - 1331.2 = 2764.8
>
> Below is the environment i'm using to run yarn and flink:
> * jdk version: 1.8*
> * flink version: 1.6.1*
> *OS: centos7*
>
> there are two questions about memory allocation i want to ask:
> 1. Since the jdk version i an using is 1.8,is it necessary to consider the *metaspace
> memory *into calculation.According to the current way to calculate memory
> size（without metaspace memory） may cause
> container running beyond physical memory limit.
> 2. Is the native memory that rocksdb using part of direct memory（limit by
> jvm parameter *MaxDirectMemorySize*）,if not how do i control the size it
> used.
>
Re: questions about yarn container（taskmanager）memory allocation
Posted by Xintong Song <to...@gmail.com>.
Hi XiangWei,
Thank you for the inputs. I agree with you that it is possible that
containers may use extra memory in 1.8. As for native memory, it is memory
used by JVM and other processes outside JVM. So it's not limited by
MaxDirectMemorySize.
The community is working on a refactoring plan about resource management of
Flink. AFAIK, the native memory is on the table, but I'm not sure whether
the metaspace memory is considered. I think we should create a jira issue
on this.
Thank you~
Xintong Song
On Mon, May 20, 2019 at 4:47 PM XiangWei Huang <xw...@gmail.com>
wrote:
> Hi all,
> Currently i am running my flink application in yarn session mode and
> using below commnad :
> *bin/yarn-session.sh -d -s 3 -jm 1024 -tm 4096*
> when taskmanager complete to started,i found the container launching
> command is :
> * bin/java -Xms2765m -Xmx2765m -XX:MaxDirectMemorySize=1331m ......*
> from flink source code,i'v learned how direct memory size and heap memory
> size calculated :
> container total memory: 4096M
> network memory fraction: 0.1
> cut off memory fraction: 0.25
> networkMemory = 4096*0.1 = 409.6
> cut off memory = (4096 - 409.6) * 0.25 = 921.6
> *directMemorySize* = 409.6 + 921.6 = 1331.2
> *heapsize *= 4096 - 1331.2 = 2764.8
>
> Below is the environment i'm using to run yarn and flink:
> * jdk version: 1.8*
> * flink version: 1.6.1*
> *OS: centos7*
>
> there are two questions about memory allocation i want to ask:
> 1. Since the jdk version i an using is 1.8,is it necessary to consider the *metaspace
> memory *into calculation.According to the current way to calculate memory
> size（without metaspace memory） may cause
> container running beyond physical memory limit.
> 2. Is the native memory that rocksdb using part of direct memory（limit by
> jvm parameter *MaxDirectMemorySize*）,if not how do i control the size it
> used.
>