You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "chenrongwei (JIRA)" <ji...@apache.org> on 2016/12/22 02:23:58 UTC
[jira] [Updated] (YARN-6017) node manager physical memory leak
[ https://issues.apache.org/jira/browse/YARN-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chenrongwei updated YARN-6017:
------------------------------
Description:
In our produce environment, node manager's jvm memory has been set to '-Xmx2048m',but we notice that after a long time running the process' actual physical memory size had been reached to 12g (we got this value by top command as follow).
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31169 data 20 0 13.2g 12g 6092 S 16.9 13.0 49183:13 java
31169: /usr/local/jdk/bin/java -Dproc_nodemanager -Xmx2048m -Dhadoop.log.dir=/home/data/programs/apache-hadoop-2.7.1/logs -Dyarn.log.dir=/home/data/programs/apache-hadoop-2.7.1/logs -Dhadoop.log.file=yarn-data-nodemanager.log -Dyarn.log.file=yarn-data-nodemanager.log -Dyarn.home.dir= -Dyarn.id.str=data -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/home/data/programs/apache-hadoop-2.7.1/lib/native -Dyarn.policy.file=hadoop-policy.xml -XX:PermSize=128M -XX:MaxPermSize=256M -XX:+UseC
Address Kbytes Mode Offset Device Mapping
0000000000400000 4 r-x-- 0000000000000000 008:00001 java
0000000000600000 4 rw--- 0000000000000000 008:00001 java
0000000000601000 10094936 rw--- 0000000000000000 000:00000 [ anon ]
0000000770000000 2228224 rw--- 0000000000000000 000:00000 [ anon ]
00000007f8000000 131072 rw--- 0000000000000000 000:00000 [ anon ]
000000325ee00000 128 r-x-- 0000000000000000 008:00001 ld-2.12.so
000000325f01f000 4 r---- 000000000001f000 008:00001 ld-2.12.so
000000325f020000 4 rw--- 0000000000020000 008:00001 ld-2.12.so
000000325f021000 4 rw--- 0000000000000000 000:00000 [ anon ]
000000325f200000 1576 r-x-- 0000000000000000 008:00001 libc-2.12.so
000000325f38a000 2048 ----- 000000000018a000 008:00001 libc-2.12.so
000000325f58a000 16 r---- 000000000018a000 008:00001 libc-2.12.so
000000325f58e000 4 rw--- 000000000018e000 008:00001 libc-2.12.so
000000325f58f000 20 rw--- 0000000000000000 000:00000 [ anon ]
000000325f600000 92 r-x-- 0000000000000000 008:00001 libpthread-2.12.so
000000325f617000 2048 ----- 0000000000017000 008:00001 libpthread-2.12.so
000000325f817000 4 r---- 0000000000017000 008:00001 libpthread-2.12.so
000000325f818000 4 rw--- 0000000000018000 008:00001 libpthread-2.12.so
000000325f819000 16 rw--- 0000000000000000 000:00000 [ anon ]
000000325fa00000 8 r-x-- 0000000000000000 008:00001 libdl-2.12.so
000000325fa02000 2048 ----- 0000000000002000 008:00001 libdl-2.12.so
000000325fc02000 4 r---- 0000000000002000 008:00001 libdl-2.12.so
000000325fc03000 4 rw--- 0000000000003000 008:00001 libdl-2.12.so
000000325fe00000 28 r-x-- 0000000000000000 008:00001 librt-2.12.so
000000325fe07000 2044 ----- 0000000000007000 008:00001 librt-2.12.so
0000003260006000 4 r---- 0000000000006000 008:00001 librt-2.12.so
0000003260007000 4 rw--- 0000000000007000 008:00001 librt-2.12.so
0000003260200000 524 r-x-- 0000000000000000 008:00001 libm-2.12.so
0000003260283000 2044 ----- 0000000000083000 008:00001 libm-2.12.so
0000003260482000 4 r---- 0000000000082000 008:00001 libm-2.12.so
0000003260483000 4 rw--- 0000000000083000 008:00001 libm-2.12.so
0000003261200000 88 r-x-- 0000000000000000 008:00001 libresolv-2.12.so
0000003261216000 2048 ----- 0000000000016000 008:00001 libresolv-2.12.so
0000003261416000 4 r---- 0000000000016000 008:00001 libresolv-2.12.so
0000003261417000 4 rw--- 0000000000017000 008:00001 libresolv-2.12.so
was:
In our produce environment, node manager's jvm memory has been set to '-Xmx2048m',but we notice that after a long time running the process' actual physical memory size had been reached to 12g (we got this value by top command as follow).
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31169 data 20 0 13.2g 12g 6092 S 16.9 13.0 49183:13 java
> node manager physical memory leak
> ---------------------------------
>
> Key: YARN-6017
> URL: https://issues.apache.org/jira/browse/YARN-6017
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Affects Versions: 2.7.1
> Environment: OS:
> Linux guomai124041 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> jvm:
> java version "1.7.0_65"
> Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
> Reporter: chenrongwei
>
> In our produce environment, node manager's jvm memory has been set to '-Xmx2048m',but we notice that after a long time running the process' actual physical memory size had been reached to 12g (we got this value by top command as follow).
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 31169 data 20 0 13.2g 12g 6092 S 16.9 13.0 49183:13 java
> 31169: /usr/local/jdk/bin/java -Dproc_nodemanager -Xmx2048m -Dhadoop.log.dir=/home/data/programs/apache-hadoop-2.7.1/logs -Dyarn.log.dir=/home/data/programs/apache-hadoop-2.7.1/logs -Dhadoop.log.file=yarn-data-nodemanager.log -Dyarn.log.file=yarn-data-nodemanager.log -Dyarn.home.dir= -Dyarn.id.str=data -Dhadoop.root.logger=INFO,RFA -Dyarn.root.logger=INFO,RFA -Djava.library.path=/home/data/programs/apache-hadoop-2.7.1/lib/native -Dyarn.policy.file=hadoop-policy.xml -XX:PermSize=128M -XX:MaxPermSize=256M -XX:+UseC
> Address Kbytes Mode Offset Device Mapping
> 0000000000400000 4 r-x-- 0000000000000000 008:00001 java
> 0000000000600000 4 rw--- 0000000000000000 008:00001 java
> 0000000000601000 10094936 rw--- 0000000000000000 000:00000 [ anon ]
> 0000000770000000 2228224 rw--- 0000000000000000 000:00000 [ anon ]
> 00000007f8000000 131072 rw--- 0000000000000000 000:00000 [ anon ]
> 000000325ee00000 128 r-x-- 0000000000000000 008:00001 ld-2.12.so
> 000000325f01f000 4 r---- 000000000001f000 008:00001 ld-2.12.so
> 000000325f020000 4 rw--- 0000000000020000 008:00001 ld-2.12.so
> 000000325f021000 4 rw--- 0000000000000000 000:00000 [ anon ]
> 000000325f200000 1576 r-x-- 0000000000000000 008:00001 libc-2.12.so
> 000000325f38a000 2048 ----- 000000000018a000 008:00001 libc-2.12.so
> 000000325f58a000 16 r---- 000000000018a000 008:00001 libc-2.12.so
> 000000325f58e000 4 rw--- 000000000018e000 008:00001 libc-2.12.so
> 000000325f58f000 20 rw--- 0000000000000000 000:00000 [ anon ]
> 000000325f600000 92 r-x-- 0000000000000000 008:00001 libpthread-2.12.so
> 000000325f617000 2048 ----- 0000000000017000 008:00001 libpthread-2.12.so
> 000000325f817000 4 r---- 0000000000017000 008:00001 libpthread-2.12.so
> 000000325f818000 4 rw--- 0000000000018000 008:00001 libpthread-2.12.so
> 000000325f819000 16 rw--- 0000000000000000 000:00000 [ anon ]
> 000000325fa00000 8 r-x-- 0000000000000000 008:00001 libdl-2.12.so
> 000000325fa02000 2048 ----- 0000000000002000 008:00001 libdl-2.12.so
> 000000325fc02000 4 r---- 0000000000002000 008:00001 libdl-2.12.so
> 000000325fc03000 4 rw--- 0000000000003000 008:00001 libdl-2.12.so
> 000000325fe00000 28 r-x-- 0000000000000000 008:00001 librt-2.12.so
> 000000325fe07000 2044 ----- 0000000000007000 008:00001 librt-2.12.so
> 0000003260006000 4 r---- 0000000000006000 008:00001 librt-2.12.so
> 0000003260007000 4 rw--- 0000000000007000 008:00001 librt-2.12.so
> 0000003260200000 524 r-x-- 0000000000000000 008:00001 libm-2.12.so
> 0000003260283000 2044 ----- 0000000000083000 008:00001 libm-2.12.so
> 0000003260482000 4 r---- 0000000000082000 008:00001 libm-2.12.so
> 0000003260483000 4 rw--- 0000000000083000 008:00001 libm-2.12.so
> 0000003261200000 88 r-x-- 0000000000000000 008:00001 libresolv-2.12.so
> 0000003261216000 2048 ----- 0000000000016000 008:00001 libresolv-2.12.so
> 0000003261416000 4 r---- 0000000000016000 008:00001 libresolv-2.12.so
> 0000003261417000 4 rw--- 0000000000017000 008:00001 libresolv-2.12.so
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org