You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Kenan Kılıçtepe <kk...@gmail.com> on 2023/09/06 12:32:03 UTC
Memory Leak
Hi,
I have Flink 1.16.2 on a single server with 64GB Ram.
Although taskmanager.memory.process.size is set to 40000m, I can see
memory usage of the task manager exceed 59GB and OS kills it because of
OOM.
I check the RSS column of application top for memory usage.
I don`t see any heap memory problem.
taskmanager.memory.process.size: 40000m
taskmanager.memory.managed.fraction: 0.53
state.backend.rocksdb.memory.managed: true
Any help is appreciated for analyzing the problem.
Thanks
Re: Memory Leak
Posted by "Neha . via user" <us...@flink.apache.org>.
We also faced the same issue with Flink 1.16.1. Please enable jemalloc as a
memory allocator, it fixed the issue for us.
On Wed, Sep 6, 2023 at 9:07 PM Kenan Kılıçtepe <kk...@gmail.com> wrote:
> Hi,
> Thanks for the answer.
> I will try the documents you have shared.
> But still it would be great if you can take a look at the numbers below
> and give some tips.
>
>
> At the moment RSS is 46.6GB although taskmanager.memory.process.size is
> set to 40000m
>
> GC Statistics:
> 2023-09-06 15:15:03,785 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Memory
> usage stats: [HEAP: 3703/18208/18208 MB, NON HEAP: 154/175/744 MB
> (used/committed/max)]
> 2023-09-06 15:15:03,785 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Direct
> memory stats: Count: 33620, Total Capacity: 1102003811, Used Memory:
> 1102003812
> 2023-09-06 15:15:03,785 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Off-heap
> pool stats: [CodeHeap 'non-nmethods': 1/3/7 MB (used/committed/max)],
> [Metaspace: 87/99/256 MB (used/committed/max)], [CodeHeap 'profiled
> nmethods': 32/35/116 MB (used/committed/max)], [Compressed Class Space:
> 10/14/248 MB (used/committed/max)], [CodeHeap 'non-profiled nmethods':
> 21/22/116 MB (used/committed/max)]
> 2023-09-06 15:15:03,785 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Garbage
> collector stats: [G1 Young Generation, GC TIME (ms): 30452, GC COUNT: 351],
> [G1 Old Generation, GC TIME (ms): 0, GC COUNT: 0]
>
> my configuration:
>
> INFO [] - Final TaskExecutor Memory configuration:
> INFO [] - Total Process Memory: 39.063gb (41943040000 bytes)
> INFO [] - Total Flink Memory: 37.813gb (40600862720 bytes)
> INFO [] - Total JVM Heap Memory: 17.781gb (19092471808 bytes)
> INFO [] - Framework: 128.000mb (134217728 bytes)
> INFO [] - Task: 17.656gb (18958254080 bytes)
> INFO [] - Total Off-heap Memory: 20.031gb (21508390912 bytes)
> INFO [] - Managed: 18.906gb (20300431360 bytes)
> INFO [] - Total JVM Direct Memory: 1.125gb (1207959552 bytes)
> INFO [] - Framework: 128.000mb (134217728 bytes)
> INFO [] - Task: 0 bytes
> INFO [] - Network: 1024.000mb (1073741824 bytes)
> INFO [] - JVM Metaspace: 256.000mb (268435456 bytes)
> INFO [] - JVM Overhead: 1024.000mb (1073741824 bytes)
>
> jcmd output:
>
> ubuntu@dzs-tef-test-01:~/flink/log$ jcmd 1035173 VM.native_memory summary
> 1035173:
>
> Native Memory Tracking:
>
> Total: reserved=23615554KB, committed=21049538KB
> - Java Heap (reserved=18644992KB, committed=18644992KB)
> (mmap: reserved=18644992KB,
> committed=18644992KB)
>
> - Class (reserved=347038KB, committed=106970KB)
> (classes #15959)
> ( instance classes #15140, array classes #819)
> (malloc=5022KB #72815)
> (mmap: reserved=342016KB, committed=101948KB)
> ( Metadata: )
> ( reserved=88064KB, committed=86948KB)
> ( used=79128KB)
> ( free=7820KB)
> ( waste=0KB =0.00%)
> ( Class space:)
> ( reserved=253952KB, committed=15000KB)
> ( used=11278KB)
> ( free=3722KB)
> ( waste=0KB =0.00%)
>
> - Thread (reserved=2404259KB, committed=262791KB)
> (thread #2328)
> (stack: reserved=2393052KB, committed=251584KB)
> (malloc=8481KB #13970)
> (arena=2726KB #4654)
>
> - Code (reserved=252334KB, committed=67866KB)
> (malloc=4650KB #21507)
> (mmap: reserved=247684KB, committed=63216KB)
>
> - GC (reserved=800181KB, committed=800181KB)
> (malloc=74637KB #63221)
> (mmap: reserved=725544KB, committed=725544KB)
>
> - Compiler (reserved=20432KB, committed=20432KB)
> (malloc=20300KB #8557)
> (arena=133KB #5)
>
> - Internal (reserved=21883KB, committed=21871KB)
> (malloc=21839KB #29146)
> (mmap: reserved=44KB, committed=32KB)
>
> - Other (reserved=1082212KB, committed=1082212KB)
> (malloc=1082212KB #34463)
>
> - Symbol (reserved=17581KB, committed=17581KB)
> (malloc=16678KB #187368)
> (arena=903KB #1)
>
> - Native Memory Tracking (reserved=9173KB, committed=9173KB)
> (malloc=1656KB #23012)
> (tracking overhead=7517KB)
>
> - Shared class space (reserved=10904KB, committed=10904KB)
> (mmap: reserved=10904KB, committed=10904KB)
>
> - Arena Chunk (reserved=288KB, committed=288KB)
> (malloc=288KB)
>
> - Logging (reserved=4KB, committed=4KB)
> (malloc=4KB #193)
>
> - Arguments (reserved=22KB, committed=22KB)
> (malloc=22KB #534)
>
> - Module (reserved=2726KB, committed=2726KB)
> (malloc=2726KB #9625)
>
> - Synchronizer (reserved=1515KB, committed=1515KB)
> (malloc=1515KB #12006)
>
> - Safepoint (reserved=8KB, committed=8KB)
> (mmap: reserved=8KB, committed=8KB)
>
>
>
>
>
> On Wed, Sep 6, 2023 at 5:06 PM Biao Geng <bi...@gmail.com> wrote:
>
>> Hi Kenan,
>> If you have confirmed the heap memory is ok(e.g. no Java OOM exception
>> and no frequent GC), then the cause may be off-heap memory over usage,
>> especially when your flink job uses some native library.
>> To diagnose such problem, you can refer to [1][2] for more details about
>> using NMT and jeprof.
>>
>> [1]
>> https://erikwramner.files.wordpress.com/2017/10/native-memory-leaks-in-java.pdf
>> <https://urldefense.com/v3/__https://erikwramner.files.wordpress.com/2017/10/native-memory-leaks-in-java.pdf__;!!BeGeivfSdT4o5A!nj2GI8qB53x1Nb2Xv0reWxXYM2PE21xDqVzP9com6hD9omjrlV_rqMcrfWrT9gDUNMM20CZldcFD-WP_b0EH$>
>> [2] https://www.evanjones.ca/java-native-leak-bug.html
>> <https://urldefense.com/v3/__https://www.evanjones.ca/java-native-leak-bug.html__;!!BeGeivfSdT4o5A!nj2GI8qB53x1Nb2Xv0reWxXYM2PE21xDqVzP9com6hD9omjrlV_rqMcrfWrT9gDUNMM20CZldcFD-a0o4f96$>
>> Best,
>> Biao Geng
>>
>> Kenan Kılıçtepe <kk...@gmail.com> 于2023年9月6日周三 20:32写道:
>>
>>> Hi,
>>>
>>> I have Flink 1.16.2 on a single server with 64GB Ram.
>>>
>>> Although taskmanager.memory.process.size is set to 40000m, I can see
>>> memory usage of the task manager exceed 59GB and OS kills it because of
>>> OOM.
>>> I check the RSS column of application top for memory usage.
>>>
>>> I don`t see any heap memory problem.
>>>
>>> taskmanager.memory.process.size: 40000m
>>> taskmanager.memory.managed.fraction: 0.53
>>> state.backend.rocksdb.memory.managed: true
>>>
>>> Any help is appreciated for analyzing the problem.
>>>
>>> Thanks
>>>
>>>
--
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential in nature and intended solely for the addressee, and are
subject to the terms and conditions of disclosure as further described
here: https://www.scd.swiggy.in/nda <https://www.scd.swiggy.in/nda>. If you
are not the intended recipient or you do not agree to the terms and
conditions of disclosure, please delete this email immediately, and notify
the sender by return email. In the event that you continue to access the
information herein or act upon it in any manner, the terms and conditions
shall be deemed accepted by you.
Re: Memory Leak
Posted by Kenan Kılıçtepe <kk...@gmail.com>.
Hi,
Thanks for the answer.
I will try the documents you have shared.
But still it would be great if you can take a look at the numbers below and
give some tips.
At the moment RSS is 46.6GB although taskmanager.memory.process.size is set
to 40000m
GC Statistics:
2023-09-06 15:15:03,785 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Memory
usage stats: [HEAP: 3703/18208/18208 MB, NON HEAP: 154/175/744 MB
(used/committed/max)]
2023-09-06 15:15:03,785 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Direct
memory stats: Count: 33620, Total Capacity: 1102003811, Used Memory:
1102003812
2023-09-06 15:15:03,785 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Off-heap
pool stats: [CodeHeap 'non-nmethods': 1/3/7 MB (used/committed/max)],
[Metaspace: 87/99/256 MB (used/committed/max)], [CodeHeap 'profiled
nmethods': 32/35/116 MB (used/committed/max)], [Compressed Class Space:
10/14/248 MB (used/committed/max)], [CodeHeap 'non-profiled nmethods':
21/22/116 MB (used/committed/max)]
2023-09-06 15:15:03,785 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Garbage
collector stats: [G1 Young Generation, GC TIME (ms): 30452, GC COUNT: 351],
[G1 Old Generation, GC TIME (ms): 0, GC COUNT: 0]
my configuration:
INFO [] - Final TaskExecutor Memory configuration:
INFO [] - Total Process Memory: 39.063gb (41943040000 bytes)
INFO [] - Total Flink Memory: 37.813gb (40600862720 bytes)
INFO [] - Total JVM Heap Memory: 17.781gb (19092471808 bytes)
INFO [] - Framework: 128.000mb (134217728 bytes)
INFO [] - Task: 17.656gb (18958254080 bytes)
INFO [] - Total Off-heap Memory: 20.031gb (21508390912 bytes)
INFO [] - Managed: 18.906gb (20300431360 bytes)
INFO [] - Total JVM Direct Memory: 1.125gb (1207959552 bytes)
INFO [] - Framework: 128.000mb (134217728 bytes)
INFO [] - Task: 0 bytes
INFO [] - Network: 1024.000mb (1073741824 bytes)
INFO [] - JVM Metaspace: 256.000mb (268435456 bytes)
INFO [] - JVM Overhead: 1024.000mb (1073741824 bytes)
jcmd output:
ubuntu@dzs-tef-test-01:~/flink/log$ jcmd 1035173 VM.native_memory summary
1035173:
Native Memory Tracking:
Total: reserved=23615554KB, committed=21049538KB
- Java Heap (reserved=18644992KB, committed=18644992KB)
(mmap: reserved=18644992KB,
committed=18644992KB)
- Class (reserved=347038KB, committed=106970KB)
(classes #15959)
( instance classes #15140, array classes #819)
(malloc=5022KB #72815)
(mmap: reserved=342016KB, committed=101948KB)
( Metadata: )
( reserved=88064KB, committed=86948KB)
( used=79128KB)
( free=7820KB)
( waste=0KB =0.00%)
( Class space:)
( reserved=253952KB, committed=15000KB)
( used=11278KB)
( free=3722KB)
( waste=0KB =0.00%)
- Thread (reserved=2404259KB, committed=262791KB)
(thread #2328)
(stack: reserved=2393052KB, committed=251584KB)
(malloc=8481KB #13970)
(arena=2726KB #4654)
- Code (reserved=252334KB, committed=67866KB)
(malloc=4650KB #21507)
(mmap: reserved=247684KB, committed=63216KB)
- GC (reserved=800181KB, committed=800181KB)
(malloc=74637KB #63221)
(mmap: reserved=725544KB, committed=725544KB)
- Compiler (reserved=20432KB, committed=20432KB)
(malloc=20300KB #8557)
(arena=133KB #5)
- Internal (reserved=21883KB, committed=21871KB)
(malloc=21839KB #29146)
(mmap: reserved=44KB, committed=32KB)
- Other (reserved=1082212KB, committed=1082212KB)
(malloc=1082212KB #34463)
- Symbol (reserved=17581KB, committed=17581KB)
(malloc=16678KB #187368)
(arena=903KB #1)
- Native Memory Tracking (reserved=9173KB, committed=9173KB)
(malloc=1656KB #23012)
(tracking overhead=7517KB)
- Shared class space (reserved=10904KB, committed=10904KB)
(mmap: reserved=10904KB, committed=10904KB)
- Arena Chunk (reserved=288KB, committed=288KB)
(malloc=288KB)
- Logging (reserved=4KB, committed=4KB)
(malloc=4KB #193)
- Arguments (reserved=22KB, committed=22KB)
(malloc=22KB #534)
- Module (reserved=2726KB, committed=2726KB)
(malloc=2726KB #9625)
- Synchronizer (reserved=1515KB, committed=1515KB)
(malloc=1515KB #12006)
- Safepoint (reserved=8KB, committed=8KB)
(mmap: reserved=8KB, committed=8KB)
On Wed, Sep 6, 2023 at 5:06 PM Biao Geng <bi...@gmail.com> wrote:
> Hi Kenan,
> If you have confirmed the heap memory is ok(e.g. no Java OOM exception and
> no frequent GC), then the cause may be off-heap memory over usage,
> especially when your flink job uses some native library.
> To diagnose such problem, you can refer to [1][2] for more details about
> using NMT and jeprof.
>
> [1]
> https://erikwramner.files.wordpress.com/2017/10/native-memory-leaks-in-java.pdf
> [2] https://www.evanjones.ca/java-native-leak-bug.html
> Best,
> Biao Geng
>
> Kenan Kılıçtepe <kk...@gmail.com> 于2023年9月6日周三 20:32写道:
>
>> Hi,
>>
>> I have Flink 1.16.2 on a single server with 64GB Ram.
>>
>> Although taskmanager.memory.process.size is set to 40000m, I can see
>> memory usage of the task manager exceed 59GB and OS kills it because of
>> OOM.
>> I check the RSS column of application top for memory usage.
>>
>> I don`t see any heap memory problem.
>>
>> taskmanager.memory.process.size: 40000m
>> taskmanager.memory.managed.fraction: 0.53
>> state.backend.rocksdb.memory.managed: true
>>
>> Any help is appreciated for analyzing the problem.
>>
>> Thanks
>>
>>
Re: Memory Leak
Posted by Biao Geng <bi...@gmail.com>.
Hi Kenan,
If you have confirmed the heap memory is ok(e.g. no Java OOM exception and
no frequent GC), then the cause may be off-heap memory over usage,
especially when your flink job uses some native library.
To diagnose such problem, you can refer to [1][2] for more details about
using NMT and jeprof.
[1]
https://erikwramner.files.wordpress.com/2017/10/native-memory-leaks-in-java.pdf
[2] https://www.evanjones.ca/java-native-leak-bug.html
Best,
Biao Geng
Kenan Kılıçtepe <kk...@gmail.com> 于2023年9月6日周三 20:32写道:
> Hi,
>
> I have Flink 1.16.2 on a single server with 64GB Ram.
>
> Although taskmanager.memory.process.size is set to 40000m, I can see
> memory usage of the task manager exceed 59GB and OS kills it because of
> OOM.
> I check the RSS column of application top for memory usage.
>
> I don`t see any heap memory problem.
>
> taskmanager.memory.process.size: 40000m
> taskmanager.memory.managed.fraction: 0.53
> state.backend.rocksdb.memory.managed: true
>
> Any help is appreciated for analyzing the problem.
>
> Thanks
>
>