Re: Measure the memory consumption of a job at runtime

["Wang, Chao" <ch...@wustl.edu>]

Thanks for the tip, Kien. Alternatively, I measured memory footprint by querying /proc/[pid]/status and by pmap.

Best regards,
Chao

From: Kien Truong <du...@gmail.com>
Date: Tuesday, February 13, 2018 at 4:06 AM
To: "user@flink.apache.org" <us...@flink.apache.org>
Subject: Re: Measure the memory consumption of a job at runtime


Hi,

Just want to share a tip, for job with rocksdb backend, you can use LD_LIBRARY_PATH to load jemalloc

containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"



Then use jemalloc heap profiler to track native memory usage by rocksdb.

https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling



In my experience, jemalloc helps a lot at reducing memory fragmentation when using rocksdb state backend.



Best regards,

Kien


On 2/9/2018 11:58 PM, Wang, Chao wrote:
That helps a lot! Thanks, Kedar!
-Chao

From: kedar mhaswade <ke...@gmail.com>
Date: Thursday, February 8, 2018 at 6:37 PM
To: "Wang, Chao" <ch...@wustl.edu>
Cc: "user@flink.apache.org"<ma...@flink.apache.org> <us...@flink.apache.org>
Subject: Re: Measure the memory consumption of a job at runtime

I was also able to attach the JMX (jconsole/VisualVM) console to any container (task manager) or job manager with these options in your flink-conf.yaml:
env.java.opts: -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=50001 -Dcom.sun.management.jmxremote.rmi.port=50001 -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.authenticate=false

That way it shows me how the memory is being allocated and collected. Hope that helps:

[Inline image 1]

On Thu, Feb 8, 2018 at 1:55 PM, Wang, Chao <ch...@wustl.edu>> wrote:
Thanks. I’ve been plugging C/C++ libraries (e.g., libgcrypt) in my Java CEPs. And I’m interested in measuring the memory overhead of JNI. Though I’m not familiar with how Flink and the underlying JVM manage memory..

Chao

From: kedar mhaswade <ke...@gmail.com>>
Date: Thursday, February 8, 2018 at 3:22 PM
To: "Wang, Chao" <ch...@wustl.edu>>
Cc: "user@flink.apache.org<ma...@flink.apache.org>" <us...@flink.apache.org>>
Subject: Re: Measure the memory consumption of a job at runtime

There appears to be some  some support on the Task Manager tab of the Flink dashboard. The one capability that is missing is to take the thread dump (ala Spark UI).
(Is there an issue/feature request for this?)

[Inline image 2]

Is that what you are looking for?


On Thu, Feb 8, 2018 at 12:33 PM, Wang, Chao <ch...@wustl.edu>> wrote:
Hi,

I would like to measure the memory consumption of a job at runtime. I came across some discussion (here: https://stackoverflow.com/questions/35315522/flink-memory-usage ), and it seems that it’s not possible two years ago. Is it possible in the current status, and if yes, how to do it?

Thank you,

Chao