Απάντηση: Re: Question about Spark cluster memory usage monitoring

[kolokasis <ko...@ics.forth.gr>]

    
Hello,a good tool to manage the cluster usage for each mode is using Grafana which use Prometheus in the background to collect metrics. 
-Iacovos


Εστάλη από τη συσκευή μου Samsung

-------- Αρχικό μήνυμα --------
Από: "Jan Brabec (janbrabe)" <ja...@cisco.com.INVALID> 
Ημερομηνία: 24/9/18  08:57  (GMT+01:00) 
Προς: Muhib Khan <mu...@gmail.com>, "Liu, Jialin" <ji...@illinois.edu> 
Κοιν.: user@spark.apache.org 
Θέμα: Re: Question about Spark cluster memory usage monitoring 



Hello,
 
Maybe Ganglia http://ganglia.info/ might be useful for you? I have only shallow experience with it, but it might be what you are looking for.
 
Best,
Jan
 

From: Muhib Khan <mu...@gmail.com>

Date: Thursday 20 September 2018 at 23:39

To: "Liu, Jialin" <ji...@illinois.edu>

Cc: "user@spark.apache.org" <us...@spark.apache.org>

Subject: Re: Question about Spark cluster memory usage monitoring


 


Hello, 

 


As far as I know, there is no API provided for tracking the execution memory of a Spark Worker node. For tracking the execution memory you will probably need to access the MemoryManager's onHeapExecutionMemoryPool and offHeapExecutionMemoryPool
 objects that track the memory allocated to tasks for execution memory and write to a log the amount for parsing it later or send it to the driver in a periodic interval to collect the memory usage in a central location. Either way, I think tracking execution
 memory may require code changes in Apache Spark. Hope this helps.


 


Regards,


Muhib


Ph.D. Student, FSU


 


 


 


On Thu, Sep 20, 2018 at 4:46 PM Liu, Jialin <ji...@illinois.edu> wrote:


Hi there,



I am currently using Spark cluster to run jobs but I really need to collect the history of actually memory usage(that’s execution memory + storage memory) of the job in the whole cluster. I know we can get the storage memory usage through either Spark UI Executor
 page or SparkContext.getMemoryExecutorStatus() API, but I could not get the real time execution memory usage.


Is there anyway I can collect total memory usage? Thank you so much!



Best,

Jialin Liu

---------------------------------------------------------------------

To unsubscribe e-mail: 
user-unsubscribe@spark.apache.org