Question about Spark cluster memory usage monitoring

["Liu, Jialin" <ji...@illinois.edu>]

Hi there,

I am currently using Spark cluster to run jobs but I really need to collect the history of actually memory usage(that’s execution memory + storage memory) of the job in the whole cluster. I know we can get the storage memory usage through either Spark UI Executor page or SparkContext.getMemoryExecutorStatus() API, but I could not get the real time execution memory usage. 
Is there anyway I can collect total memory usage? Thank you so much!

Best,
Jialin Liu
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Question about Spark cluster memory usage monitoring

["Jan Brabec (janbrabe)" <ja...@cisco.com.INVALID>]

Hello,

Maybe Ganglia http://ganglia.info/ might be useful for you? I have only shallow experience with it, but it might be what you are looking for.

Best,
Jan

From: Muhib Khan <mu...@gmail.com>
Date: Thursday 20 September 2018 at 23:39
To: "Liu, Jialin" <ji...@illinois.edu>
Cc: "user@spark.apache.org" <us...@spark.apache.org>
Subject: Re: Question about Spark cluster memory usage monitoring

Hello,

As far as I know, there is no API provided for tracking the execution memory of a Spark Worker node. For tracking the execution memory you will probably need to access the MemoryManager's onHeapExecutionMemoryPool and offHeapExecutionMemoryPool objects that track the memory allocated to tasks for execution memory and write to a log the amount for parsing it later or send it to the driver in a periodic interval to collect the memory usage in a central location. Either way, I think tracking execution memory may require code changes in Apache Spark. Hope this helps.

Regards,
Muhib
Ph.D. Student, FSU



On Thu, Sep 20, 2018 at 4:46 PM Liu, Jialin <ji...@illinois.edu>> wrote:
Hi there,

I am currently using Spark cluster to run jobs but I really need to collect the history of actually memory usage(that’s execution memory + storage memory) of the job in the whole cluster. I know we can get the storage memory usage through either Spark UI Executor page or SparkContext.getMemoryExecutorStatus() API, but I could not get the real time execution memory usage.
Is there anyway I can collect total memory usage? Thank you so much!

Best,
Jialin Liu
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>

Re: Question about Spark cluster memory usage monitoring

[Muhib Khan <mu...@gmail.com>]

Hello,

As far as I know, there is no API provided for tracking the execution
memory of a Spark Worker node. For tracking the execution memory you will
probably need to access the MemoryManager's onHeapExecutionMemoryPool
and offHeapExecutionMemoryPool objects that track the memory allocated to
tasks for execution memory and write to a log the amount for parsing it
later or send it to the driver in a periodic interval to collect the memory
usage in a central location. Either way, I think tracking execution memory
may require code changes in Apache Spark. Hope this helps.

Regards,
Muhib
Ph.D. Student, FSU



On Thu, Sep 20, 2018 at 4:46 PM Liu, Jialin <ji...@illinois.edu> wrote:

> Hi there,
>
> I am currently using Spark cluster to run jobs but I really need to
> collect the history of actually memory usage(that’s execution memory +
> storage memory) of the job in the whole cluster. I know we can get the
> storage memory usage through either Spark UI Executor page or
> SparkContext.getMemoryExecutorStatus() API, but I could not get the real
> time execution memory usage.
> Is there anyway I can collect total memory usage? Thank you so much!
>
> Best,
> Jialin Liu
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>