CPU usage from Event log

[Prasad Bhalerao <pr...@gmail.com>]

Hi,

I am trying to calculate CPU utilization of an Executor(JVM level CPU
usage) using Event log. Can someone please help me with this?

1) Which column/properties to select
2) the correct formula to derive cpu usage

Has anyone done anything similar to this?

We have many pipelines and those are using very huge EMR clusters. I am
trying to find out the cpu utilization and memory utilization of the nodes.
This will help me find out if the clusters are under utilized and reduce
the nodes,

Is there a better way to get these stats without changing the code?


Thanks,
Prasad

Re: CPU usage from Event log

[Artemis User <ar...@dtechspace.com>]

I am not sure what column/properties you are referring to.  But the 
event log in Spark deals with application level "events', not JVM-level 
metrics.  To retrieve the JVM metrics, you need to use the REST API 
provided in Spark.  Please see 
https://spark.apache.org/docs/latest/monitoring.html for details..

On 3/9/22 10:21 AM, Prasad Bhalerao wrote:
> Hi,
>
> I am trying to calculate CPU utilization of an Executor(JVM level CPU 
> usage) using Event log. Can someone please help me with this?
>
> 1) Which column/properties to select
> 2) the correct formula to derive cpu usage
>
> Has anyone done anything similar to this?
>
> We have many pipelines and those are using very huge EMR clusters. I 
> am trying to find out the cpu utilization and memory utilization of 
> the nodes. This will help me find out if the clusters are under 
> utilized and reduce the nodes,
>
> Is there a better way to get these stats without changing the code?
>
>
> Thanks,
> Prasad


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org