Spark Prometheus Metrics for Executors Not Working

[paulp <pp...@outbrain.com.INVALID>]

Hi, 

recently our team has evaluated the prometheusServlet configuration in order
to have Spark master, worker, driver and executor native Prometheus metrics.
We are running a Spark streaming standalone cluster version 3.0.1 on
physical servers.

metrics.properties

*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet
*.sink.prometheusServlet.path=/metrics/prometheus 
master.sink.prometheusServlet.path=/metrics/master/prometheus
applications.sink.prometheusServlet.path=/metrics/applications/prometheus

I'm able to see Spark master, worker and driver metrics but cannot see any
spark executor metrics (curling the relevant Prometheus endpoints just
results in empty result).

For the purposes of executor metrics verification, I have taken one of our
jobs and added the following parameters to the job configuration. These are
the parameters I pass alongside our spark-submit command:

--conf spark.ui.prometheus.enabled=true \
--conf spark.executor.processTreeMetrics.enabled=true

Spark official documentation mentions that Prometheus executor endpoint is
still experimental which may explain this inconsistent behaviour.

Any help will be highly appreciated!



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

RE: Spark Prometheus Metrics for Executors Not Working

[Luca Canali <Lu...@cern.ch>]

The PrometheusServlet adds a servlet within the existing Spark UI to serve metrics data in Prometheus format. 
Similarly to what happens with the MetricsServlet, the Prometheus servlet does not work on executors, as executors do not have a Spark UI end point to which the servlet could attach to.  
Currently Apache Spark does not have an official Prometheus sink for the metrics system.  
Spark has a Graphite sink that can be used with InfluxDB 1.x, which could be useful in this context. 
I also see that a Prometheus sink is available as an external libraries from an open source project.

Best,
Luca

-----Original Message-----
From: paulp <pp...@outbrain.com.INVALID> 
Sent: Monday, May 24, 2021 17:09
To: user@spark.apache.org
Subject: Spark Prometheus Metrics for Executors Not Working

Hi, 

recently our team has evaluated the prometheusServlet configuration in order to have Spark master, worker, driver and executor native Prometheus metrics.
We are running a Spark streaming standalone cluster version 3.0.1 on physical servers.

metrics.properties

*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet
*.sink.prometheusServlet.path=/metrics/prometheus
master.sink.prometheusServlet.path=/metrics/master/prometheus
applications.sink.prometheusServlet.path=/metrics/applications/prometheus

I'm able to see Spark master, worker and driver metrics but cannot see any spark executor metrics (curling the relevant Prometheus endpoints just results in empty result).

For the purposes of executor metrics verification, I have taken one of our jobs and added the following parameters to the job configuration. These are the parameters I pass alongside our spark-submit command:

--conf spark.ui.prometheus.enabled=true \ --conf spark.executor.processTreeMetrics.enabled=true

Spark official documentation mentions that Prometheus executor endpoint is still experimental which may explain this inconsistent behaviour.

Any help will be highly appreciated!



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org