You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mati (Jira)" <ji...@apache.org> on 2021/05/24 13:31:00 UTC

[jira] [Created] (SPARK-35502) Spark Executor metrics are not produced/showed

Mati created SPARK-35502:
----------------------------

             Summary: Spark Executor metrics are not produced/showed
                 Key: SPARK-35502
                 URL: https://issues.apache.org/jira/browse/SPARK-35502
             Project: Spark
          Issue Type: Bug
          Components: Spark Submit
    Affects Versions: 3.0.1
            Reporter: Mati


Recently we have enabled prometheusServlet configuration in order to have spark master, worker, driver and executor metrics.

We can see and using spark master, worker and driver executors but can't see spark executor metrics.

We are running spark streaming standalone cluster in version 3.0.1 over physical servers.

 
We have taken one of our jobs and added the following parameters to the job configuration, but couldn't see executer metrics by curling both driver and executor workers of this job:
 
These are the parameters:
--conf spark.ui.prometheus.enabled=true \
--conf spark.executor.processTreeMetrics.enabled=true
 
Curl commands:
[00764f](root@sparktest-40005-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
[00764f](root@sparktest-40005-prod-chidc2:~)#
Driver of this job - sparktest-40004:
[e35005](root@sparktest-40004-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
[e35005](root@sparktest-40004-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
 
Out UI port is on 4050
 
 I understood that the executor Prometheus endpoint is still experimental which may explain the inconsistent behaviour we see but is there a plan to fix it?
 
Are there any known issues regarding this?
 
 
 
 
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org