[jira] [Created] (SPARK-33194) Spark 3.0 streaming metrics are not in Prometheus

["Jeremie Piotte (Jira)" <ji...@apache.org>]

Jeremie Piotte created SPARK-33194:
--------------------------------------

             Summary: Spark 3.0 streaming metrics are not in Prometheus
                 Key: SPARK-33194
                 URL: https://issues.apache.org/jira/browse/SPARK-33194
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 3.0.0
            Reporter: Jeremie Piotte


I'm running a Spark 3.0 application (Spark Structured Streaming) on Kubernetes and I'm trying to use the new native Prometheus metric sink. I'm able to make it work and get all the metrics described [here|https://spark.apache.org/docs/latest/monitoring.html#executor-metrics].

However, the metrics I really need are the ones provided upon enabling the following config: spark.sql.streaming.metricsEnabled, as proposed in [this Spark Summit presentation|https://speakerdeck.com/dongjoon/native-support-of-prometheus-monitoring-in-apache-spark-3?slide=32]. Now, even with that config set to "true", I can't see any streaming metrics under {{/metrics/executors/prometheus}} as advertised. One thing to note is that I can see them under {{metrics/json}}, therefore, we know that the configuration was properly applied.

Why aren't streaming metrics sent to the Prometheus sink? Do I need to add some additional configuration? Is that not supported yet?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org