(spark) branch master updated: [SPARK-46886][CORE] Enable `spark.ui.prometheus.enabled` by default

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new c8d116bcfde9 [SPARK-46886][CORE] Enable `spark.ui.prometheus.enabled` by default
c8d116bcfde9 is described below

commit c8d116bcfde938a9e4b33ace1e8257c2798000f4
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Sat Jan 27 15:15:26 2024 -0800

    [SPARK-46886][CORE] Enable `spark.ui.prometheus.enabled` by default
    
    ### What changes were proposed in this pull request?
    
    `spark.ui.prometheus.enabled` has been used since Apache Spark 3.0.0.
    
    - https://github.com/apache/spark/pull/25770
    
    This PR aims to enable `spark.ui.prometheus.enabled` by default like Driver `JSON` API in Apache Spark 4.0.0.
    
    |              |                JSON End Point                    |            Prometheus End Point         |
    | ------- | ------------------------------------ | --------------------------------- |
    | Driver   | /api/v1/applications/{id}/executors/   | /metrics/executors/prometheus/   |
    
    ### Why are the changes needed?
    
    **BEFORE**
    ```
    $ bin/spark-shell
    $ curl -s http://localhost:4040/metrics/executors/prometheus | wc -l
           0
    ```
    
    **AFTER**
    ```
    $ bin/spark-shell
     $ curl -s http://localhost:4040/metrics/executors/prometheus | wc -l
          20
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, this is only a new endpoint.
    
    ### How was this patch tested?
    
    Pass the CIs and do manual test.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #44912 from dongjoon-hyun/SPARK-46886.
    
    Authored-by: Dongjoon Hyun <dh...@apple.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 core/src/main/scala/org/apache/spark/internal/config/UI.scala           | 2 +-
 .../main/scala/org/apache/spark/status/api/v1/PrometheusResource.scala  | 2 +-
 docs/monitoring.md                                                      | 1 -
 3 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/internal/config/UI.scala b/core/src/main/scala/org/apache/spark/internal/config/UI.scala
index 320808d5018c..086c83552732 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/UI.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/UI.scala
@@ -114,7 +114,7 @@ private[spark] object UI {
       "For master/worker/driver metrics, you need to configure `conf/metrics.properties`.")
     .version("3.0.0")
     .booleanConf
-    .createWithDefault(false)
+    .createWithDefault(true)
 
   val UI_X_XSS_PROTECTION = ConfigBuilder("spark.ui.xXssProtection")
     .doc("Value for HTTP X-XSS-Protection response header")
diff --git a/core/src/main/scala/org/apache/spark/status/api/v1/PrometheusResource.scala b/core/src/main/scala/org/apache/spark/status/api/v1/PrometheusResource.scala
index 8cfed4a4bd39..c4e3bdc64ee3 100644
--- a/core/src/main/scala/org/apache/spark/status/api/v1/PrometheusResource.scala
+++ b/core/src/main/scala/org/apache/spark/status/api/v1/PrometheusResource.scala
@@ -31,7 +31,7 @@ import org.apache.spark.ui.SparkUI
  * :: Experimental ::
  * This aims to expose Executor metrics like REST API which is documented in
  *
- *    https://spark.apache.org/docs/3.0.0/monitoring.html#executor-metrics
+ *    https://spark.apache.org/docs/latest/monitoring.html#executor-metrics
  *
  * Note that this is based on ExecutorSummary which is different from ExecutorSource.
  */
diff --git a/docs/monitoring.md b/docs/monitoring.md
index 056543deb094..8d3dbe375b82 100644
--- a/docs/monitoring.md
+++ b/docs/monitoring.md
@@ -821,7 +821,6 @@ A list of the available metrics, with a short description:
 Executor-level metrics are sent from each executor to the driver as part of the Heartbeat to describe the performance metrics of Executor itself like JVM heap memory, GC information.
 Executor metric values and their measured memory peak values per executor are exposed via the REST API in JSON format and in Prometheus format.
 The JSON end point is exposed at: `/applications/[app-id]/executors`, and the Prometheus endpoint at: `/metrics/executors/prometheus`.
-The Prometheus endpoint is conditional to a configuration parameter: `spark.ui.prometheus.enabled=true` (the default is `false`).
 In addition, aggregated per-stage peak values of the executor memory metrics are written to the event log if
 `spark.eventLog.logStageExecutorMetrics` is true.
 Executor memory metrics are also exposed via the Spark metrics system based on the [Dropwizard metrics library](https://metrics.dropwizard.io/4.2.0).


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org