[spark] branch branch-2.4 updated: [SPARK-25277][YARN] YARN applicationMaster metrics should not register static metrics

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-2.4 by this push:
     new 1c57da3  [SPARK-25277][YARN] YARN applicationMaster metrics should not register static metrics
1c57da3 is described below

commit 1c57da3683f4d3dc5b58892f6f1f81190f12c9b5
Author: Luca Canali <lu...@cern.ch>
AuthorDate: Wed Dec 12 16:18:22 2018 -0800

    [SPARK-25277][YARN] YARN applicationMaster metrics should not register static metrics
    
    ## What changes were proposed in this pull request?
    
    YARN applicationMaster metrics registration introduced in SPARK-24594 causes further registration of static metrics (Codegenerator and HiveExternalCatalog) and of JVM metrics, which I believe do not belong in this context.
    This looks like an unintended side effect of using the start method of [[MetricsSystem]].
    A possible solution proposed here, is to introduce startNoRegisterSources to avoid these additional registrations of static sources and of JVM sources in the case of YARN applicationMaster metrics (this could be useful for other metrics that may be added in the future).
    
    ## How was this patch tested?
    
    Manually tested on a YARN cluster,
    
    Closes #22279 from LucaCanali/YarnMetricsRemoveExtraSourceRegistration.
    
    Lead-authored-by: Luca Canali <lu...@cern.ch>
    Co-authored-by: LucaCanali <lu...@cern.ch>
    Signed-off-by: Marcelo Vanzin <va...@cloudera.com>
---
 core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala  | 8 +++++---
 .../scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala    | 3 ++-
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
index 657d75c..4a97e28 100644
--- a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
+++ b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
@@ -94,11 +94,13 @@ private[spark] class MetricsSystem private (
 
   metricsConfig.initialize()
 
-  def start() {
+  def start(registerStaticSources: Boolean = true) {
     require(!running, "Attempting to start a MetricsSystem that is already running")
     running = true
-    StaticSources.allSources.foreach(registerSource)
-    registerSources()
+    if (registerStaticSources) {
+      StaticSources.allSources.foreach(registerSource)
+      registerSources()
+    }
     registerSinks()
     sinks.foreach(_.start)
   }
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
index 5ff826a..f9a009f 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
@@ -451,7 +451,8 @@ private[spark] class ApplicationMaster(args: ApplicationMasterArguments) extends
     val ms = MetricsSystem.createMetricsSystem("applicationMaster", sparkConf, securityMgr)
     val prefix = _sparkConf.get(YARN_METRICS_NAMESPACE).getOrElse(appId)
     ms.registerSource(new ApplicationMasterSource(prefix, allocator))
-    ms.start()
+    // do not register static sources in this case as per SPARK-25277
+    ms.start(false)
     metricsSystem = Some(ms)
     reporterThread = launchReporterThread()
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org