[GitHub] squito commented on a change in pull request #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics

[GitBox <gi...@apache.org>]

squito commented on a change in pull request #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
URL: https://github.com/apache/spark/pull/22874#discussion_r243059130
 
 

 ##########
 File path: core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala
 ##########
 @@ -99,6 +100,61 @@ case object ProcessTreeMetrics extends ExecutorMetricType {
   }
 }
 
+case object GarbageCollectionMetrics extends ExecutorMetricType {
+  import GC_TYPE._
+  override val names = Seq(
+    "MinorGCCount",
+    "MinorGCTime",
+    "MajorGCCount",
+    "MajorGCTime"
+  )
+  override private[spark] def getMetricValues(memoryManager: MemoryManager): Array[Long] = {
+    var allMetrics = GCMetrics(0, 0, 0, 0)
+    ManagementFactory.getGarbageCollectorMXBeans.asScala.foreach { mxBean =>
+      val metrics = mxBean.getName match {
+        case `copy` | `psScavenge` | `parNew` | `g1Young` =>
 
 Review comment:
   this is a good point.  I don't think the GarbageCollectorMXBeans (or anything else in the JVM) define a mapping to major & minor GC.  So we could either:
   
   a) manually update this mapping ourselves, as new GC methods come around.  also easy enough to make this configurable at runtime, so we don't have to cover all cases, but I think we'd still try to do as much as possible
   
   b) don't map them into major / minor at all, and just report the metrics prefixed with the collector name.  This is a little tricky, as then the number of metrics is determined at runtime, based on what collectors are active in the given JVM.  But that should be manageable (it would wreck havoc if you had inconsistent JVMs across executors, though).
   
   do you have thoughts on what might be more useful to the end user?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org