You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/04/02 18:40:26 UTC

[GitHub] [spark] martin-g commented on a change in pull request #36037: [SPARK-38758][CORE][WEB UI] Add heap dump

martin-g commented on a change in pull request #36037:
URL: https://github.com/apache/spark/pull/36037#discussion_r841105686



##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala
##########
@@ -78,6 +78,9 @@ class BlockManagerStorageEndpoint(
     case TriggerThreadDump =>
       context.reply(Utils.getThreadDump())
 
+    case TriggerHeapDump =>
+      context.reply(Utils.getHeapDump)

Review comment:
       ```suggestion
         context.reply(Utils.getHeapDump())
   ```

##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2272,6 +2273,25 @@ private[spark] object Utils extends Logging {
     }.map(threadInfoToThreadStackTrace)
   }
 
+  /** Return a heap dump */
+  def getHeapDump: InputStream = {
+    try {
+      val clazz = classOf[HotSpotDiagnosticMXBean]

Review comment:
       Above you used `Class.forName()` but here you import it and load it unconditionally.
   It would be safer to use `Class.forName()` again.

##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2272,6 +2273,25 @@ private[spark] object Utils extends Logging {
     }.map(threadInfoToThreadStackTrace)
   }
 
+  /** Return a heap dump */
+  def getHeapDump: InputStream = {
+    try {
+      val clazz = classOf[HotSpotDiagnosticMXBean]
+      val server = ManagementFactory.getPlatformMBeanServer
+      val bean = ManagementFactory.newPlatformMXBeanProxy(server
+        , "com.sun.management:type=HotSpotDiagnostic"
+        , clazz)
+      val tmpDir = createTempDir("heap-dump")
+      tmpDir.deleteOnExit()
+      val absolutePath = s"${tmpDir.getAbsolutePath}/${System.nanoTime()}.hprof"

Review comment:
       `System.nanoTime()` does not give you a unique value. There is a high risk of name clashing here.

##########
File path: core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala
##########
@@ -75,6 +75,46 @@ private[v1] class AbstractApplicationResource extends BaseAppResource {
     }
   }
 
+  @GET
+  @Path("executors/{executorId}/heaps")

Review comment:
       ```suggestion
     @Path("executors/{executorId}/heapdump")
   ```
   `/heapdump` or `/heap` looks better to me. 
   There is no one-to-one match here with `/threads`

##########
File path: core/src/main/scala/org/apache/spark/internal/config/UI.scala
##########
@@ -92,6 +92,11 @@ private[spark] object UI {
     .booleanConf
     .createWithDefault(true)
 
+  val UI_HEAP_DUMPS_ENABLED = ConfigBuilder("spark.ui.threadDumpsEnabled")
+    .version("3.4.0")
+    .booleanConf
+    .createWithDefault(true)

Review comment:
       IMO the default should be `false`.
   Heap dump causes "stop the world" in JVM and it might lead to security risks if it is enabled by default.

##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -718,6 +718,27 @@ class SparkContext(config: SparkConf) extends Logging {
     }
   }
 
+  /**
+   * Called by the web UI to obtain executor heap dumps.  This method may be expensive.
+   * Logs an error and returns None if we failed to obtain a heap dump, which could occur due

Review comment:
       ```suggestion
      * Logs an error and returns None if it fails to obtain a heap dump, which could occur due
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org