You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "parthchandra (via GitHub)" <gi...@apache.org> on 2023/11/26 23:59:03 UTC
[PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
parthchandra opened a new pull request, #44021:
URL: https://github.com/apache/spark/pull/44021
### What changes were proposed in this pull request?
This adds support for the async profiler to Spark
### Why are the changes needed?
Profiling of JVM applications on a cluster is cumbersome and it can be complicated to save the output of the profiler especially if the cluster is on K8s where the executor pods are removed and any files saved to the local file system become inaccessible. This feature makes it simple to turn profiling on/off, includes the jar/binaries needed for profiling, and makes it simple to save output to an HDFS location.
### Does this PR introduce _any_ user-facing change?
This PR introduces three new configuration parameters. These are described in the documentation.
### How was this patch tested?
Tested manually on EKS
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424407928
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+
+
+/**
+ * Spark plugin to do code profiling of executors
Review Comment:
`code` -> `JVM`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424396432
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,109 @@
+# Spark JVM Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -DskipTests -Pjvm-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/v2.10/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://access.redhat.com/documentation/es-es/red_hat_build_of_openjdk/17/html/using_jdk_flight_recorder_with_red_hat_build_of_openjdk/openjdk-flight-recorded-overview) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor :
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.profiler.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td><code>false</code></td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.dfsDir</code></td>
+ <td>(none)</td>
+ <td>
+ An HDFS compatible path to which the profiler's output files are copied. The output files will be written as <i>dfsDir/application_id/profile-appname-exec-executor_id.jfr</i> <br/>
+ If no <i>dfsDir</i> is specified then the files are not copied over. Users should ensure there is sufficient disk space available otherwise it may lead to corrupt jfr files.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.localDir</code></td>
+ <td><code>.</code> i.e. the executor's working dir</td>
+ <td>
+ The local directory in the executor container to write the jfr files to. If not specified the file will be written to the executor's working directory. Users should ensure there is sufficient disk space available on the system as running out of space may result in corrupt jfr file and even cause jobs to fail on systems like K8s.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.fraction</code></td>
+ <td>0.10</td>
+ <td>
+ The fraction of executors on which to enable code profiling. The executors to be profiled are picked at random.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.writeInterval</code></td>
+ <td>30</td>
+ <td>
+ Time interval, in seconds, after which the profiler output will be synced to dfs.
+ </td>
+ <td>4.0.0</td>
+</tr>
+</table>
+
+### Kubernetes
+On Kubernetes, spark will try to shut down the executor pods while the profiler files are still being saved. To prevent this set
+```
+ spark.kubernetes.executor.deleteOnTermination=false
+```
+
+### Example
+```
+./bin/spark-submit \
+ --class <main-class> \
+ --master <master-url> \
+ --deploy-mode <deploy-mode> \
+ -c spark.executor.extraJavaOptions="-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer" \
+ -c spark.plugins=org.apache.spark.executor.profiler.ExecutorProfilerPlugin \
+ -c spark.executor.profiling.enabled=true \
+ -c spark.executor.profiling.outputDir=s3a://my-bucket/spark/profiles/ \
Review Comment:
`spark.executor.profiling.dfsDir`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408066036
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
Review Comment:
Let's remove the trailing space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408094677
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.{EXECUTOR_CODE_PROFILING_ENABLED, EXECUTOR_CODE_PROFILING_FRACTION}
+
+
+/**
+ * Spark plugin to do code profiling of executors
+ *
+ */
+class ExecutorProfilerPlugin extends SparkPlugin {
+ override def driverPlugin(): DriverPlugin = null
+
+ // No-op
+ override def executorPlugin(): ExecutorPlugin = new CodeProfilerExecutorPlugin
+}
+
+class CodeProfilerExecutorPlugin extends ExecutorPlugin with Logging {
+
+ private var sparkConf: SparkConf = _
+ private var pluginCtx: PluginContext = _
+ private var profiler: ExecutorCodeProfiler = _
+ private var codeProfilingEnabled: Boolean = _
+ private var codeProfilingFraction: Double = _
+ private val rand: Random = new Random(System.currentTimeMillis())
+
+ override def init(ctx: PluginContext, extraConf: JMap[String, String]): Unit = {
+ pluginCtx = ctx
+ sparkConf = ctx.conf()
+ codeProfilingEnabled = sparkConf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ codeProfilingFraction = sparkConf.get(EXECUTOR_CODE_PROFILING_FRACTION)
+
+ if (codeProfilingEnabled) {
+ if (rand.nextInt(100) * 0.01 < codeProfilingFraction) {
+ logInfo(s"Executor id ${pluginCtx.executorID()} selected for code profiling")
+ profiler = new ExecutorCodeProfiler(sparkConf, pluginCtx.executorID())
+ profiler.start()
+ }
+ }
+ Map.empty[String, String].asJava
+ }
+
+ override def shutdown(): Unit = {
+
Review Comment:
nit. Remove this line.
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.{EXECUTOR_CODE_PROFILING_ENABLED, EXECUTOR_CODE_PROFILING_FRACTION}
+
+
+/**
+ * Spark plugin to do code profiling of executors
+ *
+ */
+class ExecutorProfilerPlugin extends SparkPlugin {
+ override def driverPlugin(): DriverPlugin = null
+
+ // No-op
+ override def executorPlugin(): ExecutorPlugin = new CodeProfilerExecutorPlugin
+}
+
+class CodeProfilerExecutorPlugin extends ExecutorPlugin with Logging {
+
+ private var sparkConf: SparkConf = _
+ private var pluginCtx: PluginContext = _
+ private var profiler: ExecutorCodeProfiler = _
+ private var codeProfilingEnabled: Boolean = _
+ private var codeProfilingFraction: Double = _
+ private val rand: Random = new Random(System.currentTimeMillis())
+
+ override def init(ctx: PluginContext, extraConf: JMap[String, String]): Unit = {
+ pluginCtx = ctx
+ sparkConf = ctx.conf()
+ codeProfilingEnabled = sparkConf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ codeProfilingFraction = sparkConf.get(EXECUTOR_CODE_PROFILING_FRACTION)
+
+ if (codeProfilingEnabled) {
+ if (rand.nextInt(100) * 0.01 < codeProfilingFraction) {
+ logInfo(s"Executor id ${pluginCtx.executorID()} selected for code profiling")
+ profiler = new ExecutorCodeProfiler(sparkConf, pluginCtx.executorID())
+ profiler.start()
+ }
+ }
+ Map.empty[String, String].asJava
+ }
+
+ override def shutdown(): Unit = {
+
+ logInfo("Executor code profiler shutting down")
+ if (profiler != null) {
+ profiler.stop()
+ }
+ }
+
+
Review Comment:
nit. Remove the above two lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1830344444
Oh, one more thing, I removed `[CONNECT]` from the title because `[CONNECT]` is for `Spark Connect`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1863253175
> A couple of queries:
>
> a) IIRC async profiler does not have a way to associate the java thread with the underlying linux thread - and so from a future evolution point of view, we cannot corelate this with a task's stack traces - is this still the case ? If yes, given work being done as part of [SPARK-45151](https://issues.apache.org/jira/browse/SPARK-45151), is there a way to make this work with whatever @yaooqinn added support for ? ([honest-profiler](https://github.com/jvm-profiling-tools/honest-profiler) has support for this, but unfortunately it does not have maven artifacts yet :-( )
I believe async_profiler uses the JVM's internal `AsyncGetCallTrace` method (https://github.com/async-profiler/async-profiler?tab=readme-ov-file#cpu-profiling) to walk the stack for a thread including native calls made by the JVM. AFAIK, honest-profiler does the same. There is also the `AsyncGetStackTrace` API in the works https ://openjdk.org/jeps/435 (not sure if this is available yet). At some point we can investigate how to leverage the new API to add native calls to the work done in SPARK-45151.
> b) This PR is creating one file per executor on hdfs - which would end up being very expensive as number of executors per application and number of applications increase. How are we looking to minimize this cost ?
There is a configuration `spark.executor.profiling.fraction` to limit the number of executors to be profiled (default is 0.1). One can also specify profiler options to limit the sampling frequecy `spark.executor.profiling.options=...,interval=10ms,...` which can limit the size of the jfr file created (useful for very long running jobs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #44021: [SPARK-46094] Support Executor JVM Profiling
URL: https://github.com/apache/spark/pull/44021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1450185201
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
Review Comment:
nit. Let's merge line 71 into 70.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Add support for code profiling executors [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1410737953
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
Review Comment:
In Spark, it is more customary to write as `-Pcode-profiler`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408072450
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
+ <td></td>
+ <td>
+ An hdfs compatible path to which the profiler's output files are copied. The output files will be written as <i>outputDir/application_id/profile-appname-exec-executor_id.jfr</i> <br/>
Review Comment:
`hdfs` -> `HDFS`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1864980397
> There is a difference between native thread id's and java thread ids. Given the async profiler output, can we map it to the corresponding task (given task's java thread id) ? My understanding is currently no - but if I am missing something, do let me know.
Yes we can map the stack traces to the java thread. Here's how it looks (this is in intellij's profiler window)
<img width="2046" alt="Screenshot 2023-12-20 at 9 46 49 AM" src="https://github.com/apache/spark/assets/6529136/e87d4b9d-4e00-4a89-aa50-c107399359bc">
> Assuming no, this means the stack traces generated are for all threads in the executor jvm - and so does not allow us to get stack traces and/or flamegraphs for a particular task, tasks of a stage, etc.
We can get individual threads and even filter to profile a single thread. This PR specifically profiles every thread in the executor.
> If yes, this would be very useful - and will allow for future evolution as part of [SPARK-44893](https://issues.apache.org/jira/browse/SPARK-44893) [1].
Ah, this JIRA makes it clearer. We can leverage the async-profiler to provide the features not yet implemented in [SPARK-45209](https://issues.apache.org/jira/browse/SPARK-45209). The current implementation uses a simple snapshot of the task stack traces which can be enhanced by using the async-profiler to get accurate profiling.
> I am not seeing a lot of value in including this into Apache Spark itself - plugin api is public, and users can leverage it to do precisely what the PR is proposing. On other hand, if the PR is integrating well with [SPARK-44893](https://issues.apache.org/jira/browse/SPARK-44893) [1] - and/or there is a path to leveraging it in that work, it would be more useful.
I think we can certainly leverage this work. This PR by itself does not have the APIs needed to enhance SPARK-45209. It would probably need to be a separate PR because it may need changes to the UI implementation. We can either get a flamegraph (covering a period of time for a task) or collapsed call traces from which a flamegraph can be produced and the choice will affect the UI.
> I am not -1 on this @dongjoon-hyun , but I am not seeing a lot of value in it: will let you make the call (also because I am on vacation, dont have my desktop handy to investigate in detail :) ).
🙏🏾
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424398819
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.dfsDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
Review Comment:
`file-system path` -> `file-system path`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424403116
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.dfsDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
+ .version("4.0.0")
+ .stringConf
+ .createOptional
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_LOCAL_DIR =
+ ConfigBuilder("spark.executor.profiling.localDir")
+ .doc("Local file system path on executor where profiler output is saved. Defaults to the " +
+ "working directory of the executor process.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(".")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OPTIONS =
+ ConfigBuilder("spark.executor.profiling.options")
+ .doc("Options to pass on to the async profiler.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault("event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_FRACTION =
+ ConfigBuilder("spark.executor.profiling.fraction")
+ .doc("Fraction of executors to profile")
+ .version("4.0.0")
+ .doubleConf
+ .checkValue(v => v >= 0.0 && v < 1.0,
+ "Fraction of executors to profile must be in [0,1)")
Review Comment:
Why do we exclude 100%?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1863766109
> If async profiler does not allow us to map the native thread to its java thread (please validate this) - we cannot map stack traces to the corresponding task threads - and so limits usability of this integration in spark.
`AsyncGetCallTrace` is used precisely to map calls in the native thread to calls in the java thread.
Not sure exactly what you are looking for here. Are you looking to profile individual tasks? It certainly can be done, but would require some changes similar to SPARK-45151 and some additional work if you want the profile available thru the UI. Or are you looking to enhance SPARK-45151 and get a stack trace that includes native calls? This is a little harder via async_profiler since there is no API to get a snapshot.
Note that getting a profile needs to be collected over a period of time and so is different from getting a snapshot as SPARK-45151 is doing.
> Simply dumping per executor flamegraphs or stack traces has limited utility (and can be done today).
I would suggest that this PR makes it trivially simple to profile with no setup required. On K8s, with ephemeral storage, it is not a simple task to dump a profile to disk and get it off the pod before the pod is destroyed (it was in fact the original motivation behind doing this).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408063248
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
Review Comment:
Just a question. Why not Windows?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1827003193
> how do you use it? would be great if it contains the example, how to run, etc.
There's a README - connector/profiler/README.md. I can add more details if you think this is not enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1450200494
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
+ } catch {
+ case e@(_: IllegalArgumentException | _: IllegalStateException | _: IOException) =>
+ logError("JVM profiling aborted. Exception occurred in profiler native code: ", e)
+ case e: Exception => logWarning("Executor JVM profiling aborted due to exception: ", e)
+ }
+ }
+ }
+
+ /** Stops the profiling and saves output to dfs location. */
+ def stop(): Unit = {
+ if (running) {
+ profiler.foreach(p => {
+ p.execute(stopcmd)
+ logInfo("JVM profiler stopped")
+ running = false
+ finishWriting()
+ })
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerDfsDir.isDefined) {
+ val applicationId = try {
+ conf.getAppId
+ } catch {
+ case _: NoSuchElementException => "local-" + System.currentTimeMillis
+ }
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name").replace(" ", "-")
+ val profilerOutputDirname = profilerDfsDir.get
+
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk(false)
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
+ writing = true
+ } catch {
+ case e: Exception =>
+ logError("Failed to start JVM profiler", e)
+ if (threadpool != null) {
+ threadpool.shutdownNow()
+ }
+ if (inputStream != null) {
+ inputStream.close()
+ }
+ if (outputStream != null) {
+ outputStream.close()
+ }
+ }
+ }
+ }
+
+ private def writeChunk(lastChunk: Boolean): Unit = {
+ if (!writing) {
+ return
+ }
+ try {
+ // stop (pause) the profiler, dump the results and then resume. This is not ideal as we miss
+ // the events while the file is being dumped, but that is the only way to make sure that
+ // the chunk of data we are copying to dfs is in a consistent state.
+ profiler.get.execute(stopcmd)
+ profiler.get.execute(dumpcmd)
+ var remaining = inputStream.available()
+ if (!lastChunk) {
+ profiler.get.execute(resumecmd)
+ }
+ while (remaining > 0) {
+ val read = inputStream.read(dataBuffer, 0, math.min(remaining, UPLOAD_SIZE))
+ outputStream.write(dataBuffer, 0, read)
+ remaining -= read
+ }
+ } catch {
+ case e: IOException => logError("Exception occurred while writing some profiler output: ", e)
+ case e@(_: IllegalArgumentException | _: IllegalStateException) =>
+ logError("Some profiler output not written." +
+ " Exception occurred in profiler native code: ", e)
+ case e: Exception => logError("Some profiler output not written. Unexpected exception: ", e)
+ }
+ }
+
+ private def finishWriting(): Unit = {
+ if (profilerDfsDir.isDefined && writing) {
+ try {
+ // shutdown background writer
+ threadpool.shutdown()
+ threadpool.awaitTermination(30, TimeUnit.SECONDS)
+ // flush remaining data
+ writeChunk(true)
+ inputStream.close()
+ outputStream.close()
+ } catch {
+ case _: InterruptedException => Thread.currentThread().interrupt()
+ case e: IOException =>
Review Comment:
Where does this come from when `writeChunk` swallows all exceptions with `case e: Exception`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424402439
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
Review Comment:
Shall we rename all `EXECUTOR_CODE_PROFILING_*` to `EXECUTOR_PROFILING_*`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1914347625
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.util.concurrent.TimeUnit
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_PROFILING_DFS_DIR =
+ ConfigBuilder("spark.executor.profiling.dfsDir")
Review Comment:
Well, @pan3793 , could you use a new JIRA for your new goal, please?
Although I understand why you adds a comment here, it doesn't look like a good practice to me. Why do we add a comment about driver-related suggestion to the irrelevant executor-related PR like `Spark Executor JVM Profiling`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1450862604
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
Review Comment:
Done
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
+ } catch {
+ case e@(_: IllegalArgumentException | _: IllegalStateException | _: IOException) =>
Review Comment:
Done
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
+ } catch {
+ case e@(_: IllegalArgumentException | _: IllegalStateException | _: IOException) =>
+ logError("JVM profiling aborted. Exception occurred in profiler native code: ", e)
+ case e: Exception => logWarning("Executor JVM profiling aborted due to exception: ", e)
+ }
+ }
+ }
+
+ /** Stops the profiling and saves output to dfs location. */
+ def stop(): Unit = {
+ if (running) {
+ profiler.foreach(p => {
+ p.execute(stopcmd)
+ logInfo("JVM profiler stopped")
+ running = false
+ finishWriting()
+ })
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerDfsDir.isDefined) {
+ val applicationId = try {
+ conf.getAppId
+ } catch {
+ case _: NoSuchElementException => "local-" + System.currentTimeMillis
+ }
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name").replace(" ", "-")
+ val profilerOutputDirname = profilerDfsDir.get
+
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk(false)
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
Review Comment:
Done. Hopefully this is better now.
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
+ } catch {
+ case e@(_: IllegalArgumentException | _: IllegalStateException | _: IOException) =>
+ logError("JVM profiling aborted. Exception occurred in profiler native code: ", e)
+ case e: Exception => logWarning("Executor JVM profiling aborted due to exception: ", e)
+ }
+ }
+ }
+
+ /** Stops the profiling and saves output to dfs location. */
+ def stop(): Unit = {
+ if (running) {
+ profiler.foreach(p => {
+ p.execute(stopcmd)
+ logInfo("JVM profiler stopped")
+ running = false
+ finishWriting()
+ })
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerDfsDir.isDefined) {
+ val applicationId = try {
+ conf.getAppId
+ } catch {
+ case _: NoSuchElementException => "local-" + System.currentTimeMillis
+ }
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name").replace(" ", "-")
+ val profilerOutputDirname = profilerDfsDir.get
+
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk(false)
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
+ writing = true
+ } catch {
+ case e: Exception =>
+ logError("Failed to start JVM profiler", e)
+ if (threadpool != null) {
+ threadpool.shutdownNow()
+ }
+ if (inputStream != null) {
+ inputStream.close()
+ }
+ if (outputStream != null) {
+ outputStream.close()
+ }
+ }
+ }
+ }
+
+ private def writeChunk(lastChunk: Boolean): Unit = {
+ if (!writing) {
+ return
+ }
+ try {
+ // stop (pause) the profiler, dump the results and then resume. This is not ideal as we miss
+ // the events while the file is being dumped, but that is the only way to make sure that
+ // the chunk of data we are copying to dfs is in a consistent state.
+ profiler.get.execute(stopcmd)
+ profiler.get.execute(dumpcmd)
+ var remaining = inputStream.available()
+ if (!lastChunk) {
+ profiler.get.execute(resumecmd)
+ }
+ while (remaining > 0) {
+ val read = inputStream.read(dataBuffer, 0, math.min(remaining, UPLOAD_SIZE))
+ outputStream.write(dataBuffer, 0, read)
+ remaining -= read
+ }
+ } catch {
+ case e: IOException => logError("Exception occurred while writing some profiler output: ", e)
+ case e@(_: IllegalArgumentException | _: IllegalStateException) =>
+ logError("Some profiler output not written." +
+ " Exception occurred in profiler native code: ", e)
+ case e: Exception => logError("Some profiler output not written. Unexpected exception: ", e)
Review Comment:
If we get a failure, we can still keep writing because only the portion (chunk) of the data being written may be lost. The output file would still be valid.
`finishWriting` is eventually called to shutdown the thread and the streams cleanly.
We could potentially stop profiling if we get any of these exceptions, but I feel that it is perhaps too drastic.
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
+ } catch {
+ case e@(_: IllegalArgumentException | _: IllegalStateException | _: IOException) =>
+ logError("JVM profiling aborted. Exception occurred in profiler native code: ", e)
+ case e: Exception => logWarning("Executor JVM profiling aborted due to exception: ", e)
+ }
+ }
+ }
+
+ /** Stops the profiling and saves output to dfs location. */
+ def stop(): Unit = {
+ if (running) {
+ profiler.foreach(p => {
+ p.execute(stopcmd)
+ logInfo("JVM profiler stopped")
+ running = false
+ finishWriting()
+ })
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerDfsDir.isDefined) {
+ val applicationId = try {
+ conf.getAppId
+ } catch {
+ case _: NoSuchElementException => "local-" + System.currentTimeMillis
+ }
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name").replace(" ", "-")
+ val profilerOutputDirname = profilerDfsDir.get
+
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk(false)
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
+ writing = true
+ } catch {
+ case e: Exception =>
+ logError("Failed to start JVM profiler", e)
+ if (threadpool != null) {
+ threadpool.shutdownNow()
+ }
+ if (inputStream != null) {
+ inputStream.close()
+ }
+ if (outputStream != null) {
+ outputStream.close()
+ }
+ }
+ }
+ }
+
+ private def writeChunk(lastChunk: Boolean): Unit = {
+ if (!writing) {
+ return
+ }
+ try {
+ // stop (pause) the profiler, dump the results and then resume. This is not ideal as we miss
+ // the events while the file is being dumped, but that is the only way to make sure that
+ // the chunk of data we are copying to dfs is in a consistent state.
+ profiler.get.execute(stopcmd)
+ profiler.get.execute(dumpcmd)
+ var remaining = inputStream.available()
+ if (!lastChunk) {
+ profiler.get.execute(resumecmd)
+ }
+ while (remaining > 0) {
+ val read = inputStream.read(dataBuffer, 0, math.min(remaining, UPLOAD_SIZE))
+ outputStream.write(dataBuffer, 0, read)
+ remaining -= read
+ }
+ } catch {
+ case e: IOException => logError("Exception occurred while writing some profiler output: ", e)
+ case e@(_: IllegalArgumentException | _: IllegalStateException) =>
Review Comment:
Done
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
+ } catch {
+ case e@(_: IllegalArgumentException | _: IllegalStateException | _: IOException) =>
+ logError("JVM profiling aborted. Exception occurred in profiler native code: ", e)
+ case e: Exception => logWarning("Executor JVM profiling aborted due to exception: ", e)
+ }
+ }
+ }
+
+ /** Stops the profiling and saves output to dfs location. */
+ def stop(): Unit = {
+ if (running) {
+ profiler.foreach(p => {
+ p.execute(stopcmd)
+ logInfo("JVM profiler stopped")
+ running = false
+ finishWriting()
+ })
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerDfsDir.isDefined) {
+ val applicationId = try {
+ conf.getAppId
+ } catch {
+ case _: NoSuchElementException => "local-" + System.currentTimeMillis
+ }
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name").replace(" ", "-")
+ val profilerOutputDirname = profilerDfsDir.get
+
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk(false)
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
+ writing = true
+ } catch {
+ case e: Exception =>
+ logError("Failed to start JVM profiler", e)
+ if (threadpool != null) {
+ threadpool.shutdownNow()
+ }
+ if (inputStream != null) {
+ inputStream.close()
+ }
+ if (outputStream != null) {
+ outputStream.close()
+ }
+ }
+ }
+ }
+
+ private def writeChunk(lastChunk: Boolean): Unit = {
+ if (!writing) {
+ return
+ }
+ try {
+ // stop (pause) the profiler, dump the results and then resume. This is not ideal as we miss
+ // the events while the file is being dumped, but that is the only way to make sure that
+ // the chunk of data we are copying to dfs is in a consistent state.
+ profiler.get.execute(stopcmd)
+ profiler.get.execute(dumpcmd)
+ var remaining = inputStream.available()
+ if (!lastChunk) {
+ profiler.get.execute(resumecmd)
+ }
+ while (remaining > 0) {
+ val read = inputStream.read(dataBuffer, 0, math.min(remaining, UPLOAD_SIZE))
+ outputStream.write(dataBuffer, 0, read)
+ remaining -= read
+ }
+ } catch {
+ case e: IOException => logError("Exception occurred while writing some profiler output: ", e)
+ case e@(_: IllegalArgumentException | _: IllegalStateException) =>
+ logError("Some profiler output not written." +
+ " Exception occurred in profiler native code: ", e)
+ case e: Exception => logError("Some profiler output not written. Unexpected exception: ", e)
+ }
+ }
+
+ private def finishWriting(): Unit = {
+ if (profilerDfsDir.isDefined && writing) {
+ try {
+ // shutdown background writer
+ threadpool.shutdown()
+ threadpool.awaitTermination(30, TimeUnit.SECONDS)
+ // flush remaining data
+ writeChunk(true)
+ inputStream.close()
+ outputStream.close()
+ } catch {
+ case _: InterruptedException => Thread.currentThread().interrupt()
+ case e: IOException =>
Review Comment:
The exceptions may come from `threadpool.shutdown`, `awaitTermination` which may throw `InterruptedException` or from `{input,output}Stream.close` which may get an `IOException`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Add support for code profiling executors [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1833859942
> There's a whole slew of errors like the following while building documentation -
>
> ```
> [error] /__w/spark/spark/Loading source file /__w/spark/spark/common/sketch/src/main/java/org/apache/spark/util/sketch/BitArray.java...
> [error] Loading source file /__w/spark/spark/common/sketch/src/main/java/org/apache/spark/util/sketch/BloomFilter.java...
> [error] Loading source file /__w/spark/spark/common/sketch/src/main/java/org/apache/spark/util/sketch/BloomFilterImpl.java...
> [error] Loading source file /__w/spark/spark/common/sketch/src/main/java/org/apache/spark/util/sketch/CountMinSketch.java...
> [error] Loading source file /__w/spark/spark/common/sketch/src/main/java/org/apache/spark/util/sketch/CountMinSketchImpl.java...
> ```
>
> How does one fix this?
The actual error is
```
Warning, treated as error:
don't know which module to import for autodocumenting 'apply_batch' (try placing a "module" or "currentmodule" directive in the document, or giving an explicit module name)
make: *** [Makefile:35: html] Error 2
------------------------------------------------
Jekyll 4.3.2 Please append `--trace` to the `build` command
for any additional information or backtrace.
------------------------------------------------
/__w/spark/spark/docs/_plugins/copy_api_dirs.rb:130:in `<top (required)>': Python doc generation failed (RuntimeError)
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/external.rb:57:in `require'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/external.rb:57:in `block in require_with_graceful_fail'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/external.rb:55:in `each'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/external.rb:55:in `require_with_graceful_fail'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/plugin_manager.rb:96:in `block in require_plugin_files'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/plugin_manager.rb:94:in `each'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/plugin_manager.rb:94:in `require_plugin_files'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/plugin_manager.rb:21:in `conscientious_require'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/site.rb:131:in `setup'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/site.rb:36:in `initialize'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/commands/build.rb:30:in `new'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/commands/build.rb:30:in `process'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/command.rb:91:in `block in process_with_graceful_fail'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/command.rb:91:in `each'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/command.rb:91:in `process_with_graceful_fail'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/lib/jekyll/commands/build.rb:18:in `block (2 levels) in init_with_program'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `block in execute'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `each'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `execute'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/program.rb:44:in `go'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary.rb:21:in `program'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.2/exe/jekyll:15:in `<top (required)>'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/bin/jekyll:23:in `load'
from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/bin/jekyll:23:in `<main>'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408090161
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorCodeProfiler.scala:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ *
+ */
+private[spark] class ExecutorCodeProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private val WRITE_INTERVAL = 30 // seconds
Review Comment:
Why this is a magic number instead of a configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408069210
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
Review Comment:
Let's unify the style. Like 74 line, one-liner (`<td><code>false</code></td>`) is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408066447
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
Review Comment:
`-` -> `:`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424411503
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
Review Comment:
We prefer a Scala-way, `Option`, for nullable variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424410457
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
Review Comment:
After renaming `EXECUTOR_CODE_PROFILING_OUTPUT_DIR`, we need to rename this variable too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Add support for code profiling executors [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1410100843
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
Review Comment:
Because async-profiler requires specific POSIX signals capabilities which Windows implements differently. So async-profiler doesn't support windows. More here: https://github.com/async-profiler/async-profiler/issues/188
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
Review Comment:
Replaced with a link that references jdk17.
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
Review Comment:
Done
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
Review Comment:
Done
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
+ <td></td>
+ <td>
+ An hdfs compatible path to which the profiler's output files are copied. The output files will be written as <i>outputDir/application_id/profile-appname-exec-executor_id.jfr</i> <br/>
+ If no outputDir is specified then the files are not copied over.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.localDir</code></td>
+ <td><code>.</code> i.e. the executor's working dir</td>
+ <td>
+ The local directory in the executor container to write the jfr files to. If not specified the file will be written to the executor's working directory.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.fraction</code></td>
+ <td>0.10</td>
+ <td>
+ The fraction of executors on which to enable code profiling. The executors to be profiled are picked at random.
+ </td>
+ <td>4.0.0</td>
+</tr>
+</table>
+
+### Kubernetes
+On Kubernetes, spark will try to shut down the executor pods while the profiler files are still being saved. To prevent this set
+```
+ spark.kubernetes.executor.deleteOnTermination=false
Review Comment:
Done
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.{EXECUTOR_CODE_PROFILING_ENABLED, EXECUTOR_CODE_PROFILING_FRACTION}
+
+
+/**
+ * Spark plugin to do code profiling of executors
+ *
+ */
+class ExecutorProfilerPlugin extends SparkPlugin {
+ override def driverPlugin(): DriverPlugin = null
+
+ // No-op
+ override def executorPlugin(): ExecutorPlugin = new CodeProfilerExecutorPlugin
+}
+
+class CodeProfilerExecutorPlugin extends ExecutorPlugin with Logging {
+
+ private var sparkConf: SparkConf = _
+ private var pluginCtx: PluginContext = _
+ private var profiler: ExecutorCodeProfiler = _
+ private var codeProfilingEnabled: Boolean = _
+ private var codeProfilingFraction: Double = _
+ private val rand: Random = new Random(System.currentTimeMillis())
+
+ override def init(ctx: PluginContext, extraConf: JMap[String, String]): Unit = {
+ pluginCtx = ctx
+ sparkConf = ctx.conf()
+ codeProfilingEnabled = sparkConf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ codeProfilingFraction = sparkConf.get(EXECUTOR_CODE_PROFILING_FRACTION)
+
+ if (codeProfilingEnabled) {
+ if (rand.nextInt(100) * 0.01 < codeProfilingFraction) {
+ logInfo(s"Executor id ${pluginCtx.executorID()} selected for code profiling")
+ profiler = new ExecutorCodeProfiler(sparkConf, pluginCtx.executorID())
+ profiler.start()
+ }
+ }
+ Map.empty[String, String].asJava
+ }
+
+ override def shutdown(): Unit = {
+
+ logInfo("Executor code profiler shutting down")
+ if (profiler != null) {
+ profiler.stop()
+ }
+ }
+
+
Review Comment:
Done
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.{EXECUTOR_CODE_PROFILING_ENABLED, EXECUTOR_CODE_PROFILING_FRACTION}
+
+
+/**
+ * Spark plugin to do code profiling of executors
+ *
+ */
+class ExecutorProfilerPlugin extends SparkPlugin {
+ override def driverPlugin(): DriverPlugin = null
+
+ // No-op
+ override def executorPlugin(): ExecutorPlugin = new CodeProfilerExecutorPlugin
+}
+
+class CodeProfilerExecutorPlugin extends ExecutorPlugin with Logging {
+
+ private var sparkConf: SparkConf = _
+ private var pluginCtx: PluginContext = _
+ private var profiler: ExecutorCodeProfiler = _
+ private var codeProfilingEnabled: Boolean = _
+ private var codeProfilingFraction: Double = _
+ private val rand: Random = new Random(System.currentTimeMillis())
+
+ override def init(ctx: PluginContext, extraConf: JMap[String, String]): Unit = {
+ pluginCtx = ctx
+ sparkConf = ctx.conf()
+ codeProfilingEnabled = sparkConf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ codeProfilingFraction = sparkConf.get(EXECUTOR_CODE_PROFILING_FRACTION)
+
+ if (codeProfilingEnabled) {
+ if (rand.nextInt(100) * 0.01 < codeProfilingFraction) {
+ logInfo(s"Executor id ${pluginCtx.executorID()} selected for code profiling")
+ profiler = new ExecutorCodeProfiler(sparkConf, pluginCtx.executorID())
+ profiler.start()
+ }
+ }
+ Map.empty[String, String].asJava
+ }
+
+ override def shutdown(): Unit = {
+
Review Comment:
Done
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
+ <td></td>
+ <td>
+ An hdfs compatible path to which the profiler's output files are copied. The output files will be written as <i>outputDir/application_id/profile-appname-exec-executor_id.jfr</i> <br/>
+ If no outputDir is specified then the files are not copied over.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.localDir</code></td>
+ <td><code>.</code> i.e. the executor's working dir</td>
+ <td>
+ The local directory in the executor container to write the jfr files to. If not specified the file will be written to the executor's working directory.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.fraction</code></td>
+ <td>0.10</td>
+ <td>
+ The fraction of executors on which to enable code profiling. The executors to be profiled are picked at random.
+ </td>
+ <td>4.0.0</td>
+</tr>
+</table>
+
+### Kubernetes
Review Comment:
🙏🏾
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
Review Comment:
Done
##########
assembly/pom.xml:
##########
@@ -204,6 +204,16 @@
</dependency>
</dependencies>
</profile>
+ <profile>
+ <id>code-profiler</id>
Review Comment:
Sure
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
+ <td></td>
+ <td>
+ An hdfs compatible path to which the profiler's output files are copied. The output files will be written as <i>outputDir/application_id/profile-appname-exec-executor_id.jfr</i> <br/>
Review Comment:
Done
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
+ <td></td>
Review Comment:
Done
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
Review Comment:
Done
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorCodeProfiler.scala:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ *
Review Comment:
Done
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
+ <td></td>
+ <td>
+ An hdfs compatible path to which the profiler's output files are copied. The output files will be written as <i>outputDir/application_id/profile-appname-exec-executor_id.jfr</i> <br/>
+ If no outputDir is specified then the files are not copied over.
Review Comment:
Running out of space in the dfs will not affect the job. However, the jfr file may be corrupted. Added the warning.
Also added the warning for localDir where out of space in the local system may cause the job to fail on K8s.
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorCodeProfiler.scala:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ *
+ */
+private[spark] class ExecutorCodeProfiler(conf: SparkConf, executorId: String) extends Logging {
Review Comment:
Renamed
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
+ <td></td>
+ <td>
+ An hdfs compatible path to which the profiler's output files are copied. The output files will be written as <i>outputDir/application_id/profile-appname-exec-executor_id.jfr</i> <br/>
+ If no outputDir is specified then the files are not copied over.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.localDir</code></td>
+ <td><code>.</code> i.e. the executor's working dir</td>
+ <td>
+ The local directory in the executor container to write the jfr files to. If not specified the file will be written to the executor's working directory.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.fraction</code></td>
+ <td>0.10</td>
+ <td>
+ The fraction of executors on which to enable code profiling. The executors to be profiled are picked at random.
+ </td>
+ <td>4.0.0</td>
+</tr>
+</table>
+
+### Kubernetes
Review Comment:
🙏🏾
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.{EXECUTOR_CODE_PROFILING_ENABLED, EXECUTOR_CODE_PROFILING_FRACTION}
+
+
+/**
+ * Spark plugin to do code profiling of executors
+ *
Review Comment:
Done
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -799,6 +799,44 @@ package object config {
.intConf
.createOptional
+ private[spark] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.outputDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
+ .version("4.0.0")
+ .stringConf
+ .createOptional
+
+ private[spark] val EXECUTOR_CODE_PROFILING_LOCAL_DIR =
+ ConfigBuilder("spark.executor.profiling.localDir")
+ .doc("Local file system path on executor where profiler output is saved. Defaults to the " +
+ "working directory of the executor process.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(".")
+
+ private[spark] val EXECUTOR_CODE_PROFILING_OPTIONS =
+ ConfigBuilder("spark.executor.profiling.options")
+ .doc("Options to pass on to the async profiler.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault("event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s")
+
+ private[spark] val EXECUTOR_CODE_PROFILING_FRACTION =
+ ConfigBuilder("spark.executor.profiling.fraction")
+ .doc("Fraction of executors to profile")
+ .version("4.0.0")
+ .doubleConf
+ .checkValue(v => v >= 0.0 && v < 1.0,
+ "Fraction of executors to profile must be in [0,1)")
+ .createWithDefault(0.1)
+
Review Comment:
Done. We don't need any yarn or kubernetes specific configuration
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorCodeProfiler.scala:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ *
+ */
+private[spark] class ExecutorCodeProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private val WRITE_INTERVAL = 30 // seconds
Review Comment:
I felt there were already too many configuration parameters and I found this to be a good value for real use cases.
Making this configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1914310340
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.util.concurrent.TimeUnit
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_PROFILING_DFS_DIR =
+ ConfigBuilder("spark.executor.profiling.dfsDir")
Review Comment:
@dongjoon-hyun, I'm also investigating adding profiling support for the driver, should I fork all configurations, or simply treat the driver as a special executor and resue those configurations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408062302
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
Review Comment:
For Java 8 reference link, it looks inappropriate because Apache Spark 4.0.0 dropped all Java versions less than 16.
Do you think we can have Java 17+ link, @parthchandra ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408060429
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
Review Comment:
Let's be specific about what we are depending on.
```
- https://github.com/async-profiler/async-profiler/blob/master/README.md
+ https://github.com/async-profiler/async-profiler/blob/v2.10/README.md
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408055050
##########
assembly/pom.xml:
##########
@@ -204,6 +204,16 @@
</dependency>
</dependencies>
</profile>
+ <profile>
+ <id>code-profiler</id>
Review Comment:
We need to avoid future confusions with Python Profiler, SPARK-40281 (`spark.python.profile.memory`).
- https://pypi.org/project/memory-profiler/
- https://docs.google.com/document/d/e/2PACX-1vR2K4TdrM1eAjNDC1bsflCNRH67UWLoC-lCv6TSUVXD91Ruksm99pYTnCeIm7Ui3RgrrRNcQU_D8-oh/pub
Shall we rename this from `code-profiler` to `jvm-profiler` (or `java-profiler`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Add support for code profiling executors [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1839224900
@dongjoon-hyun addressed your comments. (Sorry it took a while to address the changes)
@HyukjinKwon, I've added an explicit usage example to the README
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1428532057
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.dfsDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
Review Comment:
extra space removed
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+
+
+/**
+ * Spark plugin to do code profiling of executors
+ */
+class ExecutorProfilerPlugin extends SparkPlugin {
+ override def driverPlugin(): DriverPlugin = null
+
+ // No-op
+ override def executorPlugin(): ExecutorPlugin = new CodeProfilerExecutorPlugin
+}
+
+class CodeProfilerExecutorPlugin extends ExecutorPlugin with Logging {
+
+ private var sparkConf: SparkConf = _
+ private var pluginCtx: PluginContext = _
+ private var profiler: ExecutorJVMProfiler = _
+ private var codeProfilingEnabled: Boolean = _
+ private var codeProfilingFraction: Double = _
+ private val rand: Random = new Random(System.currentTimeMillis())
+
+ override def init(ctx: PluginContext, extraConf: JMap[String, String]): Unit = {
+ pluginCtx = ctx
+ sparkConf = ctx.conf()
+ codeProfilingEnabled = sparkConf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ codeProfilingFraction = sparkConf.get(EXECUTOR_CODE_PROFILING_FRACTION)
Review Comment:
Done
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
Review Comment:
Yes, of course. Done
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
Review Comment:
Done
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.dfsDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
+ .version("4.0.0")
+ .stringConf
+ .createOptional
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_LOCAL_DIR =
+ ConfigBuilder("spark.executor.profiling.localDir")
+ .doc("Local file system path on executor where profiler output is saved. Defaults to the " +
+ "working directory of the executor process.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(".")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OPTIONS =
+ ConfigBuilder("spark.executor.profiling.options")
+ .doc("Options to pass on to the async profiler.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault("event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_FRACTION =
+ ConfigBuilder("spark.executor.profiling.fraction")
+ .doc("Fraction of executors to profile")
+ .version("4.0.0")
+ .doubleConf
+ .checkValue(v => v >= 0.0 && v < 1.0,
+ "Fraction of executors to profile must be in [0,1)")
+ .createWithDefault(0.1)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_WRITE_INTERVAL =
+ ConfigBuilder("spark.executor.profiling.writeInterval")
+ .doc("Time interval in seconds after which the profiler output will be synced to dfs")
+ .version("4.0.0")
+ .intConf
Review Comment:
Added the check. There is no upper time limit.
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.dfsDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
+ .version("4.0.0")
+ .stringConf
+ .createOptional
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_LOCAL_DIR =
+ ConfigBuilder("spark.executor.profiling.localDir")
+ .doc("Local file system path on executor where profiler output is saved. Defaults to the " +
+ "working directory of the executor process.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(".")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OPTIONS =
+ ConfigBuilder("spark.executor.profiling.options")
+ .doc("Options to pass on to the async profiler.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault("event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_FRACTION =
+ ConfigBuilder("spark.executor.profiling.fraction")
+ .doc("Fraction of executors to profile")
+ .version("4.0.0")
+ .doubleConf
+ .checkValue(v => v >= 0.0 && v < 1.0,
+ "Fraction of executors to profile must be in [0,1)")
Review Comment:
Corrected to include 100%
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
Review Comment:
Okay
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,109 @@
+# Spark JVM Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -DskipTests -Pjvm-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/v2.10/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://access.redhat.com/documentation/es-es/red_hat_build_of_openjdk/17/html/using_jdk_flight_recorder_with_red_hat_build_of_openjdk/openjdk-flight-recorded-overview) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor :
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.profiler.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td><code>false</code></td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.dfsDir</code></td>
+ <td>(none)</td>
+ <td>
+ An HDFS compatible path to which the profiler's output files are copied. The output files will be written as <i>dfsDir/application_id/profile-appname-exec-executor_id.jfr</i> <br/>
+ If no <i>dfsDir</i> is specified then the files are not copied over. Users should ensure there is sufficient disk space available otherwise it may lead to corrupt jfr files.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.localDir</code></td>
+ <td><code>.</code> i.e. the executor's working dir</td>
+ <td>
+ The local directory in the executor container to write the jfr files to. If not specified the file will be written to the executor's working directory. Users should ensure there is sufficient disk space available on the system as running out of space may result in corrupt jfr file and even cause jobs to fail on systems like K8s.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.fraction</code></td>
+ <td>0.10</td>
+ <td>
+ The fraction of executors on which to enable code profiling. The executors to be profiled are picked at random.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.writeInterval</code></td>
+ <td>30</td>
+ <td>
+ Time interval, in seconds, after which the profiler output will be synced to dfs.
+ </td>
+ <td>4.0.0</td>
+</tr>
+</table>
+
+### Kubernetes
+On Kubernetes, spark will try to shut down the executor pods while the profiler files are still being saved. To prevent this set
+```
+ spark.kubernetes.executor.deleteOnTermination=false
+```
+
+### Example
+```
+./bin/spark-submit \
+ --class <main-class> \
+ --master <master-url> \
+ --deploy-mode <deploy-mode> \
+ -c spark.executor.extraJavaOptions="-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer" \
+ -c spark.plugins=org.apache.spark.executor.profiler.ExecutorProfilerPlugin \
+ -c spark.executor.profiling.enabled=true \
+ -c spark.executor.profiling.outputDir=s3a://my-bucket/spark/profiles/ \
Review Comment:
Oops. Fixed.
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.dfsDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
+ .version("4.0.0")
+ .stringConf
+ .createOptional
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_LOCAL_DIR =
+ ConfigBuilder("spark.executor.profiling.localDir")
+ .doc("Local file system path on executor where profiler output is saved. Defaults to the " +
+ "working directory of the executor process.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(".")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OPTIONS =
+ ConfigBuilder("spark.executor.profiling.options")
+ .doc("Options to pass on to the async profiler.")
Review Comment:
I've added a section on the options in the README.
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
+ logWarning("Executor code profiling aborted due to exception: ", e)
+ return
+ }
+ logInfo("Executor code profiling started.")
+ running = true
+ }
+ startWriting()
+ }
+
+ /** Stops the profiling and saves output to hdfs location. */
+ def stop(): Unit = {
+ if (profiler != null && running) {
+ profiler.execute(stopcmd)
+ logInfo("Code profiler stopped")
+ running = false
+ finishWriting()
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerOutputDir.isDefined) {
+ val applicationId = conf.getAppId
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name");
+ val profilerOutputDirname = profilerOutputDir.get
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk()
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
+ writing = true
+ } catch {
+ case e: Exception =>
+ logError("Failed to start code profiler", e)
+ if (threadpool != null) {
+ threadpool.shutdownNow()
+ }
+ if (inputStream != null) {
+ inputStream.close()
+ }
+ if (outputStream != null) {
+ outputStream.close()
+ }
+ }
+ }
+ }
+
+ private def writeChunk(): Unit = {
+ if (!writing) {
+ return
+ }
+ try {
+ // stop (pause) the profiler, dump the results and then resume. This is not ideal as we miss
+ // the events while the file is being dumped, but that is the only way to make sure that
+ // the chunk of data we are copying to dfs is in a consistent state.
+ profiler.execute(stopcmd)
+ profiler.execute(dumpcmd)
+ var remaining = inputStream.available()
+ profiler.execute(resumecmd)
+ while (remaining > 0) {
+ val read = inputStream.read(dataBuffer, 0, math.min(remaining, UPLOAD_SIZE))
+ outputStream.write(dataBuffer, 0, read)
+ remaining -= read
+ }
+ } catch {
+ case e: Exception => logError("Exception occurred while writing profiler output", e)
+ }
+ }
+
+ private def finishWriting(): Unit = {
+ if (profilerOutputDir.isDefined && writing) {
+ try {
+ // shutdown background writer
+ threadpool.shutdown()
+ threadpool.awaitTermination(30, TimeUnit.SECONDS)
+ // flush remaining data
+ writeChunk()
+ inputStream.close()
+ outputStream.close()
+ } catch {
+ case e: Exception =>
Review Comment:
I've added more specific handling.
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+
+
+/**
+ * Spark plugin to do code profiling of executors
Review Comment:
Done
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+
+
+/**
+ * Spark plugin to do code profiling of executors
+ */
+class ExecutorProfilerPlugin extends SparkPlugin {
+ override def driverPlugin(): DriverPlugin = null
+
+ // No-op
+ override def executorPlugin(): ExecutorPlugin = new CodeProfilerExecutorPlugin
+}
+
+class CodeProfilerExecutorPlugin extends ExecutorPlugin with Logging {
Review Comment:
Sure
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
Review Comment:
Changed to use Option
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
Review Comment:
Done
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
+ logWarning("Executor code profiling aborted due to exception: ", e)
+ return
+ }
+ logInfo("Executor code profiling started.")
+ running = true
+ }
+ startWriting()
+ }
+
+ /** Stops the profiling and saves output to hdfs location. */
+ def stop(): Unit = {
+ if (profiler != null && running) {
+ profiler.execute(stopcmd)
+ logInfo("Code profiler stopped")
+ running = false
+ finishWriting()
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerOutputDir.isDefined) {
+ val applicationId = conf.getAppId
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name");
+ val profilerOutputDirname = profilerOutputDir.get
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk()
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
+ writing = true
+ } catch {
+ case e: Exception =>
+ logError("Failed to start code profiler", e)
+ if (threadpool != null) {
+ threadpool.shutdownNow()
+ }
+ if (inputStream != null) {
+ inputStream.close()
+ }
+ if (outputStream != null) {
+ outputStream.close()
+ }
+ }
+ }
+ }
+
+ private def writeChunk(): Unit = {
+ if (!writing) {
+ return
+ }
+ try {
+ // stop (pause) the profiler, dump the results and then resume. This is not ideal as we miss
+ // the events while the file is being dumped, but that is the only way to make sure that
+ // the chunk of data we are copying to dfs is in a consistent state.
+ profiler.execute(stopcmd)
+ profiler.execute(dumpcmd)
+ var remaining = inputStream.available()
+ profiler.execute(resumecmd)
+ while (remaining > 0) {
+ val read = inputStream.read(dataBuffer, 0, math.min(remaining, UPLOAD_SIZE))
+ outputStream.write(dataBuffer, 0, read)
+ remaining -= read
+ }
+ } catch {
+ case e: Exception => logError("Exception occurred while writing profiler output", e)
+ }
+ }
+
+ private def finishWriting(): Unit = {
+ if (profilerOutputDir.isDefined && writing) {
+ try {
+ // shutdown background writer
+ threadpool.shutdown()
+ threadpool.awaitTermination(30, TimeUnit.SECONDS)
+ // flush remaining data
+ writeChunk()
+ inputStream.close()
+ outputStream.close()
+ } catch {
+ case e: Exception =>
+ logError("Exception in completing profiler output", e)
Review Comment:
Made it a little more descriptive.
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
+ logWarning("Executor code profiling aborted due to exception: ", e)
+ return
Review Comment:
If the profiler fails to start we don't want the job to fail, so I feel it is better to log a warning and keep going.
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
+ logWarning("Executor code profiling aborted due to exception: ", e)
+ return
+ }
+ logInfo("Executor code profiling started.")
+ running = true
+ }
+ startWriting()
+ }
+
+ /** Stops the profiling and saves output to hdfs location. */
Review Comment:
changed
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
Review Comment:
We really want to catch all exceptions here. Irrespective of what the exception was, we want to log it (so user can take corrective action) but we don't want the job to fail.
I'm now checking for the specific exceptions thrown by the profiler but am also catching any other exceptions
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
+ logWarning("Executor code profiling aborted due to exception: ", e)
+ return
+ }
+ logInfo("Executor code profiling started.")
+ running = true
+ }
+ startWriting()
+ }
+
+ /** Stops the profiling and saves output to hdfs location. */
+ def stop(): Unit = {
+ if (profiler != null && running) {
+ profiler.execute(stopcmd)
+ logInfo("Code profiler stopped")
Review Comment:
Of course
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1861829492
@dongjoon-hyun I've addressed your comments. I also made some minor fixes to address some errors I saw while running locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1450194572
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
+ } catch {
+ case e@(_: IllegalArgumentException | _: IllegalStateException | _: IOException) =>
+ logError("JVM profiling aborted. Exception occurred in profiler native code: ", e)
+ case e: Exception => logWarning("Executor JVM profiling aborted due to exception: ", e)
+ }
+ }
+ }
+
+ /** Stops the profiling and saves output to dfs location. */
+ def stop(): Unit = {
+ if (running) {
+ profiler.foreach(p => {
+ p.execute(stopcmd)
+ logInfo("JVM profiler stopped")
+ running = false
+ finishWriting()
+ })
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerDfsDir.isDefined) {
+ val applicationId = try {
+ conf.getAppId
+ } catch {
+ case _: NoSuchElementException => "local-" + System.currentTimeMillis
+ }
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name").replace(" ", "-")
+ val profilerOutputDirname = profilerDfsDir.get
+
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk(false)
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
+ writing = true
+ } catch {
+ case e: Exception =>
+ logError("Failed to start JVM profiler", e)
+ if (threadpool != null) {
+ threadpool.shutdownNow()
+ }
+ if (inputStream != null) {
+ inputStream.close()
+ }
+ if (outputStream != null) {
+ outputStream.close()
+ }
+ }
+ }
+ }
+
+ private def writeChunk(lastChunk: Boolean): Unit = {
+ if (!writing) {
+ return
+ }
+ try {
+ // stop (pause) the profiler, dump the results and then resume. This is not ideal as we miss
+ // the events while the file is being dumped, but that is the only way to make sure that
+ // the chunk of data we are copying to dfs is in a consistent state.
+ profiler.get.execute(stopcmd)
+ profiler.get.execute(dumpcmd)
+ var remaining = inputStream.available()
+ if (!lastChunk) {
+ profiler.get.execute(resumecmd)
+ }
+ while (remaining > 0) {
+ val read = inputStream.read(dataBuffer, 0, math.min(remaining, UPLOAD_SIZE))
+ outputStream.write(dataBuffer, 0, read)
+ remaining -= read
+ }
+ } catch {
+ case e: IOException => logError("Exception occurred while writing some profiler output: ", e)
+ case e@(_: IllegalArgumentException | _: IllegalStateException) =>
Review Comment:
ditto. `e@(` -> `e @ (`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1894323461
Thank you @dongjoon-hyun @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1866836994
That was pretty cool stuff you did in Safari!
I think we may not have to do too much work ourselves.
The way I see it, async_profiler is doing the mapping of the java threads and stack traces already for us (and we know that both async_profiler and honest_profiler avoid the safepoint bias problem so this is as good as it gets). In addition there is a filter API to filter on one or more threads so async_profiler collects events only for the given thread(s). The API for filtering takes a `java.lang.Thread` as input.
The way I see it potentially working is: when a user asks to profile a task, we start profiling for only the task's thread similar to the way a task stack trace is being done today. Then we ship over the the collected data and display it.
I'll have to play around with this though. There might be some gotchas in profiling multiple threads simultaneously and/or some APIs might be private making it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1861974153
A couple of comments:
a) IIRC async profiler does not have a way to associate the java thread with the underlying linux thread - and so from a future evolution point of view, we cannot corelate this with a task's stack traces.
Given work being done as part of SPARK-45151, is there a way to make this work with whatever @yaooqinn added support for ?
b) This PR is creating one file per executor on hdfs - which would end up being very expensive as number of executors per application and number of applications increase.
How are we looking to minimize this cost ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408049330
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
Review Comment:
Please remove the trailing spaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408125008
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -799,6 +799,44 @@ package object config {
.intConf
.createOptional
+ private[spark] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.outputDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
+ .version("4.0.0")
+ .stringConf
+ .createOptional
+
+ private[spark] val EXECUTOR_CODE_PROFILING_LOCAL_DIR =
+ ConfigBuilder("spark.executor.profiling.localDir")
+ .doc("Local file system path on executor where profiler output is saved. Defaults to the " +
+ "working directory of the executor process.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(".")
+
+ private[spark] val EXECUTOR_CODE_PROFILING_OPTIONS =
+ ConfigBuilder("spark.executor.profiling.options")
+ .doc("Options to pass on to the async profiler.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault("event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s")
+
+ private[spark] val EXECUTOR_CODE_PROFILING_FRACTION =
+ ConfigBuilder("spark.executor.profiling.fraction")
+ .doc("Fraction of executors to profile")
+ .version("4.0.0")
+ .doubleConf
+ .checkValue(v => v >= 0.0 && v < 1.0,
+ "Fraction of executors to profile must be in [0,1)")
+ .createWithDefault(0.1)
+
Review Comment:
For YARN and K8s, we have `Config.scala`.
https://github.com/apache/spark/blob/7a0d0411aa02e7e1b6beb393966ada2c54c09870/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala#L26
https://github.com/apache/spark/blob/7a0d0411aa02e7e1b6beb393966ada2c54c09870/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala#L27
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1450198369
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
+ } catch {
+ case e@(_: IllegalArgumentException | _: IllegalStateException | _: IOException) =>
+ logError("JVM profiling aborted. Exception occurred in profiler native code: ", e)
+ case e: Exception => logWarning("Executor JVM profiling aborted due to exception: ", e)
+ }
+ }
+ }
+
+ /** Stops the profiling and saves output to dfs location. */
+ def stop(): Unit = {
+ if (running) {
+ profiler.foreach(p => {
+ p.execute(stopcmd)
+ logInfo("JVM profiler stopped")
+ running = false
+ finishWriting()
+ })
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerDfsDir.isDefined) {
+ val applicationId = try {
+ conf.getAppId
+ } catch {
+ case _: NoSuchElementException => "local-" + System.currentTimeMillis
+ }
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name").replace(" ", "-")
+ val profilerOutputDirname = profilerDfsDir.get
+
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk(false)
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
+ writing = true
+ } catch {
+ case e: Exception =>
+ logError("Failed to start JVM profiler", e)
+ if (threadpool != null) {
+ threadpool.shutdownNow()
+ }
+ if (inputStream != null) {
+ inputStream.close()
+ }
+ if (outputStream != null) {
+ outputStream.close()
+ }
+ }
+ }
+ }
+
+ private def writeChunk(lastChunk: Boolean): Unit = {
+ if (!writing) {
+ return
+ }
+ try {
+ // stop (pause) the profiler, dump the results and then resume. This is not ideal as we miss
+ // the events while the file is being dumped, but that is the only way to make sure that
+ // the chunk of data we are copying to dfs is in a consistent state.
+ profiler.get.execute(stopcmd)
+ profiler.get.execute(dumpcmd)
+ var remaining = inputStream.available()
+ if (!lastChunk) {
+ profiler.get.execute(resumecmd)
+ }
+ while (remaining > 0) {
+ val read = inputStream.read(dataBuffer, 0, math.min(remaining, UPLOAD_SIZE))
+ outputStream.write(dataBuffer, 0, read)
+ remaining -= read
+ }
+ } catch {
+ case e: IOException => logError("Exception occurred while writing some profiler output: ", e)
+ case e@(_: IllegalArgumentException | _: IllegalStateException) =>
+ logError("Some profiler output not written." +
+ " Exception occurred in profiler native code: ", e)
+ case e: Exception => logError("Some profiler output not written. Unexpected exception: ", e)
Review Comment:
I'm wondering what we can get in this case because the flag `writing` is still `true`. Do we need to keep `writing` because we need to invoke `finishWriting`? However, it seems that we cannot invoke `inputStream.close()` and `outputStream.close()` eventually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1892797713
Merged to master for Apache Spark 4.0.0.
Thank you, @parthchandra and all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1828297913
There's a whole slew of errors like the following while building documentation -
```
[error] /__w/spark/spark/Loading source file /__w/spark/spark/common/sketch/src/main/java/org/apache/spark/util/sketch/BitArray.java...
[error] Loading source file /__w/spark/spark/common/sketch/src/main/java/org/apache/spark/util/sketch/BloomFilter.java...
[error] Loading source file /__w/spark/spark/common/sketch/src/main/java/org/apache/spark/util/sketch/BloomFilterImpl.java...
[error] Loading source file /__w/spark/spark/common/sketch/src/main/java/org/apache/spark/util/sketch/CountMinSketch.java...
[error] Loading source file /__w/spark/spark/common/sketch/src/main/java/org/apache/spark/util/sketch/CountMinSketchImpl.java...
```
How does one fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424416052
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
+ logWarning("Executor code profiling aborted due to exception: ", e)
+ return
+ }
+ logInfo("Executor code profiling started.")
+ running = true
+ }
+ startWriting()
+ }
+
+ /** Stops the profiling and saves output to hdfs location. */
Review Comment:
Since `hdfs` could be misleading, `dfs`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1450186962
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
+ } catch {
+ case e@(_: IllegalArgumentException | _: IllegalStateException | _: IOException) =>
Review Comment:
nit. Add proper spaces?
```scala
- case e@(
+ case e @ (
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424412268
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
Review Comment:
Please change this after we change `profiler` into `Option[AsyncProfiler]`
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
Review Comment:
We prefer a Scala-way, `Option[AsyncProfiler]`, for nullable variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424404258
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.dfsDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
+ .version("4.0.0")
+ .stringConf
+ .createOptional
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_LOCAL_DIR =
+ ConfigBuilder("spark.executor.profiling.localDir")
+ .doc("Local file system path on executor where profiler output is saved. Defaults to the " +
+ "working directory of the executor process.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(".")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OPTIONS =
+ ConfigBuilder("spark.executor.profiling.options")
+ .doc("Options to pass on to the async profiler.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault("event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_FRACTION =
+ ConfigBuilder("spark.executor.profiling.fraction")
+ .doc("Fraction of executors to profile")
+ .version("4.0.0")
+ .doubleConf
+ .checkValue(v => v >= 0.0 && v < 1.0,
+ "Fraction of executors to profile must be in [0,1)")
+ .createWithDefault(0.1)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_WRITE_INTERVAL =
+ ConfigBuilder("spark.executor.profiling.writeInterval")
+ .doc("Time interval in seconds after which the profiler output will be synced to dfs")
+ .version("4.0.0")
+ .intConf
Review Comment:
Please use `.timeConf(TimeUnit.SECONDS)` instead of `intConf`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424406829
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+
+
+/**
+ * Spark plugin to do code profiling of executors
+ */
+class ExecutorProfilerPlugin extends SparkPlugin {
+ override def driverPlugin(): DriverPlugin = null
+
+ // No-op
+ override def executorPlugin(): ExecutorPlugin = new CodeProfilerExecutorPlugin
+}
+
+class CodeProfilerExecutorPlugin extends ExecutorPlugin with Logging {
+
+ private var sparkConf: SparkConf = _
+ private var pluginCtx: PluginContext = _
+ private var profiler: ExecutorJVMProfiler = _
+ private var codeProfilingEnabled: Boolean = _
+ private var codeProfilingFraction: Double = _
+ private val rand: Random = new Random(System.currentTimeMillis())
+
+ override def init(ctx: PluginContext, extraConf: JMap[String, String]): Unit = {
+ pluginCtx = ctx
+ sparkConf = ctx.conf()
+ codeProfilingEnabled = sparkConf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ codeProfilingFraction = sparkConf.get(EXECUTOR_CODE_PROFILING_FRACTION)
Review Comment:
Please move this into the inside of `codeProfilingEnabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1863873954
> `AsyncGetCallTrace` is used precisely to map calls in the native thread to calls in the java thread. Not sure exactly what you are looking for here. Are you looking to profile individual tasks? It certainly can be done, but would require some changes similar to [SPARK-45151](https://issues.apache.org/jira/browse/SPARK-45151) and some additional work if you want the profile available thru the UI. Or are you looking to enhance [SPARK-45151](https://issues.apache.org/jira/browse/SPARK-45151) and get a stack trace that includes native calls? This is a little harder via async_profiler since there is no API to get a snapshot. Note that getting a profile needs to be collected over a period of time and so is different from getting a snapshot as [SPARK-45151](https://issues.apache.org/jira/browse/SPARK-45151) is doing.
There is a difference between native thread id's and java thread ids.
Given the async profiler output, can we map it to the corresponding task (given task's java thread id) ?
My understanding is currently no - but if I am missing something, do let me know.
Assuming no, this means the stack traces generated are for all threads in the executor jvm - and so does not allow us to get stack traces and/or flamegraphs for a particular task, tasks of a stage, etc.
If yes, this would be very useful - and will allow for future evolution as part of SPARK-44893 [1].
>
> > Simply dumping per executor flamegraphs or stack traces has limited utility (and can be done today).
>
> I would suggest that this PR makes it trivially simple to profile with no setup required. On K8s, with ephemeral storage, it is not a simple task to dump a profile to disk and get it off the pod before the pod is destroyed (it was in fact the original motivation behind doing this).
I am not seeing a lot of value in including this into Apache Spark itself - plugin api is public, and users can leverage it to do precisely what the PR is proposing.
On other hand, if the PR is integrating well with SPARK-44893 [1] - and/or there is a path to leveraging it in that work, it would be more useful.
I am not exactly -1 on this @dongjoon-hyun , but I am not seeing a lot of value in it: will let you make the call.
[1] This is the jira I was trying to paste, but github mobile messed it up - and ended up referencing a subtask !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1450191904
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
+ } catch {
+ case e@(_: IllegalArgumentException | _: IllegalStateException | _: IOException) =>
+ logError("JVM profiling aborted. Exception occurred in profiler native code: ", e)
+ case e: Exception => logWarning("Executor JVM profiling aborted due to exception: ", e)
+ }
+ }
+ }
+
+ /** Stops the profiling and saves output to dfs location. */
+ def stop(): Unit = {
+ if (running) {
+ profiler.foreach(p => {
+ p.execute(stopcmd)
+ logInfo("JVM profiler stopped")
+ running = false
+ finishWriting()
+ })
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerDfsDir.isDefined) {
+ val applicationId = try {
+ conf.getAppId
+ } catch {
+ case _: NoSuchElementException => "local-" + System.currentTimeMillis
+ }
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name").replace(" ", "-")
+ val profilerOutputDirname = profilerDfsDir.get
+
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk(false)
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
Review Comment:
The indentation of the above four lines looks weird to me. Could you revise?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1888868394
Thank you all. Especially, @mridulm for many ideas and advices on this PR reviews.
- Ya, as you mentioned, `honest-profiler` has been abandoned about two years unfortunately. We should not depend on it.
- Like `Safari` (2019), there has been many profiler approaches outside the community because it's valuable. However, IIUC, it's not easily available to the Apache Spark users in these days.
- I believe the value of this PR is capable of providing `Apache Spark 4.0.0` users the foundation of easy pre-defined profiling with additional help like `spark.executor.profiling.fraction`.
- To be clear, we don't recommend users to enable this for all executors of all jobs.
- Lastly, this plugin way is non-intrusive and has no maintain cost because it's based on the Spark standard. For the proposed integrated and detailed analysis, I believe we can achieve them in the future in a way because we agreed lots of values of that integrated analysis.
I hope we can merge this as a part of Apache Spark 4.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1868388067
Sorry for being away, @mridulm and @parthchandra . I've been traveling in South Korea since 14th December. I'll catch up the discussion and will revisit this PR on January. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408092014
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorCodeProfiler.scala:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ *
+ */
+private[spark] class ExecutorCodeProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private val WRITE_INTERVAL = 30 // seconds
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
Review Comment:
Could you give me a link for this method, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408082240
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorCodeProfiler.scala:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ *
Review Comment:
nit. Remove the empty line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1826981460
how do you use it? would be great if it contains the example, how to run, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408094353
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.{EXECUTOR_CODE_PROFILING_ENABLED, EXECUTOR_CODE_PROFILING_FRACTION}
+
+
+/**
+ * Spark plugin to do code profiling of executors
+ *
Review Comment:
nit. Remove this line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408074358
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
+ <td></td>
+ <td>
+ An hdfs compatible path to which the profiler's output files are copied. The output files will be written as <i>outputDir/application_id/profile-appname-exec-executor_id.jfr</i> <br/>
+ If no outputDir is specified then the files are not copied over.
Review Comment:
Please add a warning about Out-Of-Disk situation because K8s is very strict about the disk usage unlike YARN or Standalone clusters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408072109
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
+ <td></td>
Review Comment:
Please use `(none)`.
https://github.com/apache/spark/blob/ec71e223fc4f2d1e3a444d5c4fff8b281dc7b58c/docs/configuration.md?plain=1#L354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408057550
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
Review Comment:
In addition, please add `-DskipTests` in order to be more copy-and-paste friendly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408080764
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
+ <td></td>
+ <td>
+ An hdfs compatible path to which the profiler's output files are copied. The output files will be written as <i>outputDir/application_id/profile-appname-exec-executor_id.jfr</i> <br/>
+ If no outputDir is specified then the files are not copied over.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.localDir</code></td>
+ <td><code>.</code> i.e. the executor's working dir</td>
+ <td>
+ The local directory in the executor container to write the jfr files to. If not specified the file will be written to the executor's working directory.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.fraction</code></td>
+ <td>0.10</td>
+ <td>
+ The fraction of executors on which to enable code profiling. The executors to be profiled are picked at random.
+ </td>
+ <td>4.0.0</td>
+</tr>
+</table>
+
+### Kubernetes
Review Comment:
Thank you for adding this.
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
+ <td></td>
+ <td>
+ An hdfs compatible path to which the profiler's output files are copied. The output files will be written as <i>outputDir/application_id/profile-appname-exec-executor_id.jfr</i> <br/>
+ If no outputDir is specified then the files are not copied over.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.localDir</code></td>
+ <td><code>.</code> i.e. the executor's working dir</td>
+ <td>
+ The local directory in the executor container to write the jfr files to. If not specified the file will be written to the executor's working directory.
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.fraction</code></td>
+ <td>0.10</td>
+ <td>
+ The fraction of executors on which to enable code profiling. The executors to be profiled are picked at random.
+ </td>
+ <td>4.0.0</td>
+</tr>
+</table>
+
+### Kubernetes
+On Kubernetes, spark will try to shut down the executor pods while the profiler files are still being saved. To prevent this set
+```
+ spark.kubernetes.executor.deleteOnTermination=false
Review Comment:
Remove the trailing spaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408079288
##########
connector/profiler/README.md:
##########
@@ -0,0 +1,86 @@
+# Spark Code Profiler Plugin
+
+## Build
+
+To build
+```
+ ./build/mvn clean package -P code-profiler
+```
+
+## Executor Code Profiling
+
+The spark profiler module enables code profiling of executors in cluster mode based on the the [async profiler](https://github.com/async-profiler/async-profiler/blob/master/README.md), a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures [Java Flight Recorder (jfr)](https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u#) files for each executor; these can be read by many tools including Java Mission Control and Intellij.
+
+The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.
+
+Code profiling is currently only supported for
+
+* Linux (x64)
+* Linux (arm 64)
+* Linux (musl, x64)
+* MacOS
+
+To get maximum profiling information set the following jvm options for the executor -
+
+```
+ -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer
+```
+
+For more information on async_profiler see the [Async Profiler Manual](https://krzysztofslusarski.github.io/2022/12/12/async-manual.html)
+
+
+To enable code profiling, first enable the code profiling plugin via
+
+```
+spark.plugins=org.apache.spark.executor.ExecutorProfilerPlugin
+```
+
+Then enable the profiling in the configuration.
+
+
+### Code profiling configuration
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
+<tr>
+ <td><code>spark.executor.profiling.enabled</code></td>
+ <td>
+ <code>false</code>
+ </td>
+ <td>
+ If true, will enable code profiling
+ </td>
+ <td>4.0.0</td>
+</tr>
+<tr>
+ <td><code>spark.executor.profiling.outputDir</code></td>
Review Comment:
Since this is remote HDFS-compatible location, let's follow our convention like the following.
https://github.com/apache/spark/blob/ec71e223fc4f2d1e3a444d5c4fff8b281dc7b58c/docs/configuration.md?plain=1#L442
In short, please rename `outputDir` to `dfsDir`.
```
- spark.executor.profiling.outputDir
+ spark.executor.profiling.dfsDir
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1839226493
Thank you for updates, @parthchandra .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424414904
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
+ logWarning("Executor code profiling aborted due to exception: ", e)
+ return
Review Comment:
May I ask why we prevent the exception propagation? I believe it's better instead of ignoring like this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424416324
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
+ logWarning("Executor code profiling aborted due to exception: ", e)
+ return
+ }
+ logInfo("Executor code profiling started.")
+ running = true
+ }
+ startWriting()
+ }
+
+ /** Stops the profiling and saves output to hdfs location. */
+ def stop(): Unit = {
+ if (profiler != null && running) {
+ profiler.execute(stopcmd)
+ logInfo("Code profiler stopped")
Review Comment:
`Code` -> `JVM`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1866074886
That sounds promising !
What is unclear to me is how we are going to do the mapping without something which ends up introducing safe point bias (essentially, cost of this operation) ...
For example, if the native to java thread mapping requires mxbean.getThreadInfo and/or similar approaches, it becomes fairly expensive.
Essentially what I am trying to make sure is - given `(native-thread-id -> timestamp -> stack_dumps+)*`, can we identify the `native-thread -> java-thread-id` ?
If yes, we can build the `java-thread-id -> task-id` in spark, and essentially get to `(task-id -> stack_dumps+)*` for all (most ?) tasks.
When we built [Safari](https://blog.cloudera.com/demystifying-spark-jobs-to-optimize-for-cost-and-performance/), this is what ended up being extremely powerful for understanding application performance - per-task stack dumps, correlated across all tasks for a stage: allowing us to understand what the stack dump for a particular stage is, what the difference between 'expensive' tasks in a stage vs average task is, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424401384
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.dfsDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
+ .version("4.0.0")
+ .stringConf
+ .createOptional
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_LOCAL_DIR =
+ ConfigBuilder("spark.executor.profiling.localDir")
+ .doc("Local file system path on executor where profiler output is saved. Defaults to the " +
+ "working directory of the executor process.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(".")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OPTIONS =
+ ConfigBuilder("spark.executor.profiling.options")
+ .doc("Options to pass on to the async profiler.")
Review Comment:
Could you add some pointer for async profiler values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424398324
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
Review Comment:
`EXECUTOR_CODE_PROFILING_DFS_DIR`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424409198
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
+ logWarning("Executor code profiling aborted due to exception: ", e)
+ return
+ }
+ logInfo("Executor code profiling started.")
+ running = true
+ }
+ startWriting()
+ }
+
+ /** Stops the profiling and saves output to hdfs location. */
+ def stop(): Unit = {
+ if (profiler != null && running) {
+ profiler.execute(stopcmd)
+ logInfo("Code profiler stopped")
+ running = false
+ finishWriting()
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerOutputDir.isDefined) {
+ val applicationId = conf.getAppId
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name");
+ val profilerOutputDirname = profilerOutputDir.get
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk()
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
+ writing = true
+ } catch {
+ case e: Exception =>
+ logError("Failed to start code profiler", e)
+ if (threadpool != null) {
+ threadpool.shutdownNow()
+ }
+ if (inputStream != null) {
+ inputStream.close()
+ }
+ if (outputStream != null) {
+ outputStream.close()
+ }
+ }
+ }
+ }
+
+ private def writeChunk(): Unit = {
+ if (!writing) {
+ return
+ }
+ try {
+ // stop (pause) the profiler, dump the results and then resume. This is not ideal as we miss
+ // the events while the file is being dumped, but that is the only way to make sure that
+ // the chunk of data we are copying to dfs is in a consistent state.
+ profiler.execute(stopcmd)
+ profiler.execute(dumpcmd)
+ var remaining = inputStream.available()
+ profiler.execute(resumecmd)
+ while (remaining > 0) {
+ val read = inputStream.read(dataBuffer, 0, math.min(remaining, UPLOAD_SIZE))
+ outputStream.write(dataBuffer, 0, read)
+ remaining -= read
+ }
+ } catch {
+ case e: Exception => logError("Exception occurred while writing profiler output", e)
+ }
+ }
+
+ private def finishWriting(): Unit = {
+ if (profilerOutputDir.isDefined && writing) {
+ try {
+ // shutdown background writer
+ threadpool.shutdown()
+ threadpool.awaitTermination(30, TimeUnit.SECONDS)
+ // flush remaining data
+ writeChunk()
+ inputStream.close()
+ outputStream.close()
+ } catch {
+ case e: Exception =>
Review Comment:
Please use a specific exception instead of `Exception`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424405666
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/package.scala:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import org.apache.spark.internal.config.ConfigBuilder
+
+package object profiler {
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.dfsDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
+ .version("4.0.0")
+ .stringConf
+ .createOptional
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_LOCAL_DIR =
+ ConfigBuilder("spark.executor.profiling.localDir")
+ .doc("Local file system path on executor where profiler output is saved. Defaults to the " +
+ "working directory of the executor process.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(".")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_OPTIONS =
+ ConfigBuilder("spark.executor.profiling.options")
+ .doc("Options to pass on to the async profiler.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault("event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s")
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_FRACTION =
+ ConfigBuilder("spark.executor.profiling.fraction")
+ .doc("Fraction of executors to profile")
+ .version("4.0.0")
+ .doubleConf
+ .checkValue(v => v >= 0.0 && v < 1.0,
+ "Fraction of executors to profile must be in [0,1)")
+ .createWithDefault(0.1)
+
+ private[profiler] val EXECUTOR_CODE_PROFILING_WRITE_INTERVAL =
+ ConfigBuilder("spark.executor.profiling.writeInterval")
+ .doc("Time interval in seconds after which the profiler output will be synced to dfs")
+ .version("4.0.0")
+ .intConf
Review Comment:
In addition, shall we add `checkValue`? For example, the following?
```scala
.checkValue(_ >= 0, "Write interval should be non-negative")
```
Is it positive or non-negative? Please specify our limit here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424407504
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorProfilerPlugin.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.util.{Map => JMap}
+
+import scala.jdk.CollectionConverters._
+import scala.util.Random
+
+import org.apache.spark.SparkConf
+import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
+import org.apache.spark.internal.Logging
+
+
+/**
+ * Spark plugin to do code profiling of executors
+ */
+class ExecutorProfilerPlugin extends SparkPlugin {
+ override def driverPlugin(): DriverPlugin = null
+
+ // No-op
+ override def executorPlugin(): ExecutorPlugin = new CodeProfilerExecutorPlugin
+}
+
+class CodeProfilerExecutorPlugin extends ExecutorPlugin with Logging {
Review Comment:
`CodeProfilerExecutorPlugin` -> `JVMProfilerExecutorPlugin`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424409883
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
+ logWarning("Executor code profiling aborted due to exception: ", e)
+ return
+ }
+ logInfo("Executor code profiling started.")
+ running = true
+ }
+ startWriting()
+ }
+
+ /** Stops the profiling and saves output to hdfs location. */
+ def stop(): Unit = {
+ if (profiler != null && running) {
+ profiler.execute(stopcmd)
+ logInfo("Code profiler stopped")
+ running = false
+ finishWriting()
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerOutputDir.isDefined) {
+ val applicationId = conf.getAppId
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name");
+ val profilerOutputDirname = profilerOutputDir.get
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk()
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
+ writing = true
+ } catch {
+ case e: Exception =>
+ logError("Failed to start code profiler", e)
+ if (threadpool != null) {
+ threadpool.shutdownNow()
+ }
+ if (inputStream != null) {
+ inputStream.close()
+ }
+ if (outputStream != null) {
+ outputStream.close()
+ }
+ }
+ }
+ }
+
+ private def writeChunk(): Unit = {
+ if (!writing) {
+ return
+ }
+ try {
+ // stop (pause) the profiler, dump the results and then resume. This is not ideal as we miss
+ // the events while the file is being dumped, but that is the only way to make sure that
+ // the chunk of data we are copying to dfs is in a consistent state.
+ profiler.execute(stopcmd)
+ profiler.execute(dumpcmd)
+ var remaining = inputStream.available()
+ profiler.execute(resumecmd)
+ while (remaining > 0) {
+ val read = inputStream.read(dataBuffer, 0, math.min(remaining, UPLOAD_SIZE))
+ outputStream.write(dataBuffer, 0, read)
+ remaining -= read
+ }
+ } catch {
+ case e: Exception => logError("Exception occurred while writing profiler output", e)
+ }
+ }
+
+ private def finishWriting(): Unit = {
+ if (profilerOutputDir.isDefined && writing) {
+ try {
+ // shutdown background writer
+ threadpool.shutdown()
+ threadpool.awaitTermination(30, TimeUnit.SECONDS)
+ // flush remaining data
+ writeChunk()
+ inputStream.close()
+ outputStream.close()
+ } catch {
+ case e: Exception =>
+ logError("Exception in completing profiler output", e)
Review Comment:
Could you add a descriptive message about what happens if we log and ignore this exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1424413358
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_CODE_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_CODE_PROFILING_OPTIONS)
+ private val profilerOutputDir = conf.get(EXECUTOR_CODE_PROFILING_OUTPUT_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_CODE_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_CODE_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: AsyncProfiler = if (enableProfiler) {
+ if (AsyncProfilerLoader.isSupported) {
+ AsyncProfilerLoader.load()
+ } else {
+ logWarning("Executor code profiling is enabled but is not supported for this platform")
+ null
+ }
+ } else {
+ null
+ }
+
+ def start(): Unit = {
+ if (profiler != null && !running) {
+ logInfo("Executor code profiling starting.")
+ try {
+ profiler.execute(startcmd)
+ } catch {
+ case e: Exception =>
Review Comment:
Please use a more specific exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408088814
##########
connector/profiler/src/main/scala/org/apache/spark/executor/ExecutorCodeProfiler.scala:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config._
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async code profiler
+ *
+ */
+private[spark] class ExecutorCodeProfiler(conf: SparkConf, executorId: String) extends Logging {
Review Comment:
Spark Executors can have Java, Puython, and R runtimes. Given that, `Code` is a little vague term.
- I'd like to propose to rename `ExecutorCodeProfiler` to `ExecutorJVMProfiler`.
- Otherwise, at least, please document it clearly that this is JVM-only feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408120317
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -799,6 +799,44 @@ package object config {
.intConf
.createOptional
+ private[spark] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.outputDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
+ .version("4.0.0")
+ .stringConf
+ .createOptional
+
+ private[spark] val EXECUTOR_CODE_PROFILING_LOCAL_DIR =
+ ConfigBuilder("spark.executor.profiling.localDir")
+ .doc("Local file system path on executor where profiler output is saved. Defaults to the " +
+ "working directory of the executor process.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(".")
+
+ private[spark] val EXECUTOR_CODE_PROFILING_OPTIONS =
+ ConfigBuilder("spark.executor.profiling.options")
+ .doc("Options to pass on to the async profiler.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault("event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s")
+
+ private[spark] val EXECUTOR_CODE_PROFILING_FRACTION =
+ ConfigBuilder("spark.executor.profiling.fraction")
+ .doc("Fraction of executors to profile")
+ .version("4.0.0")
+ .doubleConf
+ .checkValue(v => v >= 0.0 && v < 1.0,
+ "Fraction of executors to profile must be in [0,1)")
+ .createWithDefault(0.1)
+
Review Comment:
If you don't mind, shall we try to move this into a seperate `package` like Kafka module under `connector`.
https://github.com/apache/spark/blob/7a0d0411aa02e7e1b6beb393966ada2c54c09870/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/package.scala#L25-L34
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094][CONNECT] Add support for code profiling executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1408120317
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -799,6 +799,44 @@ package object config {
.intConf
.createOptional
+ private[spark] val EXECUTOR_CODE_PROFILING_ENABLED =
+ ConfigBuilder("spark.executor.profiling.enabled")
+ .doc("Turn on code profiling via async_profiler in executors.")
+ .version("4.0.0")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val EXECUTOR_CODE_PROFILING_OUTPUT_DIR =
+ ConfigBuilder("spark.executor.profiling.outputDir")
+ .doc("HDFS compatible file-system path to where the profiler will write output jfr files.")
+ .version("4.0.0")
+ .stringConf
+ .createOptional
+
+ private[spark] val EXECUTOR_CODE_PROFILING_LOCAL_DIR =
+ ConfigBuilder("spark.executor.profiling.localDir")
+ .doc("Local file system path on executor where profiler output is saved. Defaults to the " +
+ "working directory of the executor process.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(".")
+
+ private[spark] val EXECUTOR_CODE_PROFILING_OPTIONS =
+ ConfigBuilder("spark.executor.profiling.options")
+ .doc("Options to pass on to the async profiler.")
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault("event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s")
+
+ private[spark] val EXECUTOR_CODE_PROFILING_FRACTION =
+ ConfigBuilder("spark.executor.profiling.fraction")
+ .doc("Fraction of executors to profile")
+ .version("4.0.0")
+ .doubleConf
+ .checkValue(v => v >= 0.0 && v < 1.0,
+ "Fraction of executors to profile must be in [0,1)")
+ .createWithDefault(0.1)
+
Review Comment:
If you don't mind, please move this into a seperate `package` like Kafka module under `connector`.
https://github.com/apache/spark/blob/7a0d0411aa02e7e1b6beb393966ada2c54c09870/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/package.scala#L25-L34
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #44021:
URL: https://github.com/apache/spark/pull/44021#issuecomment-1863659695
If async profiler does not allow us to map the native thread to its java thread (please validate this) - we cannot map stack traces to the corresponding task threads - and so limits usability of this integration in spark.
If this limitation exists, we should explore alternatives which might support it and is easy to integrate (honest profiler supports it but is not easy to integrate iirc).
Simply dumping per executor flamegraphs or stack traces has limited utility (and can be done today) unless we have a path to integrating with SPARK-45151 IMO.
Thoughts @dongjoon-hyun ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46094] Support Executor JVM Profiling [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #44021:
URL: https://github.com/apache/spark/pull/44021#discussion_r1451054921
##########
connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.executor.profiler
+
+import java.io.{BufferedInputStream, FileInputStream, InputStream, IOException}
+import java.net.URI
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import one.profiler.{AsyncProfiler, AsyncProfilerLoader}
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.ThreadUtils
+
+
+/**
+ * A class that enables the async JVM code profiler
+ */
+private[spark] class ExecutorJVMProfiler(conf: SparkConf, executorId: String) extends Logging {
+
+ private var running = false
+ private val enableProfiler = conf.get(EXECUTOR_PROFILING_ENABLED)
+ private val profilerOptions = conf.get(EXECUTOR_PROFILING_OPTIONS)
+ private val profilerDfsDir = conf.get(EXECUTOR_PROFILING_DFS_DIR)
+ private val profilerLocalDir = conf.get(EXECUTOR_PROFILING_LOCAL_DIR)
+ private val writeInterval = conf.get(EXECUTOR_PROFILING_WRITE_INTERVAL)
+
+ private val startcmd = s"start,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val stopcmd = s"stop,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val dumpcmd = s"dump,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+ private val resumecmd = s"resume,$profilerOptions,file=$profilerLocalDir/profile.jfr"
+
+ private val UPLOAD_SIZE = 8 * 1024 * 1024 // 8 MB
+ private var outputStream: FSDataOutputStream = _
+ private var inputStream: InputStream = _
+ private val dataBuffer = new Array[Byte](UPLOAD_SIZE)
+ private var threadpool: ScheduledExecutorService = _
+ private var writing: Boolean = false
+
+ val profiler: Option[AsyncProfiler] = {
+ Option(
+ if (enableProfiler && AsyncProfilerLoader.isSupported) AsyncProfilerLoader.load() else null
+ )
+ }
+
+ def start(): Unit = {
+ if (!running) {
+ try {
+ profiler.foreach(p => {
+ p.execute(startcmd)
+ logInfo("Executor JVM profiling started.")
+ running = true
+ startWriting()
+ }
+ )
+ } catch {
+ case e@(_: IllegalArgumentException | _: IllegalStateException | _: IOException) =>
+ logError("JVM profiling aborted. Exception occurred in profiler native code: ", e)
+ case e: Exception => logWarning("Executor JVM profiling aborted due to exception: ", e)
+ }
+ }
+ }
+
+ /** Stops the profiling and saves output to dfs location. */
+ def stop(): Unit = {
+ if (running) {
+ profiler.foreach(p => {
+ p.execute(stopcmd)
+ logInfo("JVM profiler stopped")
+ running = false
+ finishWriting()
+ })
+ }
+ }
+
+ private def startWriting(): Unit = {
+ if (profilerDfsDir.isDefined) {
+ val applicationId = try {
+ conf.getAppId
+ } catch {
+ case _: NoSuchElementException => "local-" + System.currentTimeMillis
+ }
+ val config = SparkHadoopUtil.get.newConfiguration(conf)
+ val appName = conf.get("spark.app.name").replace(" ", "-")
+ val profilerOutputDirname = profilerDfsDir.get
+
+ val profileOutputFile =
+ s"$profilerOutputDirname/$applicationId/profile-$appName-exec-$executorId.jfr"
+ val fs = FileSystem.get(new URI(profileOutputFile), config);
+ val filenamePath = new Path(profileOutputFile)
+ outputStream = fs.create(filenamePath)
+ try {
+ if (fs.exists(filenamePath)) {
+ fs.delete(filenamePath, true)
+ }
+ logInfo(s"Copying executor profiling file to $profileOutputFile")
+ inputStream = new BufferedInputStream(new FileInputStream(s"$profilerLocalDir/profile.jfr"))
+ threadpool = ThreadUtils.newDaemonSingleThreadScheduledExecutor("profilerOutputThread")
+ threadpool.scheduleWithFixedDelay(new Runnable() {
+ override def run(): Unit = writeChunk(false)
+ }, writeInterval, writeInterval,
+ TimeUnit.SECONDS)
+ writing = true
+ } catch {
+ case e: Exception =>
+ logError("Failed to start JVM profiler", e)
+ if (threadpool != null) {
+ threadpool.shutdownNow()
+ }
+ if (inputStream != null) {
+ inputStream.close()
+ }
+ if (outputStream != null) {
+ outputStream.close()
+ }
+ }
+ }
+ }
+
+ private def writeChunk(lastChunk: Boolean): Unit = {
+ if (!writing) {
+ return
+ }
+ try {
+ // stop (pause) the profiler, dump the results and then resume. This is not ideal as we miss
+ // the events while the file is being dumped, but that is the only way to make sure that
+ // the chunk of data we are copying to dfs is in a consistent state.
+ profiler.get.execute(stopcmd)
+ profiler.get.execute(dumpcmd)
+ var remaining = inputStream.available()
+ if (!lastChunk) {
+ profiler.get.execute(resumecmd)
+ }
+ while (remaining > 0) {
+ val read = inputStream.read(dataBuffer, 0, math.min(remaining, UPLOAD_SIZE))
+ outputStream.write(dataBuffer, 0, read)
+ remaining -= read
+ }
+ } catch {
+ case e: IOException => logError("Exception occurred while writing some profiler output: ", e)
+ case e@(_: IllegalArgumentException | _: IllegalStateException) =>
+ logError("Some profiler output not written." +
+ " Exception occurred in profiler native code: ", e)
+ case e: Exception => logError("Some profiler output not written. Unexpected exception: ", e)
Review Comment:
Got it. Since we have the error log, it might be okay for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org