(spark) branch master updated: [SPARK-59412][CORE] Support short class names of built-in plugins in `spark.plugins`

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 651556eb4ab9 [SPARK-59412][CORE] Support short class names of built-in plugins in `spark.plugins`
651556eb4ab9 is described below

commit 651556eb4ab9dea20af1e722e420de4dd839244a
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Sep 10 12:31:25 2026 -0700

    [SPARK-59412][CORE] Support short class names of built-in plugins in `spark.plugins`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to support short class names of built-in plugins in `spark.plugins` and `spark.plugins.defaultList`.
    
    | Short class name | Fully-qualified class name |
    | --- | --- |
    | `DriverTimeoutPlugin` | `org.apache.spark.deploy.DriverTimeoutPlugin` |
    | `RedirectConsolePlugin` | `org.apache.spark.deploy.RedirectConsolePlugin` |
    | `ProfilerPlugin` | `org.apache.spark.profiler.ProfilerPlugin` |
    | `ExecutorPVCResizePlugin` | `org.apache.spark.scheduler.cluster.k8s.ExecutorPVCResizePlugin` |
    | `ExecutorResizePlugin` | `org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin` |
    | `ExecutorRollPlugin` | `org.apache.spark.scheduler.cluster.k8s.ExecutorRollPlugin` |
    | `SparkConnectPlugin` | `org.apache.spark.sql.connect.SparkConnectPlugin` |
    
    Each item of the `spark.plugins` config entry is resolved via `transform` after `spark.plugins.defaultList` is prepended.
    - A short class name is matched exactly (case-sensitive) and converted to its fully-qualified class name.
    - All other values are passed through unchanged.
    - Since `PluginContainer` removes duplicates after the resolution, a plugin is loaded only once even if both its short and fully-qualified class names are given.
    
    ### Why are the changes needed?
    
    To improve the usability of built-in plugins. Currently, users must specify long fully-qualified class names.
    
    ```
    --conf spark.plugins=org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin,org.apache.spark.scheduler.cluster.k8s.ExecutorPVCResizePlugin
    ```
    
    After this PR, users can use the following instead.
    
    ```
    --conf spark.plugins=ExecutorResizePlugin,ExecutorPVCResizePlugin
    ```
    
    This is consistent with other Spark configurations which accept short names for built-in implementations, such as `spark.shuffle.manager` (`sort`) and `spark.io.compression.codec` (`zstd`).
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, there is no behavior change for existing configurations with fully-qualified class names.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case in `PluginContainerSuite`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5
    
    Closes #58711 from dongjoon-hyun/SPARK-59412.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../scala/org/apache/spark/internal/config/package.scala | 16 +++++++++++++++-
 .../spark/internal/plugin/PluginContainerSuite.scala     | 15 +++++++++++++++
 2 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 64a05fece570..87373d777d93 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -2162,13 +2162,27 @@ package object config {
 
   private[spark] val DEFAULT_PLUGINS_LIST = "spark.plugins.defaultList"
 
+  // A map from the short class names of built-in plugins to their fully-qualified class names.
+  private val BUILTIN_PLUGINS: Map[String, String] = Seq(
+    "org.apache.spark.deploy.DriverTimeoutPlugin",
+    "org.apache.spark.deploy.RedirectConsolePlugin",
+    "org.apache.spark.profiler.ProfilerPlugin",
+    "org.apache.spark.scheduler.cluster.k8s.ExecutorPVCResizePlugin",
+    "org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin",
+    "org.apache.spark.scheduler.cluster.k8s.ExecutorRollPlugin",
+    "org.apache.spark.sql.connect.SparkConnectPlugin"
+  ).map(name => name.substring(name.lastIndexOf('.') + 1) -> name).toMap
+
   private[spark] val PLUGINS =
     ConfigBuilder("spark.plugins")
       .withPrepended(DEFAULT_PLUGINS_LIST, separator = ",")
       .doc("Comma-separated list of class names implementing " +
-        "org.apache.spark.api.plugin.SparkPlugin to load into the application.")
+        "org.apache.spark.api.plugin.SparkPlugin to load into the application. " +
+        "Built-in plugins can also be specified by their short class names, " +
+        "e.g. `DriverTimeoutPlugin`.")
       .version("3.0.0")
       .stringConf
+      .transform(name => BUILTIN_PLUGINS.getOrElse(name, name))
       .toSequence
       .createWithDefault(Nil)
 
diff --git a/core/src/test/scala/org/apache/spark/internal/plugin/PluginContainerSuite.scala b/core/src/test/scala/org/apache/spark/internal/plugin/PluginContainerSuite.scala
index bc6a9ec21333..99165585aefa 100644
--- a/core/src/test/scala/org/apache/spark/internal/plugin/PluginContainerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/internal/plugin/PluginContainerSuite.scala
@@ -34,6 +34,7 @@ import org.scalatest.concurrent.Eventually.{eventually, interval, timeout}
 import org.apache.spark._
 import org.apache.spark.TestUtils._
 import org.apache.spark.api.plugin._
+import org.apache.spark.deploy.{DriverTimeoutPlugin, RedirectConsolePlugin}
 import org.apache.spark.internal.config._
 import org.apache.spark.launcher.SparkLauncher
 import org.apache.spark.memory.MemoryMode
@@ -132,6 +133,20 @@ class PluginContainerSuite extends SparkFunSuite with LocalSparkContext {
     assert(TestSparkPlugin.driverPlugin != null)
   }
 
+  test("SPARK-59412: short class names of built-in plugins") {
+    val conf = new SparkConf()
+      .set(DEFAULT_PLUGINS_LIST, "RedirectConsolePlugin")
+      .set(PLUGINS.key, "DriverTimeoutPlugin,org.apache.spark.deploy.DriverTimeoutPlugin," +
+        s"drivertimeoutplugin,${classOf[TestSparkPlugin].getName()}")
+
+    assert(conf.get(PLUGINS) === Seq(
+      classOf[RedirectConsolePlugin].getName(),
+      classOf[DriverTimeoutPlugin].getName(),
+      classOf[DriverTimeoutPlugin].getName(),
+      "drivertimeoutplugin",
+      classOf[TestSparkPlugin].getName()))
+  }
+
   test("SPARK-33088: executor tasks trigger plugin calls") {
     val conf = new SparkConf()
       .setAppName(getClass().getName())


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org