(spark-docker) branch master updated: [SPARK-59799] Support `spark.executor.extraLibraryPath` in `entrypoint.sh.template`

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-docker.git


The following commit(s) were added to refs/heads/master by this push:
     new ec08c63  [SPARK-59799] Support `spark.executor.extraLibraryPath` in `entrypoint.sh.template`
ec08c63 is described below

commit ec08c634a9eec5003460d1c948e1fdd6df5ad775
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sat Sep 26 09:29:11 2026 -0700

    [SPARK-59799] Support `spark.executor.extraLibraryPath` in `entrypoint.sh.template`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to support `spark.executor.extraLibraryPath` in `entrypoint.sh.template` like apache/spark#59070.
    
    ### Why are the changes needed?
    
    Since apache/spark#59070, Spark passes `spark.executor.extraLibraryPath` to K8s executors as `SPARK_EXECUTOR_LIBRARY_PATH`. The `entrypoint.sh` should prepend it to `LD_LIBRARY_PATH`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `spark.executor.extraLibraryPath` takes effect for K8s executors in the new images. The existing images are not changed.
    
    ### How was this patch tested?
    
    Manually tested with `apache/spark:4.2.0-scala` and the new `entrypoint.sh`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #134 from dongjoon-hyun/dongjoon/spark-pr-59070-issue-35b60f.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 OVERVIEW.md            | 1 +
 entrypoint.sh.template | 4 ++++
 2 files changed, 5 insertions(+)

diff --git a/OVERVIEW.md b/OVERVIEW.md
index 0465555..69d10b1 100644
--- a/OVERVIEW.md
+++ b/OVERVIEW.md
@@ -70,6 +70,7 @@ The environment variables of entrypoint.sh are listed below:
 | SPARK_DIST_CLASSPATH | Distribution-defined classpath to add to processes |
 | SPARK_DRIVER_BIND_ADDRESS | Hostname or IP address where to bind listening sockets. See also `spark.driver.bindAddress` |
 | SPARK_EXECUTOR_JAVA_OPTS | The Java opts of Spark Executor |
+| SPARK_EXECUTOR_LIBRARY_PATH | The extra library path to be prepended to `LD_LIBRARY_PATH` of Spark Executor. See also `spark.executor.extraLibraryPath` |
 | SPARK_APPLICATION_ID | A unique identifier for the Spark application |
 | SPARK_EXECUTOR_POD_IP | The Pod IP address of spark executor |
 | SPARK_RESOURCE_PROFILE_ID | The resource profile ID |
diff --git a/entrypoint.sh.template b/entrypoint.sh.template
index c576d8f..2680a69 100644
--- a/entrypoint.sh.template
+++ b/entrypoint.sh.template
@@ -103,6 +103,10 @@ case "$1" in
     ;;
   executor)
     shift 1
+    # Prepend spark.executor.extraLibraryPath to the existing library path
+    if [ -n "$SPARK_EXECUTOR_LIBRARY_PATH" ]; then
+      export LD_LIBRARY_PATH="$SPARK_EXECUTOR_LIBRARY_PATH${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
+    fi
     CMD=(
       ${JAVA_HOME}/bin/java
       "${SPARK_EXECUTOR_JAVA_OPTS[@]}"


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org