You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/09/27 23:18:12 UTC
[PR] [SPARK-59798][4.2][K8S] Support `spark.executor.extraLibraryPath` for K8s executors [spark]
dongjoon-hyun opened a new pull request, #59085:
URL: https://github.com/apache/spark/pull/59085
### What changes were proposed in this pull request?
This PR is a backport of #59070 to `branch-4.2`.
This PR aims to support `spark.executor.extraLibraryPath` for K8s executors.
- `BasicExecutorFeatureStep` passes it to the executor container as `SPARK_EXECUTOR_LIBRARY_PATH`.
- `entrypoint.sh` prepends `SPARK_EXECUTOR_LIBRARY_PATH` to `LD_LIBRARY_PATH` before launching the executor JVM.
The previous cherry-pick to `branch-4.2` was reverted because the new unit test failed to compile. Unlike `master` and `branch-4.x`, `BasicExecutorFeatureStep` of `branch-4.2` still takes `SecurityManager` because SPARK-59479 (#58768) is not in `branch-4.2`. So, the new unit test passes `new SecurityManager(baseConf)` like the other test cases of `BasicExecutorFeatureStepSuite`. This is the only difference from the original commit.
### Why are the changes needed?
Currently, only Standalone and YARN apply `spark.executor.extraLibraryPath`, and K8s silently ignores it. Like Standalone and YARN, this PR prepends the paths to the existing `LD_LIBRARY_PATH` so that the value from the image or `spark.executorEnv.LD_LIBRARY_PATH` is preserved.
In addition, `apache/spark-docker` already supports `SPARK_EXECUTOR_LIBRARY_PATH` in `entrypoint.sh.template` via SPARK-59799 (apache/spark-docker#134). Since `add-dockerfiles.sh` generates `entrypoint.sh` of new Spark versions from this template, this PR makes `branch-4.3` consistent with its upcoming official docker images.
### Does this PR introduce _any_ user-facing change?
Yes. `spark.executor.extraLibraryPath` takes effect for K8s executors when the image uses Spark's `entrypoint.sh`. Previously, it was silently ignored.
### How was this patch tested?
Pass the CIs with the newly added unit test and K8s integration test.
In addition, the new unit test and the compilation of K8s integration tests are verified locally.
```
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests 'kubernetes/testOnly *BasicExecutorFeatureStepSuite' 'kubernetes-integration-tests/Test/compile'
...
[info] - SPARK-59798: extra library path gets translated into an environment variable (11 milliseconds)
...
[info] Tests: succeeded 40, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59798][4.2][K8S] Support `spark.executor.extraLibraryPath` for K8s executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #59085: [SPARK-59798][4.2][K8S] Support `spark.executor.extraLibraryPath` for K8s executors
URL: https://github.com/apache/spark/pull/59085
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59798][4.2][K8S] Support `spark.executor.extraLibraryPath` for K8s executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59085:
URL: https://github.com/apache/spark/pull/59085#issuecomment-5861580206
**Merge Summary:**
- merged into branch-4.2 https://github.com/apache/spark/commit/7a6fdcd31d117caf657342ae47e4f9074890ca61
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59798][4.2][K8S] Support `spark.executor.extraLibraryPath` for K8s executors [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59085:
URL: https://github.com/apache/spark/pull/59085#issuecomment-5860830390
Thank you, @HyukjinKwon .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org