You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/06/25 19:40:44 UTC

(spark) branch master updated: [SPARK-57700][K8S] Support opt-in `jemalloc` in Spark Docker images

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new d1bcb02a73a1 [SPARK-57700][K8S] Support opt-in `jemalloc` in Spark Docker images
d1bcb02a73a1 is described below

commit d1bcb02a73a1d354b8555fda57827f1dc71f7ae2
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Jun 25 12:40:32 2026 -0700

    [SPARK-57700][K8S] Support opt-in `jemalloc` in Spark Docker images
    
    ### What changes were proposed in this pull request?
    
    This PR aims to install `libjemalloc2` in the Spark Docker image while the default allocator is unchanged — the library is not preloaded unless a user opts in:
    
    ```
    spark.executorEnv.LD_PRELOAD=libjemalloc.so.2
    spark.kubernetes.driverEnv.LD_PRELOAD=libjemalloc.so.2
    ```
    
    ### Why are the changes needed?
    
    [jemalloc](http://jemalloc.net) reduces native/off-heap memory fragmentation, especially `off-heap`, `PySpark`, `Apache Arrow`, and `Spark-accellerator`  use-cases. Shipping the package lets operators enable it with one env var instead of building a custom image, with no regression for anyone else.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No behavior change because the default allocator is unchanged; the image just additionally ships `libjemalloc2`.
    
    ```
    $ ls -al /usr/lib/aarch64-linux-gnu/libjemalloc.so.2
    -rw-r--r-- 1 root root 527176 Apr 18  2024 /usr/lib/aarch64-linux-gnu/libjemalloc.so.2
    ```
    
    ### How was this patch tested?
    
    Pass the CIs.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 4.8
    
    Closes #56782 from dongjoon-hyun/SPARK-57700.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../kubernetes/docker/src/main/dockerfiles/spark/Dockerfile             | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
index f7c7908aeab9..c7ec49cebbc1 100644
--- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
+++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
@@ -36,7 +36,7 @@ ARG spark_uid=185
 RUN set -ex && \
     apt-get update && \
     ln -s /lib /lib64 && \
-    apt install -y --no-install-recommends bash tini libc6 libpam-modules krb5-user libnss3 procps net-tools logrotate libssl-dev && \
+    apt install -y --no-install-recommends bash tini libc6 libjemalloc2 libpam-modules krb5-user libnss3 procps net-tools logrotate libssl-dev && \
     mkdir -p /opt/spark && \
     mkdir -p /opt/spark/examples && \
     mkdir -p /opt/spark/work-dir && \


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org