You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/10/19 17:39:56 UTC

[spark] branch master updated: [SPARK-40833][K8S] Cleanup apt lists cache

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 3ed2732740a [SPARK-40833][K8S] Cleanup apt lists cache
3ed2732740a is described below

commit 3ed2732740a2657fe4771e8818ad8c9ff30554c3
Author: Yikun Jiang <yi...@gmail.com>
AuthorDate: Wed Oct 19 10:39:41 2022 -0700

    [SPARK-40833][K8S] Cleanup apt lists cache
    
    ### What changes were proposed in this pull request?
    Remove unused apt lists cache
    
    ### Why are the changes needed?
    Clean cache to reduce docker image size.
    
    This is also [recommanded](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#run) by docker community:
    
    ```
    $ docker run --user 0:0 -ti apache/spark bash
    root5d1ca347279e:/opt/spark/work-dir# ls /var/lib/apt/lists/
    auxfiles                                                                     lock
    deb.debian.org_debian_dists_bullseye-updates_InRelease                       partial
    deb.debian.org_debian_dists_bullseye-updates_main_binary-arm64_Packages.lz4  security.debian.org_debian-security_dists_bullseye-security_InRelease
    deb.debian.org_debian_dists_bullseye_InRelease                               security.debian.org_debian-security_dists_bullseye-security_main_binary-arm64_Packages.lz4
    deb.debian.org_debian_dists_bullseye_main_binary-arm64_Packages.lz4
    root5d1ca347279e:/opt/spark/work-dir# du --max-depth=1 -h /var/lib/apt/lists/
    4.0K    /var/lib/apt/lists/partial
    4.0K    /var/lib/apt/lists/auxfiles
    17M     /var/lib/apt/lists/
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    Yes in some level, image size is reduced.
    
    ### How was this patch tested?
    K8s CI passed
    
    Closes #38298 from Yikun/SPARK-40513.
    
    Authored-by: Yikun Jiang <yi...@gmail.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../kubernetes/docker/src/main/dockerfiles/spark/Dockerfile             | 2 +-
 .../kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17      | 2 +-
 .../kubernetes/docker/src/main/dockerfiles/spark/bindings/R/Dockerfile  | 2 +-
 .../docker/src/main/dockerfiles/spark/bindings/python/Dockerfile        | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
index fc529afb1be..30338b6f91c 100644
--- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
+++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
@@ -39,7 +39,7 @@ RUN set -ex && \
     ln -sv /bin/bash /bin/sh && \
     echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \
     chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
-    rm -rf /var/cache/apt/*
+    rm -rf /var/cache/apt/* && rm -rf /var/lib/apt/lists/*
 
 COPY jars /opt/spark/jars
 COPY bin /opt/spark/bin
diff --git a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17
index 5a360682071..194242996ca 100644
--- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17
+++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17
@@ -39,7 +39,7 @@ RUN set -ex && \
     ln -sv /bin/bash /bin/sh && \
     echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \
     chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
-    rm -rf /var/cache/apt/*
+    rm -rf /var/cache/apt/* && rm -rf /var/lib/apt/lists/*
 
 COPY jars /opt/spark/jars
 COPY bin /opt/spark/bin
diff --git a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/R/Dockerfile b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/R/Dockerfile
index 03e42106230..e4d62cf45f5 100644
--- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/R/Dockerfile
+++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/R/Dockerfile
@@ -29,7 +29,7 @@ RUN mkdir ${SPARK_HOME}/R
 RUN \
   apt-get update && \
   apt install -y r-base r-base-dev && \
-  rm -rf /var/cache/apt/*
+  rm -rf /var/cache/apt/* && rm -rf /var/lib/apt/lists/*
 
 COPY R ${SPARK_HOME}/R
 ENV R_HOME /usr/lib/R
diff --git a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile
index 85d92e39ed2..740aa7f0c43 100644
--- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile
+++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile
@@ -28,7 +28,7 @@ RUN apt-get update && \
     apt install -y python3 python3-pip && \
     pip3 install --upgrade pip setuptools && \
     # Removed the .cache to save space
-    rm -rf /root/.cache && rm -rf /var/cache/apt/*
+    rm -rf /root/.cache && rm -rf /var/cache/apt/* && rm -rf /var/lib/apt/lists/*
 
 COPY python/pyspark ${SPARK_HOME}/python/pyspark
 COPY python/lib ${SPARK_HOME}/python/lib


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org