You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/10/11 07:35:50 UTC

[spark] branch master updated: [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 64ac59d7129 [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images
64ac59d7129 is described below

commit 64ac59d71296e631ded97b332660d3db5675623a
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Wed Oct 11 00:35:38 2023 -0700

    [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images
    
    ### What changes were proposed in this pull request?
    
    This PR aims to add a symbolic link file, `spark-examples.jar`, in the example jar directory.
    
    ```
    $ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars  | tail -n6
    total 1620
    drwxr-xr-x 1 root root    4096 Oct 11 04:37 .
    drwxr-xr-x 1 root root    4096 Sep  9 02:08 ..
    -rw-r--r-- 1 root root   78803 Sep  9 02:08 scopt_2.12-3.7.1.jar
    -rw-r--r-- 1 root root 1564255 Sep  9 02:08 spark-examples_2.12-3.5.0.jar
    lrwxrwxrwx 1 root root      29 Oct 11 04:37 spark-examples.jar -> spark-examples_2.12-3.5.0.jar
    ```
    
    ### Why are the changes needed?
    
    Like PySpark example (`pi.py`), we can submit the examples without considering the version numbers which was painful before.
    ```
    bin/spark-submit \
    --master k8s://$K8S_MASTER \
    --deploy-mode cluster \
    ...
    --class org.apache.spark.examples.SparkPi \
    local:///opt/spark/examples/jars/spark-examples.jar 10000
    ```
    
    The following is the driver pod log.
    ```
    + exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit ...
    --deploy-mode client
    --properties-file /opt/spark/conf/spark.properties
    --class org.apache.spark.examples.SparkPi
    local:///opt/spark/examples/jars/spark-examples.jar 10000
    Files  local:///opt/spark/examples/jars/spark-examples.jar from /opt/spark/examples/jars/spark-examples.jar to /opt/spark/work-dir/./spark-examples.jar
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, this is an additional file.
    
    ### How was this patch tested?
    
    Manually build the docker image and do `ls`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #43324 from dongjoon-hyun/SPARK-45497.
    
    Authored-by: Dongjoon Hyun <dh...@apple.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../kubernetes/docker/src/main/dockerfiles/spark/Dockerfile              | 1 +
 1 file changed, 1 insertion(+)

diff --git a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
index 02559b8de2d..b80e72c768c 100644
--- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
+++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
 COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
 COPY kubernetes/dockerfiles/spark/decom.sh /opt/
 COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
 COPY kubernetes/tests /opt/spark/tests
 COPY data /opt/spark/data
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org