You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2023/10/11 04:40:52 UTC
[PR] [SPARK-45497][K8S] Add an symbolic link file `spark-examples.jar` in K8s Docker images [spark]
dongjoon-hyun opened a new pull request, #43324:
URL: https://github.com/apache/spark/pull/43324
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354308076
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
No, I asked this because I always run `ln` command with link source under current path. I don't know that you can run `ln` with a source in different path. Interesting. 😄
If you test it, then it should be okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #43324:
URL: https://github.com/apache/spark/pull/43324#issuecomment-1758040534
Thank you @dongjoon-hyun for clarifying my confusion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354308076
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
No, I asked this because I always run `ln` command with link source under current path. I don't know that you can run `ln` with a source in different path. Interesting.
If you test it, then it should be okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354308076
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
No, I asked this because I always run `ln` command with link source under current path (or it is absolute path). I don't know that you can run `ln` with a source in different path. Interesting. 😄
If you test it, then it should be okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354139948
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
No~ The symbolic link file is created at `jar` directory.
```
$ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars | tail -n6
total 1620
drwxr-xr-x 1 root root 4096 Oct 11 04:37 .
drwxr-xr-x 1 root root 4096 Sep 9 02:08 ..
-rw-r--r-- 1 root root 78803 Sep 9 02:08 scopt_2.12-3.7.1.jar
-rw-r--r-- 1 root root 1564255 Sep 9 02:08 spark-examples_2.12-3.5.0.jar
lrwxrwxrwx 1 root root 29 Oct 11 04:37 spark-examples.jar -> spark-examples_2.12-3.5.0.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #43324:
URL: https://github.com/apache/spark/pull/43324#issuecomment-1757025862
Merged to master for Apache Spark 4.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354221825
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
Hmm, yea, `spark-examples_2.12-3.5.0.jar` is the link source. My question is, does the source exist under the current path of running `ln` command?
> No~ The symbolic link file is created at jar directory.
Does `ln` command run under `jar` directory? I don't see there is command changing to `jar` directory before `ln`.
Do I miss anything here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354275625
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
It seems that we are repeating the same questions. Maybe, are you asking because something is not working, @viirya ? Does it fail in your environment?
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
It seems that we are repeating the same questions and answers. Maybe, are you asking because something is not working, @viirya ? Does it fail in your environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354311845
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
> You can do that in your mac.
Yea, I just tested it locally. It works. 👍
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
> You can do that in your mac.
Yea, I just tested it locally. It works. 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #43324: [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images
URL: https://github.com/apache/spark/pull/43324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354133514
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
This does `ln -s spark-examples_2.12-3.5.0.jar /opt/spark/examples/jars/spark-examples.jar`, but is `spark-examples_2.12-3.5.0.jar` under current path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354244110
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
😄 I understand why you are confused. `ln` command does need to switch directory. You can do that in your mac.
```
$ ls examples/jars
scopt_2.12-3.7.1.jar spark-examples_2.12-3.5.0.jar
$ ln -s spark-examples_2.12-3.5.0.jar examples/jars/spark-examples.jar
$ ls -al examples/jars
total 3216
drwxr-xr-x 5 dongjoon staff 160 Oct 10 23:41 .
drwxr-xr-x 4 dongjoon staff 128 Sep 8 19:08 ..
-rw-r--r-- 1 dongjoon staff 78803 Sep 8 19:08 scopt_2.12-3.7.1.jar
lrwxr-xr-x 1 dongjoon staff 29 Oct 10 23:41 spark-examples.jar -> spark-examples_2.12-3.5.0.jar
-rw-r--r-- 1 dongjoon staff 1564255 Sep 8 19:08 spark-examples_2.12-3.5.0.jar
```
BTW, this is tested in the cluster already, @viirya ~
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
😄 I understand why you are confused. `ln` command doesn't need to switch directory. You can do that in your mac.
```
$ ls examples/jars
scopt_2.12-3.7.1.jar spark-examples_2.12-3.5.0.jar
$ ln -s spark-examples_2.12-3.5.0.jar examples/jars/spark-examples.jar
$ ls -al examples/jars
total 3216
drwxr-xr-x 5 dongjoon staff 160 Oct 10 23:41 .
drwxr-xr-x 4 dongjoon staff 128 Sep 8 19:08 ..
-rw-r--r-- 1 dongjoon staff 78803 Sep 8 19:08 scopt_2.12-3.7.1.jar
lrwxr-xr-x 1 dongjoon staff 29 Oct 10 23:41 spark-examples.jar -> spark-examples_2.12-3.5.0.jar
-rw-r--r-- 1 dongjoon staff 1564255 Sep 8 19:08 spark-examples_2.12-3.5.0.jar
```
BTW, this is tested in the cluster already, @viirya ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354143123
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
It's not important for `ln` command. Only the first argument is used as a link location for the newly generate symbolic file.
> is spark-examples_2.12-3.5.0.jar under current path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354244110
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
😄 I understand why you are confused. `ln` command does need to switch directory. You can do that in your mac.
```
$ ls examples/jars
scopt_2.12-3.7.1.jar spark-examples_2.12-3.5.0.jar
$ ln -s spark-examples_2.12-3.5.0.jar examples/jars/spark-examples.jar
$ ls -al examples/jars
total 3216
drwxr-xr-x 5 dongjoon staff 160 Oct 10 23:41 .
drwxr-xr-x 4 dongjoon staff 128 Sep 8 19:08 ..
-rw-r--r-- 1 dongjoon staff 78803 Sep 8 19:08 scopt_2.12-3.7.1.jar
lrwxr-xr-x 1 dongjoon staff 29 Oct 10 23:41 spark-examples.jar -> spark-examples_2.12-3.5.0.jar
-rw-r--r-- 1 dongjoon staff 1564255 Sep 8 19:08 spark-examples_2.12-3.5.0.jar
```
BTW, this is tested in the cluster already~
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
😄 I understand why you are confused. `ln` command does need to switch directory. You can do that in your mac.
```
$ ls examples/jars
scopt_2.12-3.7.1.jar spark-examples_2.12-3.5.0.jar
$ ln -s spark-examples_2.12-3.5.0.jar examples/jars/spark-examples.jar
$ ls -al examples/jars
total 3216
drwxr-xr-x 5 dongjoon staff 160 Oct 10 23:41 .
drwxr-xr-x 4 dongjoon staff 128 Sep 8 19:08 ..
-rw-r--r-- 1 dongjoon staff 78803 Sep 8 19:08 scopt_2.12-3.7.1.jar
lrwxr-xr-x 1 dongjoon staff 29 Oct 10 23:41 spark-examples.jar -> spark-examples_2.12-3.5.0.jar
-rw-r--r-- 1 dongjoon staff 1564255 Sep 8 19:08 spark-examples_2.12-3.5.0.jar
```
BTW, this is tested in the cluster already~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #43324:
URL: https://github.com/apache/spark/pull/43324#issuecomment-1756819125
Could you review this PR when you have some time, @viirya ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #43324:
URL: https://github.com/apache/spark/pull/43324#issuecomment-1757042884
Late LGTM and looks useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #43324:
URL: https://github.com/apache/spark/pull/43324#discussion_r1354140989
##########
resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile:
##########
@@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
+RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
Review Comment:
The first argument is a link source and the second argument is the location of newly created symbolic link. Since we don't use directory in the symbolic link, this relation is maintained even during copying the whole Spark directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #43324:
URL: https://github.com/apache/spark/pull/43324#issuecomment-1757024883
Thank you so much for your patience, @viirya ! I must be clear about the `ln` usage (which I used) from the beginning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org