You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yikun (via GitHub)" <gi...@apache.org> on 2023/07/27 00:43:30 UTC
[GitHub] [spark-docker] Yikun commented on a diff in pull request #52: Add Support for Scala 2.13 in Spark 3.4.1
Yikun commented on code in PR #52:
URL: https://github.com/apache/spark-docker/pull/52#discussion_r1274663861
##########
3.4.1/scala2.12-java11-ubuntu/Dockerfile:
##########
@@ -36,15 +36,16 @@ RUN set -ex; \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.4.1/spark-3.4.1-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.4.1/spark-3.4.1-bin-hadoop3.tgz.asc \
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.4.1/spark-3.4.1-bin- \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.4.1/spark-3.4.1-bin- \
GPG_KEY=F28C9C925C188C35E345614DEDA00CE834F0FC5C
RUN set -ex; \
+ if [ "2.12" = "2.13" ]; then export BIN_FILE_SUFFIX="hadoop3-scala2.13.tgz"; else export BIN_FILE_SUFFIX="hadoop3.tgz"; fi; \
Review Comment:
As I mentioned in `Dockerfile.template`, if we address suffix concat in template.py then this dockerfile will not be changed.
##########
versions.json:
##########
@@ -22,8 +22,16 @@
"path": "3.4.1/scala2.12-java11-ubuntu",
"tags": [
"3.4.1-scala2.12-java11-ubuntu",
- "3.4.1-scala",
- "scala"
+ "3.4.1-scala2.12",
+ "scala2.12"
Review Comment:
```suggestion
"scala2.12",
"3.4.1-scala",
"scala"
```
I believe the `scala`, `3.4.1-scala` tag should also be kept.
##########
add-dockerfiles.sh:
##########
@@ -33,6 +33,7 @@ scala2.12-java11-python3-r-ubuntu
scala2.12-java11-python3-ubuntu
scala2.12-java11-r-ubuntu
scala2.12-java11-ubuntu
+scala2.13-java11-ubuntu
Review Comment:
You only add the scala one, do you think python3/r/all should also be added in sometime? (Yep, just a question, IMO we can add scala2.13 now, and add others on demand in future)
##########
Dockerfile.template:
##########
@@ -36,15 +36,16 @@ RUN set -ex; \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-{{ SPARK_VERSION }}/spark-{{ SPARK_VERSION }}-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-{{ SPARK_VERSION }}/spark-{{ SPARK_VERSION }}-bin-hadoop3.tgz.asc \
Review Comment:
```
ENV SPARK_TGZ_URL={{ SPARK_TGZ_URL }} \
SPARK_TGZ_ASC_URL= {{ SPARK_TGZ_ASC_URL }} \
```
Could we only change the template in here, and address suffix and prefix in https://github.com/apache/spark-docker/blob/master/tools/template.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org