You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by yi...@apache.org on 2022/11/01 03:14:16 UTC
[spark-docker] branch master updated: [SPARK-40969] Replace spark TGZ url with apache archive url
This is an automated email from the ASF dual-hosted git repository.
yikun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-docker.git
The following commit(s) were added to refs/heads/master by this push:
new 243ce20 [SPARK-40969] Replace spark TGZ url with apache archive url
243ce20 is described below
commit 243ce201296c20ae48b32a87d254800e8ad197ef
Author: Qian.Sun <qi...@gmail.com>
AuthorDate: Tue Nov 1 11:14:04 2022 +0800
[SPARK-40969] Replace spark TGZ url with apache archive url
### What changes were proposed in this pull request?
This PR aims to replace spark TGZ url with apache archive url.
### Why are the changes needed?
```
#13 [linux/amd64 4/9] RUN set -ex; export SPARK_TMP="$(mktemp -d)"; cd $SPARK_TMP; wget -nv -O spark.tgz "https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz"; wget -nv -O spark.tgz.asc "https://downloads.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz.asc"; export GNUPGHOME="$(mktemp -d)"; gpg --keyserver hkps://keys.openpgp.org --recv-key "80FB8EBE8EBA68504989703491B5DC815DBF10D3" || gpg --keyserver hkps://keyserver.ubuntu.com [...]
#0 0.132 ++ mktemp -d
#0 0.133 + export SPARK_TMP=/tmp/tmp.oEdW8CyP9h
#0 0.133 + SPARK_TMP=/tmp/tmp.oEdW8CyP9h
#0 0.133 + cd /tmp/tmp.oEdW8CyP9h
#0 0.133 + wget -nv -O spark.tgz https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz
#0 0.152 https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz:
#0 0.152 2022-10-31 04:06:44 ERROR 404: Not Found.
#13 ERROR: process "/bin/sh -c set -ex; export SPARK_TMP=\"$(mktemp -d)\"; cd $SPARK_TMP; wget -nv -O spark.tgz \"$SPARK_TGZ_URL\"; wget -nv -O spark.tgz.asc \"$SPARK_TGZ_ASC_URL\"; export GNUPGHOME=\"$(mktemp -d)\"; gpg --keyserver hkps://keys.openpgp.org --recv-key \"$GPG_KEY\" || gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys \"$GPG_KEY\"; gpg --batch --verify spark.tgz.asc spark.tgz; gpgconf --kill all; rm -rf \"$GNUPGHOME\" spark.t [...]
```
Old url `https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz` is not found. Better to use unity apache archive url.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No need to add new tests.
Closes #22 from dcoliversun/SPARK-40969.
Authored-by: Qian.Sun <qi...@gmail.com>
Signed-off-by: Yikun Jiang <yi...@gmail.com>
---
3.3.0/scala2.12-java11-python3-r-ubuntu/Dockerfile | 4 ++--
3.3.0/scala2.12-java11-python3-ubuntu/Dockerfile | 4 ++--
3.3.0/scala2.12-java11-r-ubuntu/Dockerfile | 4 ++--
3.3.0/scala2.12-java11-ubuntu/Dockerfile | 4 ++--
3.3.1/scala2.12-java11-python3-r-ubuntu/Dockerfile | 4 ++--
3.3.1/scala2.12-java11-python3-ubuntu/Dockerfile | 4 ++--
3.3.1/scala2.12-java11-r-ubuntu/Dockerfile | 4 ++--
3.3.1/scala2.12-java11-ubuntu/Dockerfile | 4 ++--
Dockerfile.template | 4 ++--
9 files changed, 18 insertions(+), 18 deletions(-)
diff --git a/3.3.0/scala2.12-java11-python3-r-ubuntu/Dockerfile b/3.3.0/scala2.12-java11-python3-r-ubuntu/Dockerfile
index 8c2761e..fb48b80 100644
--- a/3.3.0/scala2.12-java11-python3-r-ubuntu/Dockerfile
+++ b/3.3.0/scala2.12-java11-python3-r-ubuntu/Dockerfile
@@ -42,8 +42,8 @@ RUN set -ex && \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://downloads.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz.asc \
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz.asc \
GPG_KEY=80FB8EBE8EBA68504989703491B5DC815DBF10D3
RUN set -ex; \
diff --git a/3.3.0/scala2.12-java11-python3-ubuntu/Dockerfile b/3.3.0/scala2.12-java11-python3-ubuntu/Dockerfile
index 6a0017a..1b6a02c 100644
--- a/3.3.0/scala2.12-java11-python3-ubuntu/Dockerfile
+++ b/3.3.0/scala2.12-java11-python3-ubuntu/Dockerfile
@@ -41,8 +41,8 @@ RUN set -ex && \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://downloads.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz.asc \
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz.asc \
GPG_KEY=80FB8EBE8EBA68504989703491B5DC815DBF10D3
RUN set -ex; \
diff --git a/3.3.0/scala2.12-java11-r-ubuntu/Dockerfile b/3.3.0/scala2.12-java11-r-ubuntu/Dockerfile
index 975e444..0b49aef 100644
--- a/3.3.0/scala2.12-java11-r-ubuntu/Dockerfile
+++ b/3.3.0/scala2.12-java11-r-ubuntu/Dockerfile
@@ -40,8 +40,8 @@ RUN set -ex && \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://downloads.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz.asc \
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz.asc \
GPG_KEY=80FB8EBE8EBA68504989703491B5DC815DBF10D3
RUN set -ex; \
diff --git a/3.3.0/scala2.12-java11-ubuntu/Dockerfile b/3.3.0/scala2.12-java11-ubuntu/Dockerfile
index c38e5dc..cbdf6c5 100644
--- a/3.3.0/scala2.12-java11-ubuntu/Dockerfile
+++ b/3.3.0/scala2.12-java11-ubuntu/Dockerfile
@@ -39,8 +39,8 @@ RUN set -ex && \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://downloads.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz.asc \
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz.asc \
GPG_KEY=80FB8EBE8EBA68504989703491B5DC815DBF10D3
RUN set -ex; \
diff --git a/3.3.1/scala2.12-java11-python3-r-ubuntu/Dockerfile b/3.3.1/scala2.12-java11-python3-r-ubuntu/Dockerfile
index b126f1f..def11ff 100644
--- a/3.3.1/scala2.12-java11-python3-r-ubuntu/Dockerfile
+++ b/3.3.1/scala2.12-java11-python3-r-ubuntu/Dockerfile
@@ -42,8 +42,8 @@ RUN set -ex && \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://dlcdn.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://downloads.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz.asc \
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz.asc \
GPG_KEY=86727D43E73A415F67A0B1A14E68B3E6CD473653
RUN set -ex; \
diff --git a/3.3.1/scala2.12-java11-python3-ubuntu/Dockerfile b/3.3.1/scala2.12-java11-python3-ubuntu/Dockerfile
index af02e65..3da8521 100644
--- a/3.3.1/scala2.12-java11-python3-ubuntu/Dockerfile
+++ b/3.3.1/scala2.12-java11-python3-ubuntu/Dockerfile
@@ -41,8 +41,8 @@ RUN set -ex && \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://dlcdn.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://downloads.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz.asc \
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz.asc \
GPG_KEY=86727D43E73A415F67A0B1A14E68B3E6CD473653
RUN set -ex; \
diff --git a/3.3.1/scala2.12-java11-r-ubuntu/Dockerfile b/3.3.1/scala2.12-java11-r-ubuntu/Dockerfile
index bc2dcf7..04ab3ea 100644
--- a/3.3.1/scala2.12-java11-r-ubuntu/Dockerfile
+++ b/3.3.1/scala2.12-java11-r-ubuntu/Dockerfile
@@ -40,8 +40,8 @@ RUN set -ex && \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://dlcdn.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://downloads.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz.asc \
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz.asc \
GPG_KEY=86727D43E73A415F67A0B1A14E68B3E6CD473653
RUN set -ex; \
diff --git a/3.3.1/scala2.12-java11-ubuntu/Dockerfile b/3.3.1/scala2.12-java11-ubuntu/Dockerfile
index c732406..8b342c4 100644
--- a/3.3.1/scala2.12-java11-ubuntu/Dockerfile
+++ b/3.3.1/scala2.12-java11-ubuntu/Dockerfile
@@ -39,8 +39,8 @@ RUN set -ex && \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://dlcdn.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://downloads.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz.asc \
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz.asc \
GPG_KEY=86727D43E73A415F67A0B1A14E68B3E6CD473653
RUN set -ex; \
diff --git a/Dockerfile.template b/Dockerfile.template
index a220247..4819cb2 100644
--- a/Dockerfile.template
+++ b/Dockerfile.template
@@ -48,8 +48,8 @@ RUN set -ex && \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://dlcdn.apache.org/spark/spark-{{ SPARK_VERSION }}/spark-{{ SPARK_VERSION }}-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://downloads.apache.org/spark/spark-{{ SPARK_VERSION }}/spark-{{ SPARK_VERSION }}-bin-hadoop3.tgz.asc \
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-{{ SPARK_VERSION }}/spark-{{ SPARK_VERSION }}-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-{{ SPARK_VERSION }}/spark-{{ SPARK_VERSION }}-bin-hadoop3.tgz.asc \
GPG_KEY={{ SPARK_GPG_KEY }}
RUN set -ex; \
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org