You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/04/20 00:01:31 UTC
(spark) branch master updated: [SPARK-56525][INFRA] Add ubuntu mirror repositories for redundancy
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 38d5dc4c576d [SPARK-56525][INFRA] Add ubuntu mirror repositories for redundancy
38d5dc4c576d is described below
commit 38d5dc4c576d1e3a28272972871c1a731dfc1777
Author: Kousuke Saruta <sa...@apache.org>
AuthorDate: Sun Apr 19 17:01:21 2026 -0700
[SPARK-56525][INFRA] Add ubuntu mirror repositories for redundancy
### What changes were proposed in this pull request?
This PR adds ubuntu mirror repositories to Dockerfiles for redundancy.
`mirrors.edge.kernel.org` seems sufficiently public and reliable.
### Why are the changes needed?
For CI stability.
Sometimes, ubuntu repositories including `archive.ubuntu.com` and `security.ubuntu.com` are very slow, causing CI failures.
https://github.com/apache/spark/actions/runs/24542208092/job/71750264055
https://github.com/apache/spark/actions/runs/24544185426/job/71756248072
https://github.com/sarutak/spark/actions/runs/24552461498/job/71781474186
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
GA.
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Opus 4.6.
Closes #55389 from sarutak/add-ubuntu-mirror.
Authored-by: Kousuke Saruta <sa...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
dev/infra/Dockerfile | 4 ++++
dev/spark-test-image/connect-gen-protos/Dockerfile | 2 ++
dev/spark-test-image/docs/Dockerfile | 2 ++
dev/spark-test-image/lint/Dockerfile | 2 ++
dev/spark-test-image/pypy-310/Dockerfile | 2 ++
dev/spark-test-image/python-310/Dockerfile | 2 ++
dev/spark-test-image/python-311/Dockerfile | 2 ++
dev/spark-test-image/python-312-classic-only/Dockerfile | 2 ++
dev/spark-test-image/python-312-pandas-3/Dockerfile | 2 ++
dev/spark-test-image/python-312/Dockerfile | 2 ++
dev/spark-test-image/python-313/Dockerfile | 2 ++
dev/spark-test-image/python-314-nogil/Dockerfile | 2 ++
dev/spark-test-image/python-314/Dockerfile | 2 ++
dev/spark-test-image/python-minimum/Dockerfile | 2 ++
dev/spark-test-image/python-ps-minimum/Dockerfile | 2 ++
dev/spark-test-image/sparkr/Dockerfile | 2 ++
16 files changed, 34 insertions(+)
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index 2f6a676d8177..0f771e4e93aa 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -29,6 +29,10 @@ ENV FULL_REFRESH_DATE=20260317
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN echo 'deb https://mirrors.edge.kernel.org/ubuntu jammy main restricted universe multiverse' > /etc/apt/sources.list.d/mirror.list && \
+ echo 'deb https://mirrors.edge.kernel.org/ubuntu jammy-updates main restricted universe multiverse' >> /etc/apt/sources.list.d/mirror.list && \
+ echo 'deb https://mirrors.edge.kernel.org/ubuntu jammy-security main restricted universe multiverse' >> /etc/apt/sources.list.d/mirror.list
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/connect-gen-protos/Dockerfile b/dev/spark-test-image/connect-gen-protos/Dockerfile
index ddd550082769..13ff7682df29 100644
--- a/dev/spark-test-image/connect-gen-protos/Dockerfile
+++ b/dev/spark-test-image/connect-gen-protos/Dockerfile
@@ -29,6 +29,8 @@ ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
ENV OPENSSL_FORCE_FIPS_MODE=0
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
ca-certificates \
curl \
diff --git a/dev/spark-test-image/docs/Dockerfile b/dev/spark-test-image/docs/Dockerfile
index 3b7306c417cf..d0d674a9e74c 100644
--- a/dev/spark-test-image/docs/Dockerfile
+++ b/dev/spark-test-image/docs/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260316
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/lint/Dockerfile b/dev/spark-test-image/lint/Dockerfile
index 799656614d82..9bbe108071bf 100644
--- a/dev/spark-test-image/lint/Dockerfile
+++ b/dev/spark-test-image/lint/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260316
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/pypy-310/Dockerfile b/dev/spark-test-image/pypy-310/Dockerfile
index f76b51498299..67b2c2c17de9 100644
--- a/dev/spark-test-image/pypy-310/Dockerfile
+++ b/dev/spark-test-image/pypy-310/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260207
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/python-310/Dockerfile b/dev/spark-test-image/python-310/Dockerfile
index c8752c8bb340..326a5008ed5b 100644
--- a/dev/spark-test-image/python-310/Dockerfile
+++ b/dev/spark-test-image/python-310/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260210
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/python-311/Dockerfile b/dev/spark-test-image/python-311/Dockerfile
index 286c03556868..8e5044aeb954 100644
--- a/dev/spark-test-image/python-311/Dockerfile
+++ b/dev/spark-test-image/python-311/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260210
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/python-312-classic-only/Dockerfile b/dev/spark-test-image/python-312-classic-only/Dockerfile
index ba7d34625cc8..da2b99cd7838 100644
--- a/dev/spark-test-image/python-312-classic-only/Dockerfile
+++ b/dev/spark-test-image/python-312-classic-only/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260210
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/python-312-pandas-3/Dockerfile b/dev/spark-test-image/python-312-pandas-3/Dockerfile
index 6b9844372b46..c54a8c284a6b 100644
--- a/dev/spark-test-image/python-312-pandas-3/Dockerfile
+++ b/dev/spark-test-image/python-312-pandas-3/Dockerfile
@@ -32,6 +32,8 @@ ENV FULL_REFRESH_DATE=20260210
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/python-312/Dockerfile b/dev/spark-test-image/python-312/Dockerfile
index 9e7f2bd8396f..8eee9e414f1d 100644
--- a/dev/spark-test-image/python-312/Dockerfile
+++ b/dev/spark-test-image/python-312/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260210
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/python-313/Dockerfile b/dev/spark-test-image/python-313/Dockerfile
index 544becb59814..6cfdd2d5a86e 100644
--- a/dev/spark-test-image/python-313/Dockerfile
+++ b/dev/spark-test-image/python-313/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260210
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/python-314-nogil/Dockerfile b/dev/spark-test-image/python-314-nogil/Dockerfile
index 3cc92f9e63f0..edfea3172992 100644
--- a/dev/spark-test-image/python-314-nogil/Dockerfile
+++ b/dev/spark-test-image/python-314-nogil/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260210
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/python-314/Dockerfile b/dev/spark-test-image/python-314/Dockerfile
index d8d7aa897961..b6bc76c6b93b 100644
--- a/dev/spark-test-image/python-314/Dockerfile
+++ b/dev/spark-test-image/python-314/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260210
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
diff --git a/dev/spark-test-image/python-minimum/Dockerfile b/dev/spark-test-image/python-minimum/Dockerfile
index b9aee4dee057..72abd5f5811b 100644
--- a/dev/spark-test-image/python-minimum/Dockerfile
+++ b/dev/spark-test-image/python-minimum/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260210
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
# Should keep the installation consistent with https://apache.github.io/spark/api/python/getting_started/install.html
RUN apt-get update && apt-get install -y \
build-essential \
diff --git a/dev/spark-test-image/python-ps-minimum/Dockerfile b/dev/spark-test-image/python-ps-minimum/Dockerfile
index 44f1d6bed264..0f970202edd5 100644
--- a/dev/spark-test-image/python-ps-minimum/Dockerfile
+++ b/dev/spark-test-image/python-ps-minimum/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260210
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
# Should keep the installation consistent with https://apache.github.io/spark/api/python/getting_started/install.html
RUN apt-get update && apt-get install -y \
build-essential \
diff --git a/dev/spark-test-image/sparkr/Dockerfile b/dev/spark-test-image/sparkr/Dockerfile
index fcb2e090c00c..cc7c47c54d58 100644
--- a/dev/spark-test-image/sparkr/Dockerfile
+++ b/dev/spark-test-image/sparkr/Dockerfile
@@ -29,6 +29,8 @@ ENV FULL_REFRESH_DATE=20260316
ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN=true
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
RUN apt-get update && apt-get install -y \
build-essential \
ca-certificates \
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org