You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/04 19:19:31 UTC
(spark-connect-swift) branch main updated: [SPARK-59264] Cache Apache Spark distributions in GitHub Actions jobs
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 80e86b6 [SPARK-59264] Cache Apache Spark distributions in GitHub Actions jobs
80e86b6 is described below
commit 80e86b661a59babb69b999bc8ec9068851a5d276
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Fri Sep 4 12:19:24 2026 -0700
[SPARK-59264] Cache Apache Spark distributions in GitHub Actions jobs
### What changes were proposed in this pull request?
This PR adds a `setup-spark` composite action that caches the downloaded Apache Spark distribution tarball, and switches all seven macOS integration test jobs to use it.
- New `.github/actions/setup-spark/action.yml`:
- `actions/cachev4` on `~/spark-dist/spark-<version>-bin-hadoop3.tgz`, keyed by version and SHA-512, so a release candidate and its final release never share a cache entry.
- Downloads from the ASF mirror redirector only on a cache miss. A `url` input overrides it for release candidates.
- Verifies SHA-512 before extracting into `/tmp/spark`.
- `.github/workflows/build_and_test.yml`: the duplicated `curl`/`tar`/`mv` preamble in seven jobs is replaced by a single `uses: ./.github/actions/setup-spark` step.
- SHA-512 verification is now applied to all versions. The values for 4.0.4 and 4.1.3 were taken from the official ASF `.sha512` files:
- [spark-4.0.4-bin-hadoop3.tgz.sha512](https://downloads.apache.org/spark/spark-4.0.4/spark-4.0.4-bin-hadoop3.tgz.sha512)
- [spark-4.1.3-bin-hadoop3.tgz.sha512](https://downloads.apache.org/spark/spark-4.1.3/spark-4.1.3-bin-hadoop3.tgz.sha512)
- `tar xvfz` is changed to `tar xfz` to remove thousands of lines of file listings from the CI logs.
### Why are the changes needed?
Seven macOS jobs each download a Spark distribution from an ASF mirror on every workflow run, roughly 2.8GB of mirror traffic per run:
| Version | Jobs |
| ------- | ---- |
| 4.0.4 | `integration-test-mac`, `integration-test-mac-spark4-iceberg` |
| 4.1.3 | `integration-test-mac-spark41`, `integration-test-token`, `integration-test-mac-spark41-iceberg` |
| 4.2.0 | `integration-test-mac-spark42` |
| 4.3.0-rc1 | `integration-test-mac-spark43` |
1. macOS runners are billed at a 10x multiplier, so mirror download time is the most expensive kind of CI time in this repository.
2. `closer.lua` can redirect to a slow mirror, which puts these jobs at risk of hitting the 20-minute timeout for reasons unrelated to the change under test.
3. It reduces the load this repository puts on ASF mirrors.
4. Only 4.2.0 and 4.3.0 verified their download. Every version is verified now.
5. The download and extraction logic was duplicated seven times, so adding a version or fixing the logic meant editing seven places.
### Does this PR introduce _any_ user-facing change?
No. This is a CI-only change.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
Closes #543 from dongjoon-hyun/SPARK-59264.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.github/actions/setup-spark/action.yml | 64 ++++++++++++++++++++++++++++++++++
.github/workflows/build_and_test.yml | 52 +++++++++++++++------------
2 files changed, 93 insertions(+), 23 deletions(-)
diff --git a/.github/actions/setup-spark/action.yml b/.github/actions/setup-spark/action.yml
new file mode 100644
index 0000000..be0b33a
--- /dev/null
+++ b/.github/actions/setup-spark/action.yml
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: 'Setup Spark'
+description: 'Download (cached) and extract an Apache Spark distribution into /tmp/spark'
+
+inputs:
+ version:
+ description: 'Spark version, e.g. 4.1.3'
+ required: true
+ url:
+ description: 'Download URL. Defaults to the ASF mirror redirector.'
+ required: false
+ default: ''
+ sha512:
+ description: 'Expected SHA-512 of the tarball. Verified when given.'
+ required: false
+ default: ''
+
+runs:
+ using: composite
+ steps:
+ # The checksum is part of the key so that a release candidate and its final
+ # release, which share a version but not their bits, never share an entry.
+ - name: Cache Spark ${{ inputs.version }} distribution
+ id: cache
+ uses: actions/cache@v4
+ with:
+ path: ~/spark-dist/spark-${{ inputs.version }}-bin-hadoop3.tgz
+ key: spark-${{ inputs.version }}-bin-hadoop3-${{ inputs.sha512 }}
+
+ - name: Download Spark ${{ inputs.version }}
+ if: steps.cache.outputs.cache-hit != 'true'
+ shell: bash
+ run: |
+ mkdir -p ~/spark-dist
+ curl -L -o ~/spark-dist/spark-${{ inputs.version }}-bin-hadoop3.tgz \
+ "${{ inputs.url || format('https://www.apache.org/dyn/closer.lua/spark/spark-{0}/spark-{0}-bin-hadoop3.tgz?action=download', inputs.version) }}"
+
+ - name: Extract Spark ${{ inputs.version }} into /tmp/spark
+ shell: bash
+ run: |
+ cd ~/spark-dist
+ if [ -n "${{ inputs.sha512 }}" ]; then
+ echo "${{ inputs.sha512 }} spark-${{ inputs.version }}-bin-hadoop3.tgz" | shasum -a 512 -c
+ fi
+ tar xfz spark-${{ inputs.version }}-bin-hadoop3.tgz -C /tmp
+ mv /tmp/spark-${{ inputs.version }}-bin-hadoop3 /tmp/spark
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index 9cc6aeb..43ac910 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -150,11 +150,12 @@ jobs:
run: sudo xcode-select -s /Applications/Xcode_26.6.app/Contents/Developer
- name: Build
run: swift test --filter NOTHING -c release
+ - uses: ./.github/actions/setup-spark
+ with:
+ version: 4.1.3
+ sha512: 391d04f40da0f6e961513d1ad72a6e3d51aef99f751c835e058a329b01369d83d17c2a9c68fad6be9496eea6687107d0d3d6893f7a53f53ddbfe669022220523
- name: Test
run: |
- curl -LO https://www.apache.org/dyn/closer.lua/spark/spark-4.1.3/spark-4.1.3-bin-hadoop3.tgz?action=download
- tar xvfz spark-4.1.3-bin-hadoop3.tgz && rm spark-4.1.3-bin-hadoop3.tgz
- mv spark-4.1.3-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh
cd -
@@ -171,12 +172,12 @@ jobs:
run: sudo xcode-select -s /Applications/Xcode_26.6.app/Contents/Developer
- name: Build
run: swift test --filter NOTHING -c release
+ - uses: ./.github/actions/setup-spark
+ with:
+ version: 4.2.0
+ sha512: 3a6559e8546ff387db8fe7a04b8fe4008853b467b972c2e343df8e14a81450534777719fc5d4998dd96519a86fdf9de140cee753c61e2e94db044d5f9555ddc4
- name: Test
run: |
- curl -LO https://www.apache.org/dyn/closer.lua/spark/spark-4.2.0/spark-4.2.0-bin-hadoop3.tgz?action=download
- echo "3a6559e8546ff387db8fe7a04b8fe4008853b467b972c2e343df8e14a81450534777719fc5d4998dd96519a86fdf9de140cee753c61e2e94db044d5f9555ddc4 spark-4.2.0-bin-hadoop3.tgz" | shasum -a 512 -c
- tar xvfz spark-4.2.0-bin-hadoop3.tgz && rm spark-4.2.0-bin-hadoop3.tgz
- mv spark-4.2.0-bin-hadoop3/ /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh -c spark.sql.timeType.enabled=true
cd -
@@ -193,12 +194,13 @@ jobs:
run: sudo xcode-select -s /Applications/Xcode_26.6.app/Contents/Developer
- name: Build
run: swift test --filter NOTHING -c release
+ - uses: ./.github/actions/setup-spark
+ with:
+ version: 4.3.0
+ url: https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc1-bin/spark-4.3.0-bin-hadoop3.tgz
+ sha512: 18563bb57fa5e5367286a49845d44aa7e808e6c0f4dd151cb32cf71fffd17164734b4a2f54f81b94d69a58b4bdbac807cf05a33241ba9106c7600f491c1b157c
- name: Test
run: |
- curl -LO https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc1-bin/spark-4.3.0-bin-hadoop3.tgz
- echo "18563bb57fa5e5367286a49845d44aa7e808e6c0f4dd151cb32cf71fffd17164734b4a2f54f81b94d69a58b4bdbac807cf05a33241ba9106c7600f491c1b157c spark-4.3.0-bin-hadoop3.tgz" | shasum -a 512 -c
- tar xvfz spark-4.3.0-bin-hadoop3.tgz && rm spark-4.3.0-bin-hadoop3.tgz
- mv spark-4.3.0-bin-hadoop3/ /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh -c spark.sql.timeType.enabled=true
cd -
@@ -215,11 +217,12 @@ jobs:
run: sudo xcode-select -s /Applications/Xcode_26.6.app/Contents/Developer
- name: Build
run: swift test --filter NOTHING -c release
+ - uses: ./.github/actions/setup-spark
+ with:
+ version: 4.0.4
+ sha512: 1b91bf197df884368293cb91e7eda866751570b0c98f38fda712030dd8256962460d68d9f2c9075bc83061428b18108faa89510d3a20e1436a41fc73cadc24dd
- name: Test
run: |
- curl -LO https://www.apache.org/dyn/closer.lua/spark/spark-4.0.4/spark-4.0.4-bin-hadoop3.tgz?action=download
- tar xvfz spark-4.0.4-bin-hadoop3.tgz && rm spark-4.0.4-bin-hadoop3.tgz
- mv spark-4.0.4-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh
cd -
@@ -237,11 +240,12 @@ jobs:
run: sudo xcode-select -s /Applications/Xcode_26.6.app/Contents/Developer
- name: Build
run: swift test --filter NOTHING -c release
+ - uses: ./.github/actions/setup-spark
+ with:
+ version: 4.1.3
+ sha512: 391d04f40da0f6e961513d1ad72a6e3d51aef99f751c835e058a329b01369d83d17c2a9c68fad6be9496eea6687107d0d3d6893f7a53f53ddbfe669022220523
- name: Test
run: |
- curl -LO https://www.apache.org/dyn/closer.lua/spark/spark-4.1.3/spark-4.1.3-bin-hadoop3.tgz?action=download
- tar xvfz spark-4.1.3-bin-hadoop3.tgz && rm spark-4.1.3-bin-hadoop3.tgz
- mv spark-4.1.3-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh
cd -
@@ -264,11 +268,12 @@ jobs:
java-version: 17
- name: Build
run: swift test --filter NOTHING -c release
+ - uses: ./.github/actions/setup-spark
+ with:
+ version: 4.0.4
+ sha512: 1b91bf197df884368293cb91e7eda866751570b0c98f38fda712030dd8256962460d68d9f2c9075bc83061428b18108faa89510d3a20e1436a41fc73cadc24dd
- name: Test
run: |
- curl -LO https://www.apache.org/dyn/closer.lua/spark/spark-4.0.4/spark-4.0.4-bin-hadoop3.tgz?action=download
- tar xvfz spark-4.0.4-bin-hadoop3.tgz && rm spark-4.0.4-bin-hadoop3.tgz
- mv spark-4.0.4-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh --packages org.apache.spark:spark-connect_2.13:4.0.4,org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.11.0 -c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c spark.sql.catalog.local.type=hadoop -c spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c spark.sql.defaultCatalog=local
# Iceberg jars are resolved at startup, so wait until the server is ready.
@@ -297,11 +302,12 @@ jobs:
java-version: 21
- name: Build
run: swift test --filter NOTHING -c release
+ - uses: ./.github/actions/setup-spark
+ with:
+ version: 4.1.3
+ sha512: 391d04f40da0f6e961513d1ad72a6e3d51aef99f751c835e058a329b01369d83d17c2a9c68fad6be9496eea6687107d0d3d6893f7a53f53ddbfe669022220523
- name: Test
run: |
- curl -LO https://www.apache.org/dyn/closer.lua/spark/spark-4.1.3/spark-4.1.3-bin-hadoop3.tgz?action=download
- tar xvfz spark-4.1.3-bin-hadoop3.tgz && rm spark-4.1.3-bin-hadoop3.tgz
- mv spark-4.1.3-bin-hadoop3 /tmp/spark
cd /tmp/spark/sbin
./start-connect-server.sh --packages org.apache.spark:spark-connect_2.13:4.1.3,org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0 -c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c spark.sql.catalog.local.type=hadoop -c spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c spark.sql.defaultCatalog=local
# Iceberg jars are resolved at startup, so wait until the server is ready.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org