You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2026/07/14 02:22:15 UTC
(kyuubi) branch branch-1.11 updated: [KYUUBI #7561] [INFRA] Cache CI engine archives per URL with a reusable action
This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch branch-1.11
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.11 by this push:
new c39ebec1c3 [KYUUBI #7561] [INFRA] Cache CI engine archives per URL with a reusable action
c39ebec1c3 is described below
commit c39ebec1c3ddb255267412e570609624bc90b55c
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Tue Jul 14 10:21:52 2026 +0800
[KYUUBI #7561] [INFRA] Cache CI engine archives per URL with a reusable action
### Why are the changes needed?
The old `cache-engine-archives` cache is one blob under a constant key, so bumping any
engine invalidates all of them. This provisions engines explicitly instead:
- New reusable action `.github/actions/download-archive` downloads a tarball and caches it
with `actions/cache` **keyed by the URL** (atomic download), so each engine caches
independently and stable versions restore instantly.
- `.github/scripts/resolve-engine-archive-urls.sh` derives each URL from the Maven
properties (honoring the build's profiles/`-D` overrides); `expose-engine-homes.sh`
publishes `SPARK_HOME`/`HIVE_HOME`/`FLINK_HOME` from `/opt`.
- Every `master.yml` job now resolves → downloads → exposes only the engines it needs, and
the build runs with `-P{spark,flink,hive}-provided`. The now-orphaned
`cache-engine-archives` action is removed.
- Flink test helpers now honor `FLINK_HOME` first (fallback to the download dir for local
runs), matching Spark/Hive.
- Spark extension modules gain a `spark-home-from-archive` profile, auto-activated only when
`SPARK_HOME` is unset, so the Standalone-cluster tests use the downloaded archive locally
but a provided `SPARK_HOME` is no longer overridden.
- The `hive-it` avatica/CALCITE-1224 hack is now a single `zip -d`.
### How was this patch tested?
Run by this PR's CI.
Locally checked: URL resolution across all matrix cases (Scala 2.13, Spark/Flink/Hive
version overrides, CDH6 empty-query), `*_HOME` resolution, the `closer.lua` redirect for
`curl -fSL`, the `zip -d` patch on a mock jar, and the extension `SPARK_HOME` profile via
the effective POM (inherits the env when set, falls back to the archive otherwise).
### Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Claude Opus 4.8
Closes #7561 from pan3793/gha-dl.
Closes #7561
1f9ad04dc [Cheng Pan] [INFRA] Restore SPARK_SCALA_VERSION for Spark extension local-cluster tests
20a94b22f [Cheng Pan] [INFRA] Bump actions/cache to v6 in download-archive action
a5c4208f0 [Cheng Pan] fix spark extension standalone tests
0010f4e4c [Cheng Pan] [INFRA] Remove orphaned cache-engine-archives action
da90a49d2 [Cheng Pan] nit
13948d3d3 [Cheng Pan] [INFRA] Cache CI engine archives per URL with a reusable action
Authored-by: Cheng Pan <ch...@apache.org>
Signed-off-by: Cheng Pan <ch...@apache.org>
(cherry picked from commit 445a906c2b41a04cbf512d01b5c6f02583821a7e)
Signed-off-by: Cheng Pan <ch...@apache.org>
---
.github/actions/cache-engine-archives/action.yaml | 30 -----
.github/actions/download-archive/action.yaml | 73 +++++++++++++
.github/scripts/expose-engine-homes.sh | 50 +++++++++
.github/scripts/resolve-engine-archive-urls.sh | 41 +++++++
.github/workflows/master.yml | 121 ++++++++++++++++-----
.../spark/kyuubi-extension-spark-3-3/pom.xml | 55 +++++++---
.../spark/kyuubi-extension-spark-3-4/pom.xml | 55 +++++++---
.../spark/kyuubi-extension-spark-3-5/pom.xml | 55 +++++++---
.../spark/kyuubi-extension-spark-4-0/pom.xml | 55 +++++++---
.../spark/kyuubi-extension-spark-4-1/pom.xml | 55 +++++++---
.../engine/flink/WithFlinkSQLEngineLocal.scala | 2 +-
.../engine/flink/WithFlinkSQLEngineOnYarn.scala | 2 +-
.../flink/WithKyuubiServerAndYarnMiniCluster.scala | 2 +-
13 files changed, 464 insertions(+), 132 deletions(-)
diff --git a/.github/actions/cache-engine-archives/action.yaml b/.github/actions/cache-engine-archives/action.yaml
deleted file mode 100644
index ff69d0b4cc..0000000000
--- a/.github/actions/cache-engine-archives/action.yaml
+++ /dev/null
@@ -1,30 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-name: cache-engine-archives
-description: 'Cache download engine archives from Apache Archives website used by Maven download plugin'
-runs:
- using: composite
- steps:
- - name: Cache Engine Archives
- uses: actions/cache@v4
- with:
- path: /tmp/engine-archives
- key: engine-archives
- - name: Show cached engine archives
- shell: bash
- run: ls -lh /tmp/engine-archives || echo ''
diff --git a/.github/actions/download-archive/action.yaml b/.github/actions/download-archive/action.yaml
new file mode 100644
index 0000000000..0f9b2a98ea
--- /dev/null
+++ b/.github/actions/download-archive/action.yaml
@@ -0,0 +1,73 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+name: download-archive
+description: 'Download and cache a binary tarball from an explicit HTTP link, optionally extracting it.'
+inputs:
+ url:
+ description: 'The HTTP(S) link to download the tarball from. Also used as the cache key.'
+ required: true
+ untar-dir:
+ description: 'Directory to extract the downloaded tarball into. When omitted, the archive is only downloaded and cached.'
+ required: false
+ default: ''
+runs:
+ using: composite
+ steps:
+ - name: Resolve archive path
+ id: archive
+ shell: bash
+ env:
+ URL: ${{ inputs.url }}
+ run: |
+ # Strip the query string before taking the file name, e.g.
+ # .../spark-3.5.8-bin-hadoop3.tgz?action=download -> spark-3.5.8-bin-hadoop3.tgz
+ name="$(basename "${URL%%\?*}")"
+ if [[ -z "$name" ]]; then
+ echo "Cannot resolve an archive file name from URL: $URL" >&2
+ exit 1
+ fi
+ path="/tmp/$name"
+ echo "path=$path" >> "$GITHUB_OUTPUT"
+ - name: Cache archive
+ id: cache
+ uses: actions/cache@v6
+ with:
+ path: ${{ steps.archive.outputs.path }}
+ key: ${{ inputs.url }}
+ - name: Download archive
+ if: steps.cache.outputs.cache-hit != 'true'
+ shell: bash
+ env:
+ URL: ${{ inputs.url }}
+ ARCHIVE_PATH: ${{ steps.archive.outputs.path }}
+ run: |
+ echo "Downloading $URL"
+ # Download to a temp path and move into place only on success, so a
+ # failed or partial download never populates the cached archive path.
+ curl -fSL --retry 3 --retry-delay 5 --connect-timeout 30 -o "$ARCHIVE_PATH.tmp" "$URL"
+ mv "$ARCHIVE_PATH.tmp" "$ARCHIVE_PATH"
+ - name: Extract archive
+ if: inputs.untar-dir != ''
+ shell: bash
+ env:
+ ARCHIVE_PATH: ${{ steps.archive.outputs.path }}
+ UNTAR_DIR: ${{ inputs.untar-dir }}
+ run: |
+ mkdir -p "$UNTAR_DIR"
+ echo "Extracting $ARCHIVE_PATH into $UNTAR_DIR"
+ tar -xf "$ARCHIVE_PATH" -C "$UNTAR_DIR"
diff --git a/.github/scripts/expose-engine-homes.sh b/.github/scripts/expose-engine-homes.sh
new file mode 100755
index 0000000000..9a1ce93285
--- /dev/null
+++ b/.github/scripts/expose-engine-homes.sh
@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Resolves the *_HOME variable for each requested engine from the archives
+# extracted under /opt by the download-archive action, and appends them to
+# $GITHUB_ENV. Each home is matched by its unique directory prefix rather than
+# derived from the archive name, since the extracted top-level directory does
+# not always match (e.g. Flink drops the -bin-scala suffix).
+#
+# Usage: expose-engine-homes.sh <engines>
+# <engines> comma-separated engine list, e.g. "spark,hive,flink" or "spark"
+
+set -euo pipefail
+
+expose_home() {
+ local engine="$1" var pattern
+ case "$engine" in
+ spark) var=SPARK_HOME; pattern='/opt/spark-*' ;;
+ hive) var=HIVE_HOME; pattern='/opt/apache-hive-*' ;;
+ flink) var=FLINK_HOME; pattern='/opt/flink-*' ;;
+ *) echo "Unknown engine: $engine" >&2; exit 1 ;;
+ esac
+ local dirs=($pattern)
+ if [[ ${#dirs[@]} -ne 1 || ! -d "${dirs[0]}" ]]; then
+ echo "Expected exactly one directory matching '$pattern', found: ${dirs[*]}" >&2
+ exit 1
+ fi
+ echo "$var=${dirs[0]}" | tee -a "$GITHUB_ENV"
+}
+
+IFS=',' read -r -a engines <<< "${1:?Usage: $0 <engines>}"
+for engine in "${engines[@]}"; do
+ expose_home "$engine"
+done
diff --git a/.github/scripts/resolve-engine-archive-urls.sh b/.github/scripts/resolve-engine-archive-urls.sh
new file mode 100755
index 0000000000..6c2149416b
--- /dev/null
+++ b/.github/scripts/resolve-engine-archive-urls.sh
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Resolves the download URL (${mirror}/${name}${query}) for each requested
+# engine's binary archive from the Maven build properties and writes them to
+# $GITHUB_OUTPUT as <engine>-url, to be consumed by the download-archive action.
+#
+# Usage: resolve-engine-archive-urls.sh <engines> [maven-args...]
+# <engines> comma-separated engine list, e.g. "spark,hive,flink" or "flink"
+# maven-args forwarded to build/mvn so the caller passes the same profiles
+# and -D overrides as the build (e.g. -Pspark-3.5 -Pscala-2.13)
+
+set -euo pipefail
+
+IFS=',' read -r -a engines <<< "${1:?Usage: $0 <engines> [maven-args...]}"
+shift
+
+for engine in "${engines[@]}"; do
+ mirror=$(./build/mvn -N -q help:evaluate -Dexpression="${engine}.archive.mirror" -DforceStdout "$@")
+ name=$(./build/mvn -N -q help:evaluate -Dexpression="${engine}.archive.name" -DforceStdout "$@")
+ query=$(./build/mvn -N -q help:evaluate -Dexpression="${engine}.archive.query" -DforceStdout "$@")
+ url="${mirror}/${name}${query}"
+ echo "${engine}: ${url}"
+ echo "${engine}-url=${url}" >> "$GITHUB_OUTPUT"
+done
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index b10576e053..f324383729 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -36,7 +36,7 @@ concurrency:
cancel-in-progress: true
env:
- MAVEN_ARGS: -Dmaven.javadoc.skip=true -Drat.skip=true -Dscalastyle.skip=true -Dspotless.check.skip -Dorg.slf4j.simpleLogger.defaultLogLevel=warn -Pjdbc-shaded,gen-policy -Dmaven.plugin.download.cache.path=/tmp/engine-archives
+ MAVEN_ARGS: -Dmaven.javadoc.skip=true -Drat.skip=true -Dscalastyle.skip=true -Dspotless.check.skip -Dorg.slf4j.simpleLogger.defaultLogLevel=warn -Pjdbc-shaded,gen-policy -Pflink-provided,hive-provided,spark-provided
KUBERNETES_VERSION: v1.30.0
MINIKUBE_VERSION: v1.33.1
@@ -111,14 +111,34 @@ jobs:
distribution: temurin
java-version: ${{ matrix.java }}
cache: 'maven'
- - name: Cache Engine Archives
- uses: ./.github/actions/cache-engine-archives
- name: Setup Maven
uses: ./.github/actions/setup-maven
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
+ - name: Resolve engine archive download URLs
+ id: engine-archives
+ run: ./.github/scripts/resolve-engine-archive-urls.sh spark,hive,flink -Pspark-${{ matrix.spark }} ${{ matrix.spark-archive }}
+ - name: Make /opt writable for engine archives
+ run: sudo chown "$(id -un)" /opt
+ - name: Download Spark
+ uses: ./.github/actions/download-archive
+ with:
+ url: ${{ steps.engine-archives.outputs.spark-url }}
+ untar-dir: /opt
+ - name: Download Hive
+ uses: ./.github/actions/download-archive
+ with:
+ url: ${{ steps.engine-archives.outputs.hive-url }}
+ untar-dir: /opt
+ - name: Download Flink
+ uses: ./.github/actions/download-archive
+ with:
+ url: ${{ steps.engine-archives.outputs.flink-url }}
+ untar-dir: /opt
+ - name: Expose engine homes
+ run: ./.github/scripts/expose-engine-homes.sh spark,hive,flink
- name: Build and test Kyuubi and Spark with maven w/o linters
run: |
if [[ "${{ matrix.java }}" == "8" && "${{ matrix.spark }}" == "3.5" && "${{ matrix.spark-archive }}" == "" ]]; then
@@ -171,8 +191,24 @@ jobs:
cache: 'maven'
- name: Setup Maven
uses: ./.github/actions/setup-maven
- - name: Cache Engine Archives
- uses: ./.github/actions/cache-engine-archives
+ - name: Resolve engine archive download URLs
+ id: engine-archives
+ run: ./.github/scripts/resolve-engine-archive-urls.sh spark,hive -Pspark-${{ matrix.spark }} -Pscala-${{ matrix.scala }}
+ - name: Make /opt writable for engine archives
+ run: sudo chown "$(id -un)" /opt
+ - name: Download Spark
+ uses: ./.github/actions/download-archive
+ with:
+ url: ${{ steps.engine-archives.outputs.spark-url }}
+ untar-dir: /opt
+ - name: Download Hive
+ uses: ./.github/actions/download-archive
+ with:
+ url: ${{ steps.engine-archives.outputs.hive-url }}
+ untar-dir: /opt
+ # Flink modules are excluded from this job, so Flink is not downloaded.
+ - name: Expose engine homes
+ run: ./.github/scripts/expose-engine-homes.sh spark,hive
- name: Build on Scala ${{ matrix.scala }}
run: |
TEST_MODULES="!externals/kyuubi-flink-sql-engine,!integration-tests/kyuubi-flink-it"
@@ -293,8 +329,18 @@ jobs:
cache: 'maven'
- name: Setup Maven
uses: ./.github/actions/setup-maven
- - name: Cache Engine Archives
- uses: ./.github/actions/cache-engine-archives
+ - name: Resolve engine archive download URLs
+ id: engine-archives
+ run: ./.github/scripts/resolve-engine-archive-urls.sh flink -Pflink-${{ matrix.flink }} ${{ matrix.flink-archive }}
+ - name: Make /opt writable for engine archives
+ run: sudo chown "$(id -un)" /opt
+ - name: Download Flink
+ uses: ./.github/actions/download-archive
+ with:
+ url: ${{ steps.engine-archives.outputs.flink-url }}
+ untar-dir: /opt
+ - name: Expose engine homes
+ run: ./.github/scripts/expose-engine-homes.sh flink
- name: Build Flink with maven w/o linters
run: |
TEST_MODULES="externals/kyuubi-flink-sql-engine,integration-tests/kyuubi-flink-it"
@@ -351,24 +397,25 @@ jobs:
cache: 'maven'
- name: Setup Maven
uses: ./.github/actions/setup-maven
- - name: Cache Engine Archives
- uses: ./.github/actions/cache-engine-archives
+ - name: Resolve engine archive download URLs
+ id: engine-archives
+ run: ./.github/scripts/resolve-engine-archive-urls.sh hive ${{ matrix.hive-archive }}
+ - name: Make /opt writable for engine archives
+ run: sudo chown "$(id -un)" /opt
+ - name: Download Hive
+ uses: ./.github/actions/download-archive
+ with:
+ url: ${{ steps.engine-archives.outputs.hive-url }}
+ untar-dir: /opt
+ - name: Expose engine homes
+ run: ./.github/scripts/expose-engine-homes.sh hive
- name: Build and test Hive with maven w/o linters
run: |
TEST_MODULES="externals/kyuubi-hive-sql-engine,integration-tests/kyuubi-hive-it"
./build/mvn ${{ matrix.hive-archive }} -pl ${TEST_MODULES} -am clean install -DskipTests
- # unpack avatica-1.8.0.jar and delete Jackson classes to workaround CALCITE-1224
- if [[ "${{ matrix.hive-archive }}" == *apache-hive-2.3.10-bin.tar.gz* ]]; then
- HIVE_2310_LIB="$PWD/externals/kyuubi-download/target/apache-hive-2.3.10-bin/lib"
- mkdir avatica-1.8.0-unpack
- cd avatica-1.8.0-unpack
- jar xf $HIVE_2310_LIB/avatica-1.8.0.jar
- rm -rf com/fasterxml/jackson org/slf4j
- jar cf avatica-1.8.0-patched.jar .
- rm $HIVE_2310_LIB/avatica-1.8.0.jar
- mv avatica-1.8.0-patched.jar $HIVE_2310_LIB/
- cd ..
- rm -rf avatica-1.8.0-unpack
+ # delete bundled Jackson/SLF4J classes from avatica-1.8.0.jar to workaround CALCITE-1224
+ if [[ "$(basename "$HIVE_HOME")" == "apache-hive-2.3.10-bin" ]]; then
+ zip -d "$HIVE_HOME/lib/avatica-1.8.0.jar" 'com/fasterxml/jackson/*' 'org/slf4j/*'
fi
./build/mvn ${{ matrix.hive-archive }} -pl ${TEST_MODULES} test
- name: Upload test logs
@@ -402,8 +449,18 @@ jobs:
cache: 'maven'
- name: Setup Maven
uses: ./.github/actions/setup-maven
- - name: Cache Engine Archives
- uses: ./.github/actions/cache-engine-archives
+ - name: Resolve engine archive download URLs
+ id: engine-archives
+ run: ./.github/scripts/resolve-engine-archive-urls.sh spark
+ - name: Make /opt writable for engine archives
+ run: sudo chown "$(id -un)" /opt
+ - name: Download Spark
+ uses: ./.github/actions/download-archive
+ with:
+ url: ${{ steps.engine-archives.outputs.spark-url }}
+ untar-dir: /opt
+ - name: Expose engine homes
+ run: ./.github/scripts/expose-engine-homes.sh spark
- name: Build and test JDBC and Trino engines, TPC-H/TPC-DS with maven w/o linters
run: |
JDBC_TEST_MODULES="externals/kyuubi-jdbc-engine,integration-tests/kyuubi-jdbc-it"
@@ -452,8 +509,6 @@ jobs:
file: build/Dockerfile.CI
load: true
tags: apache/kyuubi:ci
- - name: Cache Engine Archives
- uses: ./.github/actions/cache-engine-archives
- name: Setup Minikube
run: |
# https://minikube.sigs.k8s.io/docs/start/
@@ -510,14 +565,24 @@ jobs:
- uses: actions/checkout@v4
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- - name: Cache Engine Archives
- uses: ./.github/actions/cache-engine-archives
- name: Setup JDK 11
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: 11
cache: 'maven'
+ - name: Resolve engine archive download URLs
+ id: engine-archives
+ run: ./.github/scripts/resolve-engine-archive-urls.sh spark
+ - name: Make /opt writable for engine archives
+ run: sudo chown "$(id -un)" /opt
+ - name: Download Spark
+ uses: ./.github/actions/download-archive
+ with:
+ url: ${{ steps.engine-archives.outputs.spark-url }}
+ untar-dir: /opt
+ - name: Expose engine homes
+ run: ./.github/scripts/expose-engine-homes.sh spark
- name: Setup Minikube
run: |
# https://minikube.sigs.k8s.io/docs/start/
@@ -574,8 +639,6 @@ jobs:
cache: 'maven'
- name: Setup Maven
uses: ./.github/actions/setup-maven
- - name: Cache Engine Archives
- uses: ./.github/actions/cache-engine-archives
- name: zookeeper integration tests
run: |
export KYUUBI_IT_ZOOKEEPER_VERSION=${{ matrix.zookeeper }}
diff --git a/extensions/spark/kyuubi-extension-spark-3-3/pom.xml b/extensions/spark/kyuubi-extension-spark-3-3/pom.xml
index 93bc2f55ec..2eaeef645c 100644
--- a/extensions/spark/kyuubi-extension-spark-3-3/pom.xml
+++ b/extensions/spark/kyuubi-extension-spark-3-3/pom.xml
@@ -145,20 +145,6 @@
</execution>
</executions>
</plugin>
- <plugin>
- <groupId>org.scalatest</groupId>
- <artifactId>scalatest-maven-plugin</artifactId>
- <configuration>
- <environmentVariables>
- <!--
- Some tests run Spark in local-cluster mode.
- This mode uses SPARK_HOME and SPARK_SCALA_VERSION to build command to launch a Spark Standalone Cluster.
- -->
- <SPARK_HOME>${spark.home}</SPARK_HOME>
- <SPARK_SCALA_VERSION>${scala.binary.version}</SPARK_SCALA_VERSION>
- </environmentVariables>
- </configuration>
- </plugin>
<plugin>
<groupId>org.antlr</groupId>
<artifactId>antlr4-maven-plugin</artifactId>
@@ -188,8 +174,49 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
+ <environmentVariables>
+ <!--
+ Some tests run Spark in local-cluster mode, which uses SPARK_HOME and
+ SPARK_SCALA_VERSION to build the command that launches a Spark Standalone
+ cluster. SPARK_SCALA_VERSION is always derived from the build here;
+ SPARK_HOME is provided by the spark-home-from-archive profile only when the
+ environment does not already supply it.
+ -->
+ <SPARK_SCALA_VERSION>${scala.binary.version}</SPARK_SCALA_VERSION>
+ </environmentVariables>
+ </configuration>
+ </plugin>
</plugins>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+ <profiles>
+ <profile>
+ <!-- When SPARK_HOME is not provided by the environment, point it at the downloaded
+ Spark archive for tests that launch a Spark Standalone cluster. -->
+ <id>spark-home-from-archive</id>
+ <activation>
+ <property>
+ <name>!env.SPARK_HOME</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
+ <environmentVariables>
+ <SPARK_HOME>${spark.home}</SPARK_HOME>
+ </environmentVariables>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
diff --git a/extensions/spark/kyuubi-extension-spark-3-4/pom.xml b/extensions/spark/kyuubi-extension-spark-3-4/pom.xml
index bb8e6f755a..d56d1cc98f 100644
--- a/extensions/spark/kyuubi-extension-spark-3-4/pom.xml
+++ b/extensions/spark/kyuubi-extension-spark-3-4/pom.xml
@@ -145,20 +145,6 @@
</execution>
</executions>
</plugin>
- <plugin>
- <groupId>org.scalatest</groupId>
- <artifactId>scalatest-maven-plugin</artifactId>
- <configuration>
- <environmentVariables>
- <!--
- Some tests run Spark in local-cluster mode.
- This mode uses SPARK_HOME and SPARK_SCALA_VERSION to build command to launch a Spark Standalone Cluster.
- -->
- <SPARK_HOME>${spark.home}</SPARK_HOME>
- <SPARK_SCALA_VERSION>${scala.binary.version}</SPARK_SCALA_VERSION>
- </environmentVariables>
- </configuration>
- </plugin>
<plugin>
<groupId>org.antlr</groupId>
<artifactId>antlr4-maven-plugin</artifactId>
@@ -188,8 +174,49 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
+ <environmentVariables>
+ <!--
+ Some tests run Spark in local-cluster mode, which uses SPARK_HOME and
+ SPARK_SCALA_VERSION to build the command that launches a Spark Standalone
+ cluster. SPARK_SCALA_VERSION is always derived from the build here;
+ SPARK_HOME is provided by the spark-home-from-archive profile only when the
+ environment does not already supply it.
+ -->
+ <SPARK_SCALA_VERSION>${scala.binary.version}</SPARK_SCALA_VERSION>
+ </environmentVariables>
+ </configuration>
+ </plugin>
</plugins>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+ <profiles>
+ <profile>
+ <!-- When SPARK_HOME is not provided by the environment, point it at the downloaded
+ Spark archive for tests that launch a Spark Standalone cluster. -->
+ <id>spark-home-from-archive</id>
+ <activation>
+ <property>
+ <name>!env.SPARK_HOME</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
+ <environmentVariables>
+ <SPARK_HOME>${spark.home}</SPARK_HOME>
+ </environmentVariables>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
diff --git a/extensions/spark/kyuubi-extension-spark-3-5/pom.xml b/extensions/spark/kyuubi-extension-spark-3-5/pom.xml
index 12c6c6ba6f..a3ff06df0c 100644
--- a/extensions/spark/kyuubi-extension-spark-3-5/pom.xml
+++ b/extensions/spark/kyuubi-extension-spark-3-5/pom.xml
@@ -145,20 +145,6 @@
</execution>
</executions>
</plugin>
- <plugin>
- <groupId>org.scalatest</groupId>
- <artifactId>scalatest-maven-plugin</artifactId>
- <configuration>
- <environmentVariables>
- <!--
- Some tests run Spark in local-cluster mode.
- This mode uses SPARK_HOME and SPARK_SCALA_VERSION to build command to launch a Spark Standalone Cluster.
- -->
- <SPARK_HOME>${spark.home}</SPARK_HOME>
- <SPARK_SCALA_VERSION>${scala.binary.version}</SPARK_SCALA_VERSION>
- </environmentVariables>
- </configuration>
- </plugin>
<plugin>
<groupId>org.antlr</groupId>
<artifactId>antlr4-maven-plugin</artifactId>
@@ -188,8 +174,49 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
+ <environmentVariables>
+ <!--
+ Some tests run Spark in local-cluster mode, which uses SPARK_HOME and
+ SPARK_SCALA_VERSION to build the command that launches a Spark Standalone
+ cluster. SPARK_SCALA_VERSION is always derived from the build here;
+ SPARK_HOME is provided by the spark-home-from-archive profile only when the
+ environment does not already supply it.
+ -->
+ <SPARK_SCALA_VERSION>${scala.binary.version}</SPARK_SCALA_VERSION>
+ </environmentVariables>
+ </configuration>
+ </plugin>
</plugins>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+ <profiles>
+ <profile>
+ <!-- When SPARK_HOME is not provided by the environment, point it at the downloaded
+ Spark archive for tests that launch a Spark Standalone cluster. -->
+ <id>spark-home-from-archive</id>
+ <activation>
+ <property>
+ <name>!env.SPARK_HOME</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
+ <environmentVariables>
+ <SPARK_HOME>${spark.home}</SPARK_HOME>
+ </environmentVariables>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
diff --git a/extensions/spark/kyuubi-extension-spark-4-0/pom.xml b/extensions/spark/kyuubi-extension-spark-4-0/pom.xml
index 08286aa527..7fc6081182 100644
--- a/extensions/spark/kyuubi-extension-spark-4-0/pom.xml
+++ b/extensions/spark/kyuubi-extension-spark-4-0/pom.xml
@@ -145,20 +145,6 @@
</execution>
</executions>
</plugin>
- <plugin>
- <groupId>org.scalatest</groupId>
- <artifactId>scalatest-maven-plugin</artifactId>
- <configuration>
- <environmentVariables>
- <!--
- Some tests run Spark in local-cluster mode.
- This mode uses SPARK_HOME and SPARK_SCALA_VERSION to build command to launch a Spark Standalone Cluster.
- -->
- <SPARK_HOME>${spark.home}</SPARK_HOME>
- <SPARK_SCALA_VERSION>${scala.binary.version}</SPARK_SCALA_VERSION>
- </environmentVariables>
- </configuration>
- </plugin>
<plugin>
<groupId>org.antlr</groupId>
<artifactId>antlr4-maven-plugin</artifactId>
@@ -188,8 +174,49 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
+ <environmentVariables>
+ <!--
+ Some tests run Spark in local-cluster mode, which uses SPARK_HOME and
+ SPARK_SCALA_VERSION to build the command that launches a Spark Standalone
+ cluster. SPARK_SCALA_VERSION is always derived from the build here;
+ SPARK_HOME is provided by the spark-home-from-archive profile only when the
+ environment does not already supply it.
+ -->
+ <SPARK_SCALA_VERSION>${scala.binary.version}</SPARK_SCALA_VERSION>
+ </environmentVariables>
+ </configuration>
+ </plugin>
</plugins>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+ <profiles>
+ <profile>
+ <!-- When SPARK_HOME is not provided by the environment, point it at the downloaded
+ Spark archive for tests that launch a Spark Standalone cluster. -->
+ <id>spark-home-from-archive</id>
+ <activation>
+ <property>
+ <name>!env.SPARK_HOME</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
+ <environmentVariables>
+ <SPARK_HOME>${spark.home}</SPARK_HOME>
+ </environmentVariables>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
diff --git a/extensions/spark/kyuubi-extension-spark-4-1/pom.xml b/extensions/spark/kyuubi-extension-spark-4-1/pom.xml
index 127e2a8b7f..57c94ad5b5 100644
--- a/extensions/spark/kyuubi-extension-spark-4-1/pom.xml
+++ b/extensions/spark/kyuubi-extension-spark-4-1/pom.xml
@@ -145,20 +145,6 @@
</execution>
</executions>
</plugin>
- <plugin>
- <groupId>org.scalatest</groupId>
- <artifactId>scalatest-maven-plugin</artifactId>
- <configuration>
- <environmentVariables>
- <!--
- Some tests run Spark in local-cluster mode.
- This mode uses SPARK_HOME and SPARK_SCALA_VERSION to build command to launch a Spark Standalone Cluster.
- -->
- <SPARK_HOME>${spark.home}</SPARK_HOME>
- <SPARK_SCALA_VERSION>${scala.binary.version}</SPARK_SCALA_VERSION>
- </environmentVariables>
- </configuration>
- </plugin>
<plugin>
<groupId>org.antlr</groupId>
<artifactId>antlr4-maven-plugin</artifactId>
@@ -188,8 +174,49 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
+ <environmentVariables>
+ <!--
+ Some tests run Spark in local-cluster mode, which uses SPARK_HOME and
+ SPARK_SCALA_VERSION to build the command that launches a Spark Standalone
+ cluster. SPARK_SCALA_VERSION is always derived from the build here;
+ SPARK_HOME is provided by the spark-home-from-archive profile only when the
+ environment does not already supply it.
+ -->
+ <SPARK_SCALA_VERSION>${scala.binary.version}</SPARK_SCALA_VERSION>
+ </environmentVariables>
+ </configuration>
+ </plugin>
</plugins>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+ <profiles>
+ <profile>
+ <!-- When SPARK_HOME is not provided by the environment, point it at the downloaded
+ Spark archive for tests that launch a Spark Standalone cluster. -->
+ <id>spark-home-from-archive</id>
+ <activation>
+ <property>
+ <name>!env.SPARK_HOME</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.scalatest</groupId>
+ <artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
+ <environmentVariables>
+ <SPARK_HOME>${spark.home}</SPARK_HOME>
+ </environmentVariables>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
diff --git a/externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/WithFlinkSQLEngineLocal.scala b/externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/WithFlinkSQLEngineLocal.scala
index 4ae7983a9e..357be1b730 100644
--- a/externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/WithFlinkSQLEngineLocal.scala
+++ b/externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/WithFlinkSQLEngineLocal.scala
@@ -75,7 +75,7 @@ trait WithFlinkSQLEngineLocal extends KyuubiFunSuite with WithFlinkTestResources
val envs = scala.collection.mutable.Map[String, String]()
val kyuubiExternals = JavaUtils.getCodeSourceLocation(getClass)
.split("externals").head
- val flinkHome = {
+ val flinkHome = sys.env.get("FLINK_HOME").filter(_.nonEmpty).map(Paths.get(_)).orElse {
val candidates = Paths.get(kyuubiExternals, "externals", "kyuubi-download", "target")
.toFile.listFiles(f => f.getName.contains("flink"))
if (candidates == null) None else candidates.map(_.toPath).headOption
diff --git a/externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/WithFlinkSQLEngineOnYarn.scala b/externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/WithFlinkSQLEngineOnYarn.scala
index d85d933343..4c1155578e 100644
--- a/externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/WithFlinkSQLEngineOnYarn.scala
+++ b/externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/WithFlinkSQLEngineOnYarn.scala
@@ -141,7 +141,7 @@ trait WithFlinkSQLEngineOnYarn extends KyuubiFunSuite with WithFlinkTestResource
val envs = scala.collection.mutable.Map[String, String]()
val kyuubiExternals = JavaUtils.getCodeSourceLocation(getClass)
.split("externals").head
- val flinkHome = {
+ val flinkHome = sys.env.get("FLINK_HOME").filter(_.nonEmpty).map(Paths.get(_)).orElse {
val candidates = Paths.get(kyuubiExternals, "externals", "kyuubi-download", "target")
.toFile.listFiles(f => f.getName.contains("flink"))
if (candidates == null) None else candidates.map(_.toPath).headOption
diff --git a/integration-tests/kyuubi-flink-it/src/test/scala/org/apache/kyuubi/it/flink/WithKyuubiServerAndYarnMiniCluster.scala b/integration-tests/kyuubi-flink-it/src/test/scala/org/apache/kyuubi/it/flink/WithKyuubiServerAndYarnMiniCluster.scala
index fc58a78335..8cd7f453be 100644
--- a/integration-tests/kyuubi-flink-it/src/test/scala/org/apache/kyuubi/it/flink/WithKyuubiServerAndYarnMiniCluster.scala
+++ b/integration-tests/kyuubi-flink-it/src/test/scala/org/apache/kyuubi/it/flink/WithKyuubiServerAndYarnMiniCluster.scala
@@ -116,7 +116,7 @@ trait WithKyuubiServerAndYarnMiniCluster extends KyuubiFunSuite with WithKyuubiS
yarnConf.writeXml(writer)
writer.close()
- val flinkHome = {
+ val flinkHome = sys.env.get("FLINK_HOME").filter(_.nonEmpty).map(Paths.get(_)).orElse {
val candidates = Paths.get(kyuubiHome, "externals", "kyuubi-download", "target")
.toFile.listFiles(f => f.getName.contains("flink"))
if (candidates == null) None else candidates.map(_.toPath).headOption