You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2026/08/13 13:44:15 UTC
(kyuubi) branch master updated: [KYUUBI #7613][BUILD] Use spark 3.5 to build the extra scala 2.12 engine for spark 4.x
This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 0e1c4a38c5 [KYUUBI #7613][BUILD] Use spark 3.5 to build the extra scala 2.12 engine for spark 4.x
0e1c4a38c5 is described below
commit 0e1c4a38c5c83ffdc6a4ecce194d78c38249b4ee
Author: wforget <64...@qq.com>
AuthorDate: Thu Aug 13 21:44:05 2026 +0800
[KYUUBI #7613][BUILD] Use spark 3.5 to build the extra scala 2.12 engine for spark 4.x
### Why are the changes needed?
`build/dist` packages Spark SQL engine artifacts for both Scala 2.12 and Scala 2.13.
When Spark 4.x and Scala 2.13 profiles are specified, the extra engine build switches the Scala profile to 2.12 but retains the Spark 4.x profile. Since Spark 4.x artifacts are unavailable for Scala 2.12, Maven dependency resolution fails.
The distribution should continue shipping both engine variants so that it can support Spark 4.x while retaining compatibility with Spark 3.5 LTS. This patch replaces the Spark 4.x or `spark-master` profile with `spark-3.5` when building the extra Scala 2.12 engine.
Closes #7613.
### How was this patch tested?
build successful using the command `./build/dist --tgz --spark-provided --flink-provided --hive-provided -Pspark-4.2,scala-2.13`.
```
+ FILTERED_ARGS=()
+ EXTRA_SCALA_212_SPARK_PROFILE=spark-3.5
+ for arg in '"$"'
+ FILTERED_ARG=-Pspark-4.2,scala-2.13
+ [[ -Pspark-4.2,scala-2.13 == *scala-2.12* ]]
+ [[ -Pspark-4.2,scala-2.13 == *scala-2.13* ]]
+ FILTERED_ARG=-Pspark-4.2,scala-2.12
+ [[ 2.13 == \2\.\1\3 ]]
+ [[ 4.2.0 == 4.* ]]
+ [[ -Pspark-4.2,scala-2.12 =~ spark-4\.[0-9]+ ]]
+ FILTERED_ARG=-Pspark-3.5,scala-2.12
+ [[ -Pspark-3.5,scala-2.12 =~ spark-4\.[0-9]+ ]]
+ FILTERED_ARG=-Pspark-3.5,scala-2.12
+ FILTERED_ARGS+=("$FILTERED_ARG")
+ '[' 2.13 = 2.12 ']'
+ EXTRA_SPARK_ENGINE_BUILD_COMMAND=("$MVN" install $MVN_DIST_OPT ${FILTERED_ARGS[]} -pl :kyuubi-spark-sql-engine_2.12 -am)
+ echo -e '$ /Users/wforget/work/git/kyuubi/build/mvn' install -DskipTests -Dmaven.source.skip -Pspark-provided -Pflink-provided -Phive-provided -Pspark-3.5,scala-2.12 -pl :kyuubi-spark-sql-engine_2.12 '-am\n'
$ /Users/wforget/work/git/kyuubi/build/mvn install -DskipTests -Dmaven.source.skip -Pspark-provided -Pflink-provided -Phive-provided -Pspark-3.5,scala-2.12 -pl :kyuubi-spark-sql-engine_2.12 -am
+ /Users/wforget/work/git/kyuubi/build/mvn install -DskipTests -Dmaven.source.skip -Pspark-provided -Pflink-provided -Phive-provided -Pspark-3.5,scala-2.12 -pl :kyuubi-spark-sql-engine_2.12 -am
Using `mvn` from path: /Users/wforget/work/git/kyuubi/build/apache-maven-3.9.14/bin/mvn
```
<img width="873" height="116" alt="image" src="https://github.com/user-attachments/assets/6c6f9059-475f-45f2-af1f-bb450a2c460d" />
### Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Codex:GPT-5.6
Closes #7614 from wForget/KYUUBI-7613.
Closes #7613
b146eb43b [wforget] Build Spark engines for both Scala versions
0d8c8f810 [wforget] Build extra Spark engine only for supported Spark versions
Authored-by: wforget <64...@qq.com>
Signed-off-by: Cheng Pan <ch...@apache.org>
---
build/dist | 23 ++++++++++++++++-------
1 file changed, 16 insertions(+), 7 deletions(-)
diff --git a/build/dist b/build/dist
index 99da17c94d..4842cc390b 100755
--- a/build/dist
+++ b/build/dist
@@ -250,15 +250,24 @@ echo -e "\$ ${BUILD_COMMAND[@]}\n"
"${BUILD_COMMAND[@]}"
FILTERED_ARGS=()
+EXTRA_SCALA_212_SPARK_PROFILE='spark-3.5'
# shellcheck disable=SC2045
for arg in "$@"; do
- if [[ $arg == *scala-2.12* ]]; then
- FILTERED_ARGS+=("${arg//scala-2.12/scala-2.13}")
- elif [[ $arg == *scala-2.13* ]]; then
- FILTERED_ARGS+=("${arg//scala-2.13/scala-2.12}")
- else
- FILTERED_ARGS+=("$arg")
- fi
+ FILTERED_ARG="$arg"
+ if [[ $arg == *scala-2.12* ]]; then
+ FILTERED_ARG="${FILTERED_ARG//scala-2.12/scala-2.13}"
+ elif [[ $arg == *scala-2.13* ]]; then
+ FILTERED_ARG="${FILTERED_ARG//scala-2.13/scala-2.12}"
+ fi
+
+ # Keep the Scala 2.12 engine compatible with Spark 3.5 when building against Spark 4.
+ if [[ "$SCALA_VERSION" == "2.13" && "$SPARK_VERSION" == 4.* ]]; then
+ while [[ $FILTERED_ARG =~ spark-4\.[0-9]+ ]]; do
+ FILTERED_ARG="${FILTERED_ARG/${BASH_REMATCH[0]}/$EXTRA_SCALA_212_SPARK_PROFILE}"
+ done
+ FILTERED_ARG="${FILTERED_ARG//spark-master/$EXTRA_SCALA_212_SPARK_PROFILE}"
+ fi
+ FILTERED_ARGS+=("$FILTERED_ARG")
done
# shellcheck disable=SC2050