You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by fe...@apache.org on 2023/10/16 15:47:15 UTC
[kyuubi] branch branch-1.8 updated: [KYUUBI #5196][FOLLOWUP] Extract spark core scala version lazily and respect engine env
This is an automated email from the ASF dual-hosted git repository.
feiwang pushed a commit to branch branch-1.8
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.8 by this push:
new e66ef4540 [KYUUBI #5196][FOLLOWUP] Extract spark core scala version lazily and respect engine env
e66ef4540 is described below
commit e66ef4540131778f958675580dd5abee87569cdb
Author: fwang12 <fw...@ebay.com>
AuthorDate: Mon Oct 16 23:46:55 2023 +0800
[KYUUBI #5196][FOLLOWUP] Extract spark core scala version lazily and respect engine env
### _Why are the changes needed?_
Only extract the spark core scala version if `SPARK_SCALA_VERSION` env is empty, and respect engine env.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes #5434 from turboFei/lazy_scala_version.
Closes #5196
fdccef77b [fwang12] lazy extract spark core scala version
Authored-by: fwang12 <fw...@ebay.com>
Signed-off-by: fwang12 <fw...@ebay.com>
(cherry picked from commit c60f5b7e8872efb15597b962da1873a4ba2e71ef)
Signed-off-by: fwang12 <fw...@ebay.com>
---
.../scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
index 02f4064af..afc96fb5e 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
@@ -108,9 +108,9 @@ class SparkProcessBuilder(
}
override protected val engineScalaBinaryVersion: String = {
- val sparkCoreScalaVersion =
+ env.get("SPARK_SCALA_VERSION").filter(StringUtils.isNotBlank).getOrElse {
extractSparkCoreScalaVersion(Paths.get(sparkHome, "jars").toFile.list())
- StringUtils.defaultIfBlank(System.getenv("SPARK_SCALA_VERSION"), sparkCoreScalaVersion)
+ }
}
override protected lazy val engineHomeDirFilter: FileFilter = file => {