You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by zj...@apache.org on 2021/07/12 09:14:12 UTC
[zeppelin] branch master updated: [ZEPPELIN-5446]
SparkInterpreterLauncher#detectSparkScalaVersion may return wrong scala
version
This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 80228c3 [ZEPPELIN-5446] SparkInterpreterLauncher#detectSparkScalaVersion may return wrong scala version
80228c3 is described below
commit 80228c35905ebd16b7c80ce40f5b2d4f67180160
Author: Jeff Zhang <zj...@apache.org>
AuthorDate: Sat Jul 10 15:16:20 2021 +0800
[ZEPPELIN-5446] SparkInterpreterLauncher#detectSparkScalaVersion may return wrong scala version
### What is this PR for?
This issue happens when user set `SPARK_HOME` in zeppelin spark interpreter setting, but zeppelin server host has `SPARK_CONF_DIR` which point to another version of spark. In this case, even when user set `SPARK_CONF_DIR` explicitly, `SparkInterpreterLauncher#detectSparkScalaVersion` would still return the scala version of the spark version of `SPARK_CONF_DIR` point to.
This PR would pass the envs built from interpreter setting to the process of `detectSparkScalaVersion`
### What type of PR is it?
[Bug Fix ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-5446
### How should this be tested?
* Manually tested
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zj...@apache.org>
Closes #4161 from zjffdu/ZEPPELIN-5446 and squashes the following commits:
ce453d8f0e [Jeff Zhang] rename envs to env
95aa5a6393 [Jeff Zhang] [ZEPPELIN-5446] SparkInterpreterLauncher#detectSparkScalaVersion may return wrong scala version
---
.../zeppelin/interpreter/launcher/SparkInterpreterLauncher.java | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java
index 440b249..d0a9aa8 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java
@@ -133,7 +133,7 @@ public class SparkInterpreterLauncher extends StandardInterpreterLauncher {
}
}
- String scalaVersion = detectSparkScalaVersion(getEnv("SPARK_HOME"));
+ String scalaVersion = detectSparkScalaVersion(getEnv("SPARK_HOME"), env);
Path scalaFolder = Paths.get(zConf.getZeppelinHome(), "/interpreter/spark/scala-" + scalaVersion);
if (!scalaFolder.toFile().exists()) {
throw new IOException("spark scala folder " + scalaFolder.toFile() + " doesn't exist");
@@ -241,11 +241,12 @@ public class SparkInterpreterLauncher extends StandardInterpreterLauncher {
}
LOGGER.info("buildEnvFromProperties: {}", env);
return env;
-
}
- private String detectSparkScalaVersion(String sparkHome) throws Exception {
+ private String detectSparkScalaVersion(String sparkHome, Map<String, String> env) throws Exception {
+ LOGGER.info("Detect scala version from SPARK_HOME: {}", sparkHome);
ProcessBuilder builder = new ProcessBuilder(sparkHome + "/bin/spark-submit", "--version");
+ builder.environment().putAll(env);
File processOutputFile = File.createTempFile("zeppelin-spark", ".out");
builder.redirectError(processOutputFile);
Process process = builder.start();