[zeppelin] branch branch-0.9 updated: [ZEPPELIN-5446] SparkInterpreterLauncher#detectSparkScalaVersion may return wrong scala version

[zj...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

zjffdu pushed a commit to branch branch-0.9
in repository https://gitbox.apache.org/repos/asf/zeppelin.git


The following commit(s) were added to refs/heads/branch-0.9 by this push:
     new 26d260c  [ZEPPELIN-5446] SparkInterpreterLauncher#detectSparkScalaVersion may return wrong scala version
26d260c is described below

commit 26d260cbad32544ad42d0cc3285d62b3a132c6f9
Author: Jeff Zhang <zj...@apache.org>
AuthorDate: Sat Jul 10 15:16:20 2021 +0800

    [ZEPPELIN-5446] SparkInterpreterLauncher#detectSparkScalaVersion may return wrong scala version
    
    ### What is this PR for?
    
    This issue happens when user set `SPARK_HOME` in zeppelin spark interpreter setting, but zeppelin server host has `SPARK_CONF_DIR` which point to another version of spark.  In this case, even when user set `SPARK_CONF_DIR` explicitly, `SparkInterpreterLauncher#detectSparkScalaVersion` would still return the scala version of the spark version of `SPARK_CONF_DIR` point to.
    
    This PR would pass the envs built from interpreter setting to the process of `detectSparkScalaVersion`
    
    ### What type of PR is it?
    [Bug Fix ]
    
    ### Todos
    * [ ] - Task
    
    ### What is the Jira issue?
    * https://issues.apache.org/jira/browse/ZEPPELIN-5446
    
    ### How should this be tested?
    * Manually tested
    
    ### Screenshots (if appropriate)
    
    ### Questions:
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No
    
    Author: Jeff Zhang <zj...@apache.org>
    
    Closes #4161 from zjffdu/ZEPPELIN-5446 and squashes the following commits:
    
    ce453d8f0e [Jeff Zhang] rename envs to env
    95aa5a6393 [Jeff Zhang] [ZEPPELIN-5446] SparkInterpreterLauncher#detectSparkScalaVersion may return wrong scala version
    
    (cherry picked from commit 80228c35905ebd16b7c80ce40f5b2d4f67180160)
    Signed-off-by: Jeff Zhang <zj...@apache.org>
---
 .../zeppelin/interpreter/launcher/SparkInterpreterLauncher.java    | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java
index 440b249..d0a9aa8 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java
@@ -133,7 +133,7 @@ public class SparkInterpreterLauncher extends StandardInterpreterLauncher {
           }
         }
 
-        String scalaVersion = detectSparkScalaVersion(getEnv("SPARK_HOME"));
+        String scalaVersion = detectSparkScalaVersion(getEnv("SPARK_HOME"), env);
         Path scalaFolder =  Paths.get(zConf.getZeppelinHome(), "/interpreter/spark/scala-" + scalaVersion);
         if (!scalaFolder.toFile().exists()) {
           throw new IOException("spark scala folder " + scalaFolder.toFile() + " doesn't exist");
@@ -241,11 +241,12 @@ public class SparkInterpreterLauncher extends StandardInterpreterLauncher {
     }
     LOGGER.info("buildEnvFromProperties: {}", env);
     return env;
-
   }
 
-  private String detectSparkScalaVersion(String sparkHome) throws Exception {
+  private String detectSparkScalaVersion(String sparkHome, Map<String, String> env) throws Exception {
+    LOGGER.info("Detect scala version from SPARK_HOME: {}", sparkHome);
     ProcessBuilder builder = new ProcessBuilder(sparkHome + "/bin/spark-submit", "--version");
+    builder.environment().putAll(env);
     File processOutputFile = File.createTempFile("zeppelin-spark", ".out");
     builder.redirectError(processOutputFile);
     Process process = builder.start();