[zeppelin] branch branch-0.10 updated: [ZEPPELIN-5499] NPE in IPySparkInterpreter

[zj...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

zjffdu pushed a commit to branch branch-0.10
in repository https://gitbox.apache.org/repos/asf/zeppelin.git


The following commit(s) were added to refs/heads/branch-0.10 by this push:
     new da038e4  [ZEPPELIN-5499] NPE in IPySparkInterpreter
da038e4 is described below

commit da038e4a457e801861365e9eef9e586188edfe4d
Author: Jeff Zhang <zj...@apache.org>
AuthorDate: Fri Aug 27 12:24:19 2021 +0800

    [ZEPPELIN-5499] NPE in IPySparkInterpreter
    
    ### What is this PR for?
    
    The root cause is the `SparkConf` in PySparkInterpreter is null when IPySparkInterpreter#open is called because SparkInterpreter is not opened yet at that time. This PR would pass `SparkConf` explicitly to avoid this issue.
    
    ### What type of PR is it?
    [Bug Fix ]
    
    ### Todos
    * [ ] - Task
    
    ### What is the Jira issue?
    * https://issues.apache.org/jira/browse/ZEPPELIN-5499
    
    ### How should this be tested?
    * CI pass
    
    ### Screenshots (if appropriate)
    
    ### Questions:
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No
    
    Author: Jeff Zhang <zj...@apache.org>
    
    Closes #4214 from zjffdu/ZEPPELIN-5499 and squashes the following commits:
    
    5a4a7a6ae5 [Jeff Zhang] [ZEPPELIN-5499] NPE in IPySparkInterpreter
    
    (cherry picked from commit f6a2d33ecdd9436978ad93b5d00b3d8d2749182c)
    Signed-off-by: Jeff Zhang <zj...@apache.org>
---
 .../main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java    | 6 ++++--
 .../src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java | 4 ++++
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
index e35dcd6..0e3729f 100644
--- a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
+++ b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
@@ -53,10 +53,12 @@ public class IPySparkInterpreter extends IPythonInterpreter {
     if (opened) {
       return;
     }
+
+    this.sparkInterpreter = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class);
     PySparkInterpreter pySparkInterpreter =
             getInterpreterInTheSameSessionByClassName(PySparkInterpreter.class, false);
-    setProperty("zeppelin.python", pySparkInterpreter.getPythonExec());
-    sparkInterpreter = getInterpreterInTheSameSessionByClassName(SparkInterpreter.class);
+    setProperty("zeppelin.python", pySparkInterpreter.getPythonExec(sparkInterpreter.getSparkContext().conf()));
+
     setProperty("zeppelin.py4j.useAuth",
             sparkInterpreter.getSparkVersion().isSecretSocketSupported() + "");
     SparkConf conf = sparkInterpreter.getSparkContext().getConf();
diff --git a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
index ca4369d..2fdc37b 100644
--- a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
+++ b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
@@ -172,6 +172,10 @@ public class PySparkInterpreter extends PythonInterpreter {
   @Override
   protected String getPythonExec() {
     SparkConf sparkConf = getSparkConf();
+    return getPythonExec(sparkConf);
+  }
+
+  String getPythonExec(SparkConf sparkConf) {
     if (StringUtils.isNotBlank(sparkConf.get("spark.pyspark.driver.python", ""))) {
       return sparkConf.get("spark.pyspark.driver.python");
     }