[jira] [Created] (ZEPPELIN-3633) ZeppelinContext Not Found in yarn-cluster Mode

["Chris Penny (JIRA)" <ji...@apache.org>]

Chris Penny created ZEPPELIN-3633:
-------------------------------------

             Summary: ZeppelinContext Not Found in yarn-cluster Mode
                 Key: ZEPPELIN-3633
                 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3633
             Project: Zeppelin
          Issue Type: Bug
          Components: zeppelin-interpreter
    Affects Versions: 0.8.0
            Reporter: Chris Penny


Hi all, Thanks for the 0.8.0 release!

We’re keen to take advantage of the yarn-cluster support to take the pressure off our Zeppelin host. However, I am having some trouble with it. The first problem was in following the documentation here:

[https://zeppelin.apache.org/docs/0.8.0/interpreter/spark.html]

This suggests that we need to modify the master configuration from “yarn-client” to “yarn-cluster”.

However, doing so results in the following error:
Warning: Master yarn-cluster is deprecated since 2.0. Please use master “yarn” with specified deploy mode instead. 
Error: Client deploy mode is not compatible with master “yarn-cluster” Run with --help for usage help or --verbose for debug output
 <stacktrace>


I got past this error with the following settings: 
master = yarn 
spark.submit.deployMode = cluster 

I’m somewhat unclear if I’m straying from the correct (documented) configuration or if the documentation needs an update. Anyway; These settings appear to work for everything except the ZeppelinContext, which is missing.
 Code: 
%spark 
z

Output: 
<console>:24: error: not found: value z

Using yarn-client mode I can identify that z is meant to be an instance of org.apache.zeppelin.spark.SparkZeppelinContext 
Code: 
%spark
 z

Output: 
res4: org.apache.zeppelin.spark.SparkZeppelinContext = org.apache.zeppelin.spark.SparkZeppelinContext@5b9282e1

However, this class is absent in cluster-mode: 
Code: %spark 
org.apache.zeppelin.spark.SparkZeppelinContext

Output:
 <console>:24: error: object zeppelin is not a member of package org.apache org.apache.zeppelin.spark.SparkZeppelinContext 
                   ^

Snooping around the Zeppelin installation I was able to locate this class in ${ZEPPELIN_INSTALL_DIR}/interpreter/spark/spark-interpreter-0.8.0.jar. I then uploaded this jar to HDFS and added it to spark.jars & spark.driver.extraClassPath. Relevant entries in driver log:
… 
Added JAR hdfs:/spark-interpreter-0.8.0.jar at hdfs:/tmp/zeppelin/spark-interpreter-0.8.0.jar with timestamp 1531732774379 
… 
CLASSPATH -> …:hdfs:/tmp/zeppelin/spark-interpreter-0.8.0.jar …
 … 
command: … file:$PWD/spark-interpreter-0.8.0.jar \ 
etc.

However, I still can’t use the ZeppelinContext or  org.apache.zeppelin.spark.SparkZeppelinContext class. At this point I’ve run out of ideas and am willing to ask for help.

Does anyone have thoughts on how I could use the ZeppelinContext in yarn cluster mode?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)