You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Hyung Sung Shim <hs...@nflabs.com> on 2016/07/01 12:08:34 UTC
Re: classnotfoundexception using zeppelin with spark authentication
Hi Jonathan.
Unfortunately I got same error in my test bed.
Do you mind create an jira issue for this?
2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <jo...@gmail.com>:
> I added this and still get the same exception. The same property is also
> set in spark-defaults.conf.
>
> After that didn't work, I also tried adding --conf spark.authenticate=true
> and --conf spark.authenticate.enableSaslEncryption=true, to match the other
> related settings in spark-defaults.conf. Still get the same
> classnotfoundexception.
>
>
> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hs...@nflabs.com>
> wrote:
>
>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
>> zeppelin and retry your code.
>>
>>
>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterhazy@gmail.com>:
>>
>>> yes it does. i only see this problem in zeppelin.
>>>
>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hs...@nflabs.com>
>>> wrote:
>>>
>>>> Hi Jonathan.
>>>> It's not easy to build the test environments but I am working on this.
>>>> I have question for you.
>>>> Does your code working well on spark-shell in the spark.authenticate
>>>> mode?
>>>>
>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>>>> jonathan.esterhazy@gmail.com>:
>>>>
>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>>>>> quite stumped.
>>>>>
>>>>> thx.
>>>>>
>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hs...@nflabs.com>
>>>>> wrote:
>>>>>
>>>>>> Thank you.
>>>>>> Let me try.
>>>>>>
>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>
>>>>>>> Hyung,
>>>>>>>
>>>>>>> Yes, here they are.
>>>>>>>
>>>>>>> zeppelin-env.sh:
>>>>>>>
>>>>>>> export ZEPPELIN_PORT=8890
>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>>>> export MASTER=yarn-client
>>>>>>> export SPARK_HOME=/usr/lib/spark
>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>> export
>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>>>> export
>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>>>>>
>>>>>>> spark-defaults.conf:
>>>>>>>
>>>>>>> spark.master yarn
>>>>>>> spark.driver.extraClassPath
>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>> spark.driver.extraLibraryPath
>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>> spark.executor.extraClassPath
>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>> spark.executor.extraLibraryPath
>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>> spark.eventLog.enabled true
>>>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps
>>>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps
>>>>>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
>>>>>>> spark.history.ui.port 18080
>>>>>>> spark.shuffle.service.enabled true
>>>>>>> spark.driver.extraJavaOptions
>>>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>>>>>> -XX:MaxHeapFreeRatio=70
>>>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>>>>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>> spark.dynamicAllocation.enabled true
>>>>>>> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails
>>>>>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>> spark.executor.memory 8640m
>>>>>>> spark.executor.cores 7
>>>>>>> spark.authenticate.enableSaslEncryption true
>>>>>>> spark.driver.memory 1g
>>>>>>> spark.network.sasl.serverAlwaysEncrypt true
>>>>>>> spark.driver.cores 1
>>>>>>> spark.ssl.protocol TLSv1.2
>>>>>>> spark.ssl.keyStorePassword password
>>>>>>> spark.yarn.maxAppAttempts 1
>>>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks
>>>>>>> spark.authenticate true
>>>>>>> spark.ssl.keyPassword password
>>>>>>> spark.ssl.enabled true
>>>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA
>>>>>>> spark.ssl.trustStore /etc/emr/security/truststore.jks
>>>>>>> spark.authenticate.secret secret
>>>>>>> spark.ssl.trustStorePassword password
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <hs...@nflabs.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi.
>>>>>>>> Could you share your conf/zeppelin-env.sh and spark-defaults.conf ?
>>>>>>>>
>>>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>
>>>>>>>>> I am having trouble using zeppelin in a spark cluster that has
>>>>>>>>> spark node authentication turned on (e.g. with spark.authenticate=true,
>>>>>>>>> spark.authenticate.secret=...)
>>>>>>>>>
>>>>>>>>> Notebook code that calls built-in spark functions (or other things
>>>>>>>>> on executor classpath) work fine, but functions defined in the notebook
>>>>>>>>> (anonymous or named) throw ClassNotFoundExceptions when called from an
>>>>>>>>> executor.
>>>>>>>>>
>>>>>>>>> For example, this code works:
>>>>>>>>>
>>>>>>>>> val rdd = sc.textFile("hdfs://my-text-file")
>>>>>>>>> rdd.take(1).foreach(println)
>>>>>>>>>
>>>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>>>>>>>>>
>>>>>>>>> but code like this...
>>>>>>>>>
>>>>>>>>> rdd.filter(_.contains("my data"))
>>>>>>>>>
>>>>>>>>> fails with
>>>>>>>>>
>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>>>>>>>>> at
>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I get the same kind of error if the filter function is defined as
>>>>>>>>> a named function in the notebook, or as a member of singleton object
>>>>>>>>> defined in the notebook.
>>>>>>>>>
>>>>>>>>> When I look at the executor's log output, I see this error:
>>>>>>>>>
>>>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to check
>>>>>>>>> existence of class
>>>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL
>>>>>>>>> class server at https://172.30.54.30:34980
>>>>>>>>> java.lang.NullPointerException
>>>>>>>>> at
>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
>>>>>>>>> at
>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If I disable spark authentication, everything works as expected. I
>>>>>>>>> am running zeppelin 0.5.6 on spark 1.6.1 with yarn.
>>>>>>>>>
>>>>>>>>> Has anyone been able to get zeppelin working with spark
>>>>>>>>> authentication?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: classnotfoundexception using zeppelin with spark authentication
Posted by Jeff Zhang <zj...@gmail.com>.
I think it is safe to merge into 0.6.1
On Fri, Jul 29, 2016 at 1:08 AM, Jonathan Esterhazy <
jonathan.esterhazy@gmail.com> wrote:
> yes, this PR fixes my classnotfoundexceptions. awesome!
>
> any idea when it might make it into a release?
>
> On Wed, Jul 27, 2016 at 11:25 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>> I think it is fixed in https://github.com/apache/zeppelin/pull/1237,
>> please try that.
>>
>> On Thu, Jul 28, 2016 at 1:58 AM, Jonathan Esterhazy <
>> jonathan.esterhazy@gmail.com> wrote:
>>
>>> I just tried this with using zeppelin master, and the classnotfound
>>> error is still there.
>>>
>>>
>>> On Thu, Jul 7, 2016 at 7:07 AM, Hyung Sung Shim <hs...@nflabs.com>
>>> wrote:
>>>
>>>> Hi. Jonathan.
>>>> If it is possible, could you use the zeppelin master on the github?
>>>> Zeppelin is working well with spark authentication when i tested.
>>>>
>>>>
>>>>
>>>> 2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy <
>>>> jonathan.esterhazy@gmail.com>:
>>>>
>>>>> Hyung, thx for your help. I opened these:
>>>>>
>>>>> https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala
>>>>> problem)
>>>>> https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking
>>>>> python problem)
>>>>>
>>>>> LMK if I can provide more info or help in some way.
>>>>>
>>>>> On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim <hs...@nflabs.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Jonathan.
>>>>>> Unfortunately I got same error in my test bed.
>>>>>> Do you mind create an jira issue for this?
>>>>>>
>>>>>> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <
>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>
>>>>>>> I added this and still get the same exception. The same property is
>>>>>>> also set in spark-defaults.conf.
>>>>>>>
>>>>>>> After that didn't work, I also tried adding --conf
>>>>>>> spark.authenticate=true and --conf
>>>>>>> spark.authenticate.enableSaslEncryption=true, to match the other related
>>>>>>> settings in spark-defaults.conf. Still get the same classnotfoundexception.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hsshim@nflabs.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>>>>>>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and
>>>>>>>> restart zeppelin and retry your code.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>
>>>>>>>>> yes it does. i only see this problem in zeppelin.
>>>>>>>>>
>>>>>>>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <
>>>>>>>>> hsshim@nflabs.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Jonathan.
>>>>>>>>>> It's not easy to build the test environments but I am working on
>>>>>>>>>> this.
>>>>>>>>>> I have question for you.
>>>>>>>>>> Does your code working well on spark-shell in the
>>>>>>>>>> spark.authenticate mode?
>>>>>>>>>>
>>>>>>>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication?
>>>>>>>>>>> I'm quite stumped.
>>>>>>>>>>>
>>>>>>>>>>> thx.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <
>>>>>>>>>>> hsshim@nflabs.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thank you.
>>>>>>>>>>>> Let me try.
>>>>>>>>>>>>
>>>>>>>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hyung,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, here they are.
>>>>>>>>>>>>>
>>>>>>>>>>>>> zeppelin-env.sh:
>>>>>>>>>>>>>
>>>>>>>>>>>>> export ZEPPELIN_PORT=8890
>>>>>>>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>>>>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>>>>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>>>>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>>>>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>>>>>>>>>> export MASTER=yarn-client
>>>>>>>>>>>>> export SPARK_HOME=/usr/lib/spark
>>>>>>>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>>>>>>>> export
>>>>>>>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>>>>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>>>>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>>>>>>>>>> export
>>>>>>>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>>>>>>>>>>>
>>>>>>>>>>>>> spark-defaults.conf:
>>>>>>>>>>>>>
>>>>>>>>>>>>> spark.master yarn
>>>>>>>>>>>>> spark.driver.extraClassPath
>>>>>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>>>>>>>
>>>>>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>>>>>>> spark.driver.extraLibraryPath
>>>>>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>>>>>>> spark.executor.extraClassPath
>>>>>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>>>>>>>
>>>>>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>>>>>>> spark.executor.extraLibraryPath
>>>>>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>>>>>>> spark.eventLog.enabled true
>>>>>>>>>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps
>>>>>>>>>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps
>>>>>>>>>>>>> spark.yarn.historyServer.address
>>>>>>>>>>>>> ip-172-30-54-30.ec2.internal:18080
>>>>>>>>>>>>> spark.history.ui.port 18080
>>>>>>>>>>>>> spark.shuffle.service.enabled true
>>>>>>>>>>>>> spark.driver.extraJavaOptions
>>>>>>>>>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>>>>>>>>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>>>>>>>>>>>> -XX:MaxHeapFreeRatio=70
>>>>>>>>>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>>>>>>>>>>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>>>>>>> spark.dynamicAllocation.enabled true
>>>>>>>>>>>>> spark.executor.extraJavaOptions -verbose:gc
>>>>>>>>>>>>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>>>>>>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>>>>>>>>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>>>>>>> spark.executor.memory 8640m
>>>>>>>>>>>>> spark.executor.cores 7
>>>>>>>>>>>>> spark.authenticate.enableSaslEncryption true
>>>>>>>>>>>>> spark.driver.memory 1g
>>>>>>>>>>>>> spark.network.sasl.serverAlwaysEncrypt true
>>>>>>>>>>>>> spark.driver.cores 1
>>>>>>>>>>>>> spark.ssl.protocol TLSv1.2
>>>>>>>>>>>>> spark.ssl.keyStorePassword password
>>>>>>>>>>>>> spark.yarn.maxAppAttempts 1
>>>>>>>>>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks
>>>>>>>>>>>>> spark.authenticate true
>>>>>>>>>>>>> spark.ssl.keyPassword password
>>>>>>>>>>>>> spark.ssl.enabled true
>>>>>>>>>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA
>>>>>>>>>>>>> spark.ssl.trustStore
>>>>>>>>>>>>> /etc/emr/security/truststore.jks
>>>>>>>>>>>>> spark.authenticate.secret secret
>>>>>>>>>>>>> spark.ssl.trustStorePassword password
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <
>>>>>>>>>>>>> hsshim@nflabs.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>> Could you share your conf/zeppelin-env.sh
>>>>>>>>>>>>>> and spark-defaults.conf ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am having trouble using zeppelin in a spark cluster that
>>>>>>>>>>>>>>> has spark node authentication turned on (e.g. with spark.authenticate=true,
>>>>>>>>>>>>>>> spark.authenticate.secret=...)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Notebook code that calls built-in spark functions (or other
>>>>>>>>>>>>>>> things on executor classpath) work fine, but functions defined in the
>>>>>>>>>>>>>>> notebook (anonymous or named) throw ClassNotFoundExceptions when called
>>>>>>>>>>>>>>> from an executor.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For example, this code works:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> val rdd = sc.textFile("hdfs://my-text-file")
>>>>>>>>>>>>>>> rdd.take(1).foreach(println)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> but code like this...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> rdd.filter(_.contains("my data"))
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> fails with
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
>>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I get the same kind of error if the filter function is
>>>>>>>>>>>>>>> defined as a named function in the notebook, or as a member of singleton
>>>>>>>>>>>>>>> object defined in the notebook.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When I look at the executor's log output, I see this error:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to
>>>>>>>>>>>>>>> check existence of class
>>>>>>>>>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL
>>>>>>>>>>>>>>> class server at https://172.30.54.30:34980
>>>>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
>>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If I disable spark authentication, everything works as
>>>>>>>>>>>>>>> expected. I am running zeppelin 0.5.6 on spark 1.6.1 with yarn.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Has anyone been able to get zeppelin working with spark
>>>>>>>>>>>>>>> authentication?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>
--
Best Regards
Jeff Zhang
Re: classnotfoundexception using zeppelin with spark authentication
Posted by Jonathan Esterhazy <jo...@gmail.com>.
yes, this PR fixes my classnotfoundexceptions. awesome!
any idea when it might make it into a release?
On Wed, Jul 27, 2016 at 11:25 PM, Jeff Zhang <zj...@gmail.com> wrote:
> I think it is fixed in https://github.com/apache/zeppelin/pull/1237,
> please try that.
>
> On Thu, Jul 28, 2016 at 1:58 AM, Jonathan Esterhazy <
> jonathan.esterhazy@gmail.com> wrote:
>
>> I just tried this with using zeppelin master, and the classnotfound error
>> is still there.
>>
>>
>> On Thu, Jul 7, 2016 at 7:07 AM, Hyung Sung Shim <hs...@nflabs.com>
>> wrote:
>>
>>> Hi. Jonathan.
>>> If it is possible, could you use the zeppelin master on the github?
>>> Zeppelin is working well with spark authentication when i tested.
>>>
>>>
>>>
>>> 2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterhazy@gmail.com>:
>>>
>>>> Hyung, thx for your help. I opened these:
>>>>
>>>> https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala
>>>> problem)
>>>> https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking
>>>> python problem)
>>>>
>>>> LMK if I can provide more info or help in some way.
>>>>
>>>> On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim <hs...@nflabs.com>
>>>> wrote:
>>>>
>>>>> Hi Jonathan.
>>>>> Unfortunately I got same error in my test bed.
>>>>> Do you mind create an jira issue for this?
>>>>>
>>>>> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <
>>>>> jonathan.esterhazy@gmail.com>:
>>>>>
>>>>>> I added this and still get the same exception. The same property is
>>>>>> also set in spark-defaults.conf.
>>>>>>
>>>>>> After that didn't work, I also tried adding --conf
>>>>>> spark.authenticate=true and --conf
>>>>>> spark.authenticate.enableSaslEncryption=true, to match the other related
>>>>>> settings in spark-defaults.conf. Still get the same classnotfoundexception.
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hs...@nflabs.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>>>>>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and
>>>>>>> restart zeppelin and retry your code.
>>>>>>>
>>>>>>>
>>>>>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>
>>>>>>>> yes it does. i only see this problem in zeppelin.
>>>>>>>>
>>>>>>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hsshim@nflabs.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi Jonathan.
>>>>>>>>> It's not easy to build the test environments but I am working on
>>>>>>>>> this.
>>>>>>>>> I have question for you.
>>>>>>>>> Does your code working well on spark-shell in the
>>>>>>>>> spark.authenticate mode?
>>>>>>>>>
>>>>>>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>
>>>>>>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication?
>>>>>>>>>> I'm quite stumped.
>>>>>>>>>>
>>>>>>>>>> thx.
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <
>>>>>>>>>> hsshim@nflabs.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thank you.
>>>>>>>>>>> Let me try.
>>>>>>>>>>>
>>>>>>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> Hyung,
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, here they are.
>>>>>>>>>>>>
>>>>>>>>>>>> zeppelin-env.sh:
>>>>>>>>>>>>
>>>>>>>>>>>> export ZEPPELIN_PORT=8890
>>>>>>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>>>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>>>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>>>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>>>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>>>>>>>>> export MASTER=yarn-client
>>>>>>>>>>>> export SPARK_HOME=/usr/lib/spark
>>>>>>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>>>>>>> export
>>>>>>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>>>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>>>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>>>>>>>>> export
>>>>>>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>>>>>>>>>>
>>>>>>>>>>>> spark-defaults.conf:
>>>>>>>>>>>>
>>>>>>>>>>>> spark.master yarn
>>>>>>>>>>>> spark.driver.extraClassPath
>>>>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>>>>>>
>>>>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>>>>>> spark.driver.extraLibraryPath
>>>>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>>>>>> spark.executor.extraClassPath
>>>>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>>>>>>
>>>>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>>>>>> spark.executor.extraLibraryPath
>>>>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>>>>>> spark.eventLog.enabled true
>>>>>>>>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps
>>>>>>>>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps
>>>>>>>>>>>> spark.yarn.historyServer.address
>>>>>>>>>>>> ip-172-30-54-30.ec2.internal:18080
>>>>>>>>>>>> spark.history.ui.port 18080
>>>>>>>>>>>> spark.shuffle.service.enabled true
>>>>>>>>>>>> spark.driver.extraJavaOptions
>>>>>>>>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>>>>>>>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>>>>>>>>>>> -XX:MaxHeapFreeRatio=70
>>>>>>>>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>>>>>>>>>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>>>>>> spark.dynamicAllocation.enabled true
>>>>>>>>>>>> spark.executor.extraJavaOptions -verbose:gc
>>>>>>>>>>>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>>>>>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>>>>>>>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>>>>>> spark.executor.memory 8640m
>>>>>>>>>>>> spark.executor.cores 7
>>>>>>>>>>>> spark.authenticate.enableSaslEncryption true
>>>>>>>>>>>> spark.driver.memory 1g
>>>>>>>>>>>> spark.network.sasl.serverAlwaysEncrypt true
>>>>>>>>>>>> spark.driver.cores 1
>>>>>>>>>>>> spark.ssl.protocol TLSv1.2
>>>>>>>>>>>> spark.ssl.keyStorePassword password
>>>>>>>>>>>> spark.yarn.maxAppAttempts 1
>>>>>>>>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks
>>>>>>>>>>>> spark.authenticate true
>>>>>>>>>>>> spark.ssl.keyPassword password
>>>>>>>>>>>> spark.ssl.enabled true
>>>>>>>>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA
>>>>>>>>>>>> spark.ssl.trustStore
>>>>>>>>>>>> /etc/emr/security/truststore.jks
>>>>>>>>>>>> spark.authenticate.secret secret
>>>>>>>>>>>> spark.ssl.trustStorePassword password
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <
>>>>>>>>>>>> hsshim@nflabs.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>> Could you share your conf/zeppelin-env.sh
>>>>>>>>>>>>> and spark-defaults.conf ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am having trouble using zeppelin in a spark cluster that
>>>>>>>>>>>>>> has spark node authentication turned on (e.g. with spark.authenticate=true,
>>>>>>>>>>>>>> spark.authenticate.secret=...)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Notebook code that calls built-in spark functions (or other
>>>>>>>>>>>>>> things on executor classpath) work fine, but functions defined in the
>>>>>>>>>>>>>> notebook (anonymous or named) throw ClassNotFoundExceptions when called
>>>>>>>>>>>>>> from an executor.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For example, this code works:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> val rdd = sc.textFile("hdfs://my-text-file")
>>>>>>>>>>>>>> rdd.take(1).foreach(println)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> but code like this...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> rdd.filter(_.contains("my data"))
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> fails with
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I get the same kind of error if the filter function is
>>>>>>>>>>>>>> defined as a named function in the notebook, or as a member of singleton
>>>>>>>>>>>>>> object defined in the notebook.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When I look at the executor's log output, I see this error:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to
>>>>>>>>>>>>>> check existence of class
>>>>>>>>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL
>>>>>>>>>>>>>> class server at https://172.30.54.30:34980
>>>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If I disable spark authentication, everything works as
>>>>>>>>>>>>>> expected. I am running zeppelin 0.5.6 on spark 1.6.1 with yarn.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Has anyone been able to get zeppelin working with spark
>>>>>>>>>>>>>> authentication?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: classnotfoundexception using zeppelin with spark authentication
Posted by Jeff Zhang <zj...@gmail.com>.
I think it is fixed in https://github.com/apache/zeppelin/pull/1237, please
try that.
On Thu, Jul 28, 2016 at 1:58 AM, Jonathan Esterhazy <
jonathan.esterhazy@gmail.com> wrote:
> I just tried this with using zeppelin master, and the classnotfound error
> is still there.
>
>
> On Thu, Jul 7, 2016 at 7:07 AM, Hyung Sung Shim <hs...@nflabs.com> wrote:
>
>> Hi. Jonathan.
>> If it is possible, could you use the zeppelin master on the github?
>> Zeppelin is working well with spark authentication when i tested.
>>
>>
>>
>> 2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterhazy@gmail.com>:
>>
>>> Hyung, thx for your help. I opened these:
>>>
>>> https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala problem)
>>> https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking
>>> python problem)
>>>
>>> LMK if I can provide more info or help in some way.
>>>
>>> On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim <hs...@nflabs.com>
>>> wrote:
>>>
>>>> Hi Jonathan.
>>>> Unfortunately I got same error in my test bed.
>>>> Do you mind create an jira issue for this?
>>>>
>>>> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <
>>>> jonathan.esterhazy@gmail.com>:
>>>>
>>>>> I added this and still get the same exception. The same property is
>>>>> also set in spark-defaults.conf.
>>>>>
>>>>> After that didn't work, I also tried adding --conf
>>>>> spark.authenticate=true and --conf
>>>>> spark.authenticate.enableSaslEncryption=true, to match the other related
>>>>> settings in spark-defaults.conf. Still get the same classnotfoundexception.
>>>>>
>>>>>
>>>>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hs...@nflabs.com>
>>>>> wrote:
>>>>>
>>>>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>>>>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and
>>>>>> restart zeppelin and retry your code.
>>>>>>
>>>>>>
>>>>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>
>>>>>>> yes it does. i only see this problem in zeppelin.
>>>>>>>
>>>>>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hs...@nflabs.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Jonathan.
>>>>>>>> It's not easy to build the test environments but I am working on
>>>>>>>> this.
>>>>>>>> I have question for you.
>>>>>>>> Does your code working well on spark-shell in the
>>>>>>>> spark.authenticate mode?
>>>>>>>>
>>>>>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>
>>>>>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication?
>>>>>>>>> I'm quite stumped.
>>>>>>>>>
>>>>>>>>> thx.
>>>>>>>>>
>>>>>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <
>>>>>>>>> hsshim@nflabs.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thank you.
>>>>>>>>>> Let me try.
>>>>>>>>>>
>>>>>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> Hyung,
>>>>>>>>>>>
>>>>>>>>>>> Yes, here they are.
>>>>>>>>>>>
>>>>>>>>>>> zeppelin-env.sh:
>>>>>>>>>>>
>>>>>>>>>>> export ZEPPELIN_PORT=8890
>>>>>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>>>>>>>> export MASTER=yarn-client
>>>>>>>>>>> export SPARK_HOME=/usr/lib/spark
>>>>>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>>>>>> export
>>>>>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>>>>>>>> export
>>>>>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>>>>>>>>>
>>>>>>>>>>> spark-defaults.conf:
>>>>>>>>>>>
>>>>>>>>>>> spark.master yarn
>>>>>>>>>>> spark.driver.extraClassPath
>>>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>>>>> spark.driver.extraLibraryPath
>>>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>>>>> spark.executor.extraClassPath
>>>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>>>>> spark.executor.extraLibraryPath
>>>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>>>>> spark.eventLog.enabled true
>>>>>>>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps
>>>>>>>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps
>>>>>>>>>>> spark.yarn.historyServer.address
>>>>>>>>>>> ip-172-30-54-30.ec2.internal:18080
>>>>>>>>>>> spark.history.ui.port 18080
>>>>>>>>>>> spark.shuffle.service.enabled true
>>>>>>>>>>> spark.driver.extraJavaOptions
>>>>>>>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>>>>>>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>>>>>>>>>> -XX:MaxHeapFreeRatio=70
>>>>>>>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>>>>>>>>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>>>>> spark.dynamicAllocation.enabled true
>>>>>>>>>>> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails
>>>>>>>>>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>>>>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>>>>>>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>>>>> spark.executor.memory 8640m
>>>>>>>>>>> spark.executor.cores 7
>>>>>>>>>>> spark.authenticate.enableSaslEncryption true
>>>>>>>>>>> spark.driver.memory 1g
>>>>>>>>>>> spark.network.sasl.serverAlwaysEncrypt true
>>>>>>>>>>> spark.driver.cores 1
>>>>>>>>>>> spark.ssl.protocol TLSv1.2
>>>>>>>>>>> spark.ssl.keyStorePassword password
>>>>>>>>>>> spark.yarn.maxAppAttempts 1
>>>>>>>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks
>>>>>>>>>>> spark.authenticate true
>>>>>>>>>>> spark.ssl.keyPassword password
>>>>>>>>>>> spark.ssl.enabled true
>>>>>>>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA
>>>>>>>>>>> spark.ssl.trustStore /etc/emr/security/truststore.jks
>>>>>>>>>>> spark.authenticate.secret secret
>>>>>>>>>>> spark.ssl.trustStorePassword password
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <
>>>>>>>>>>> hsshim@nflabs.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi.
>>>>>>>>>>>> Could you share your conf/zeppelin-env.sh
>>>>>>>>>>>> and spark-defaults.conf ?
>>>>>>>>>>>>
>>>>>>>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>> I am having trouble using zeppelin in a spark cluster that has
>>>>>>>>>>>>> spark node authentication turned on (e.g. with spark.authenticate=true,
>>>>>>>>>>>>> spark.authenticate.secret=...)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Notebook code that calls built-in spark functions (or other
>>>>>>>>>>>>> things on executor classpath) work fine, but functions defined in the
>>>>>>>>>>>>> notebook (anonymous or named) throw ClassNotFoundExceptions when called
>>>>>>>>>>>>> from an executor.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For example, this code works:
>>>>>>>>>>>>>
>>>>>>>>>>>>> val rdd = sc.textFile("hdfs://my-text-file")
>>>>>>>>>>>>> rdd.take(1).foreach(println)
>>>>>>>>>>>>>
>>>>>>>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>>>>>>>>>>>>>
>>>>>>>>>>>>> but code like this...
>>>>>>>>>>>>>
>>>>>>>>>>>>> rdd.filter(_.contains("my data"))
>>>>>>>>>>>>>
>>>>>>>>>>>>> fails with
>>>>>>>>>>>>>
>>>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>>>> ...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I get the same kind of error if the filter function is defined
>>>>>>>>>>>>> as a named function in the notebook, or as a member of singleton object
>>>>>>>>>>>>> defined in the notebook.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When I look at the executor's log output, I see this error:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to
>>>>>>>>>>>>> check existence of class
>>>>>>>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL
>>>>>>>>>>>>> class server at https://172.30.54.30:34980
>>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>>>>
>>>>>>>>>>>>> ...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> If I disable spark authentication, everything works as
>>>>>>>>>>>>> expected. I am running zeppelin 0.5.6 on spark 1.6.1 with yarn.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Has anyone been able to get zeppelin working with spark
>>>>>>>>>>>>> authentication?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
Best Regards
Jeff Zhang
Re: classnotfoundexception using zeppelin with spark authentication
Posted by Jonathan Esterhazy <jo...@gmail.com>.
I just tried this with using zeppelin master, and the classnotfound error
is still there.
On Thu, Jul 7, 2016 at 7:07 AM, Hyung Sung Shim <hs...@nflabs.com> wrote:
> Hi. Jonathan.
> If it is possible, could you use the zeppelin master on the github?
> Zeppelin is working well with spark authentication when i tested.
>
>
>
> 2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy <jonathan.esterhazy@gmail.com
> >:
>
>> Hyung, thx for your help. I opened these:
>>
>> https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala problem)
>> https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking
>> python problem)
>>
>> LMK if I can provide more info or help in some way.
>>
>> On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim <hs...@nflabs.com>
>> wrote:
>>
>>> Hi Jonathan.
>>> Unfortunately I got same error in my test bed.
>>> Do you mind create an jira issue for this?
>>>
>>> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterhazy@gmail.com>:
>>>
>>>> I added this and still get the same exception. The same property is
>>>> also set in spark-defaults.conf.
>>>>
>>>> After that didn't work, I also tried adding --conf
>>>> spark.authenticate=true and --conf
>>>> spark.authenticate.enableSaslEncryption=true, to match the other related
>>>> settings in spark-defaults.conf. Still get the same classnotfoundexception.
>>>>
>>>>
>>>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hs...@nflabs.com>
>>>> wrote:
>>>>
>>>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>>>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and
>>>>> restart zeppelin and retry your code.
>>>>>
>>>>>
>>>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>>>>> jonathan.esterhazy@gmail.com>:
>>>>>
>>>>>> yes it does. i only see this problem in zeppelin.
>>>>>>
>>>>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hs...@nflabs.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Jonathan.
>>>>>>> It's not easy to build the test environments but I am working on
>>>>>>> this.
>>>>>>> I have question for you.
>>>>>>> Does your code working well on spark-shell in the spark.authenticate
>>>>>>> mode?
>>>>>>>
>>>>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>
>>>>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication?
>>>>>>>> I'm quite stumped.
>>>>>>>>
>>>>>>>> thx.
>>>>>>>>
>>>>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hsshim@nflabs.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>> Let me try.
>>>>>>>>>
>>>>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>
>>>>>>>>>> Hyung,
>>>>>>>>>>
>>>>>>>>>> Yes, here they are.
>>>>>>>>>>
>>>>>>>>>> zeppelin-env.sh:
>>>>>>>>>>
>>>>>>>>>> export ZEPPELIN_PORT=8890
>>>>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>>>>>>> export MASTER=yarn-client
>>>>>>>>>> export SPARK_HOME=/usr/lib/spark
>>>>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>>>>> export
>>>>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>>>>>>> export
>>>>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>>>>>>>>
>>>>>>>>>> spark-defaults.conf:
>>>>>>>>>>
>>>>>>>>>> spark.master yarn
>>>>>>>>>> spark.driver.extraClassPath
>>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>>>> spark.driver.extraLibraryPath
>>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>>>> spark.executor.extraClassPath
>>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>>>> spark.executor.extraLibraryPath
>>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>>>> spark.eventLog.enabled true
>>>>>>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps
>>>>>>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps
>>>>>>>>>> spark.yarn.historyServer.address
>>>>>>>>>> ip-172-30-54-30.ec2.internal:18080
>>>>>>>>>> spark.history.ui.port 18080
>>>>>>>>>> spark.shuffle.service.enabled true
>>>>>>>>>> spark.driver.extraJavaOptions
>>>>>>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>>>>>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>>>>>>>>> -XX:MaxHeapFreeRatio=70
>>>>>>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>>>>>>>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>>>> spark.dynamicAllocation.enabled true
>>>>>>>>>> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails
>>>>>>>>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>>>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>>>>>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>>>> spark.executor.memory 8640m
>>>>>>>>>> spark.executor.cores 7
>>>>>>>>>> spark.authenticate.enableSaslEncryption true
>>>>>>>>>> spark.driver.memory 1g
>>>>>>>>>> spark.network.sasl.serverAlwaysEncrypt true
>>>>>>>>>> spark.driver.cores 1
>>>>>>>>>> spark.ssl.protocol TLSv1.2
>>>>>>>>>> spark.ssl.keyStorePassword password
>>>>>>>>>> spark.yarn.maxAppAttempts 1
>>>>>>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks
>>>>>>>>>> spark.authenticate true
>>>>>>>>>> spark.ssl.keyPassword password
>>>>>>>>>> spark.ssl.enabled true
>>>>>>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA
>>>>>>>>>> spark.ssl.trustStore /etc/emr/security/truststore.jks
>>>>>>>>>> spark.authenticate.secret secret
>>>>>>>>>> spark.ssl.trustStorePassword password
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <
>>>>>>>>>> hsshim@nflabs.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi.
>>>>>>>>>>> Could you share your conf/zeppelin-env.sh
>>>>>>>>>>> and spark-defaults.conf ?
>>>>>>>>>>>
>>>>>>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> I am having trouble using zeppelin in a spark cluster that has
>>>>>>>>>>>> spark node authentication turned on (e.g. with spark.authenticate=true,
>>>>>>>>>>>> spark.authenticate.secret=...)
>>>>>>>>>>>>
>>>>>>>>>>>> Notebook code that calls built-in spark functions (or other
>>>>>>>>>>>> things on executor classpath) work fine, but functions defined in the
>>>>>>>>>>>> notebook (anonymous or named) throw ClassNotFoundExceptions when called
>>>>>>>>>>>> from an executor.
>>>>>>>>>>>>
>>>>>>>>>>>> For example, this code works:
>>>>>>>>>>>>
>>>>>>>>>>>> val rdd = sc.textFile("hdfs://my-text-file")
>>>>>>>>>>>> rdd.take(1).foreach(println)
>>>>>>>>>>>>
>>>>>>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>>>>>>>>>>>>
>>>>>>>>>>>> but code like this...
>>>>>>>>>>>>
>>>>>>>>>>>> rdd.filter(_.contains("my data"))
>>>>>>>>>>>>
>>>>>>>>>>>> fails with
>>>>>>>>>>>>
>>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>>> ...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I get the same kind of error if the filter function is defined
>>>>>>>>>>>> as a named function in the notebook, or as a member of singleton object
>>>>>>>>>>>> defined in the notebook.
>>>>>>>>>>>>
>>>>>>>>>>>> When I look at the executor's log output, I see this error:
>>>>>>>>>>>>
>>>>>>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to
>>>>>>>>>>>> check existence of class
>>>>>>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL
>>>>>>>>>>>> class server at https://172.30.54.30:34980
>>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>>>
>>>>>>>>>>>> ...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If I disable spark authentication, everything works as
>>>>>>>>>>>> expected. I am running zeppelin 0.5.6 on spark 1.6.1 with yarn.
>>>>>>>>>>>>
>>>>>>>>>>>> Has anyone been able to get zeppelin working with spark
>>>>>>>>>>>> authentication?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: classnotfoundexception using zeppelin with spark authentication
Posted by Hyung Sung Shim <hs...@nflabs.com>.
Hi. Jonathan.
If it is possible, could you use the zeppelin master on the github?
Zeppelin is working well with spark authentication when i tested.
2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy <jo...@gmail.com>:
> Hyung, thx for your help. I opened these:
>
> https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala problem)
> https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking
> python problem)
>
> LMK if I can provide more info or help in some way.
>
> On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim <hs...@nflabs.com> wrote:
>
>> Hi Jonathan.
>> Unfortunately I got same error in my test bed.
>> Do you mind create an jira issue for this?
>>
>> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterhazy@gmail.com>:
>>
>>> I added this and still get the same exception. The same property is also
>>> set in spark-defaults.conf.
>>>
>>> After that didn't work, I also tried adding --conf
>>> spark.authenticate=true and --conf
>>> spark.authenticate.enableSaslEncryption=true, to match the other related
>>> settings in spark-defaults.conf. Still get the same classnotfoundexception.
>>>
>>>
>>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hs...@nflabs.com>
>>> wrote:
>>>
>>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
>>>> zeppelin and retry your code.
>>>>
>>>>
>>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>>>> jonathan.esterhazy@gmail.com>:
>>>>
>>>>> yes it does. i only see this problem in zeppelin.
>>>>>
>>>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hs...@nflabs.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Jonathan.
>>>>>> It's not easy to build the test environments but I am working on this.
>>>>>> I have question for you.
>>>>>> Does your code working well on spark-shell in the spark.authenticate
>>>>>> mode?
>>>>>>
>>>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>
>>>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>>>>>>> quite stumped.
>>>>>>>
>>>>>>> thx.
>>>>>>>
>>>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hs...@nflabs.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thank you.
>>>>>>>> Let me try.
>>>>>>>>
>>>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>
>>>>>>>>> Hyung,
>>>>>>>>>
>>>>>>>>> Yes, here they are.
>>>>>>>>>
>>>>>>>>> zeppelin-env.sh:
>>>>>>>>>
>>>>>>>>> export ZEPPELIN_PORT=8890
>>>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>>>>>> export MASTER=yarn-client
>>>>>>>>> export SPARK_HOME=/usr/lib/spark
>>>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>>>> export
>>>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>>>>>> export
>>>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>>>>>>>
>>>>>>>>> spark-defaults.conf:
>>>>>>>>>
>>>>>>>>> spark.master yarn
>>>>>>>>> spark.driver.extraClassPath
>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>>> spark.driver.extraLibraryPath
>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>>> spark.executor.extraClassPath
>>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>>> spark.executor.extraLibraryPath
>>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>>> spark.eventLog.enabled true
>>>>>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps
>>>>>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps
>>>>>>>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
>>>>>>>>> spark.history.ui.port 18080
>>>>>>>>> spark.shuffle.service.enabled true
>>>>>>>>> spark.driver.extraJavaOptions
>>>>>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>>>>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>>>>>>>> -XX:MaxHeapFreeRatio=70
>>>>>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>>>>>>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>>> spark.dynamicAllocation.enabled true
>>>>>>>>> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails
>>>>>>>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>>>>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>>> spark.executor.memory 8640m
>>>>>>>>> spark.executor.cores 7
>>>>>>>>> spark.authenticate.enableSaslEncryption true
>>>>>>>>> spark.driver.memory 1g
>>>>>>>>> spark.network.sasl.serverAlwaysEncrypt true
>>>>>>>>> spark.driver.cores 1
>>>>>>>>> spark.ssl.protocol TLSv1.2
>>>>>>>>> spark.ssl.keyStorePassword password
>>>>>>>>> spark.yarn.maxAppAttempts 1
>>>>>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks
>>>>>>>>> spark.authenticate true
>>>>>>>>> spark.ssl.keyPassword password
>>>>>>>>> spark.ssl.enabled true
>>>>>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA
>>>>>>>>> spark.ssl.trustStore /etc/emr/security/truststore.jks
>>>>>>>>> spark.authenticate.secret secret
>>>>>>>>> spark.ssl.trustStorePassword password
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <
>>>>>>>>> hsshim@nflabs.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi.
>>>>>>>>>> Could you share your conf/zeppelin-env.sh and spark-defaults.conf
>>>>>>>>>> ?
>>>>>>>>>>
>>>>>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> I am having trouble using zeppelin in a spark cluster that has
>>>>>>>>>>> spark node authentication turned on (e.g. with spark.authenticate=true,
>>>>>>>>>>> spark.authenticate.secret=...)
>>>>>>>>>>>
>>>>>>>>>>> Notebook code that calls built-in spark functions (or other
>>>>>>>>>>> things on executor classpath) work fine, but functions defined in the
>>>>>>>>>>> notebook (anonymous or named) throw ClassNotFoundExceptions when called
>>>>>>>>>>> from an executor.
>>>>>>>>>>>
>>>>>>>>>>> For example, this code works:
>>>>>>>>>>>
>>>>>>>>>>> val rdd = sc.textFile("hdfs://my-text-file")
>>>>>>>>>>> rdd.take(1).foreach(println)
>>>>>>>>>>>
>>>>>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>>>>>>>>>>>
>>>>>>>>>>> but code like this...
>>>>>>>>>>>
>>>>>>>>>>> rdd.filter(_.contains("my data"))
>>>>>>>>>>>
>>>>>>>>>>> fails with
>>>>>>>>>>>
>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>> ...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I get the same kind of error if the filter function is defined
>>>>>>>>>>> as a named function in the notebook, or as a member of singleton object
>>>>>>>>>>> defined in the notebook.
>>>>>>>>>>>
>>>>>>>>>>> When I look at the executor's log output, I see this error:
>>>>>>>>>>>
>>>>>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to
>>>>>>>>>>> check existence of class
>>>>>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL
>>>>>>>>>>> class server at https://172.30.54.30:34980
>>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>>
>>>>>>>>>>> ...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If I disable spark authentication, everything works as expected.
>>>>>>>>>>> I am running zeppelin 0.5.6 on spark 1.6.1 with yarn.
>>>>>>>>>>>
>>>>>>>>>>> Has anyone been able to get zeppelin working with spark
>>>>>>>>>>> authentication?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: classnotfoundexception using zeppelin with spark authentication
Posted by Jonathan Esterhazy <jo...@gmail.com>.
Hyung, thx for your help. I opened these:
https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala problem)
https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking python
problem)
LMK if I can provide more info or help in some way.
On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim <hs...@nflabs.com> wrote:
> Hi Jonathan.
> Unfortunately I got same error in my test bed.
> Do you mind create an jira issue for this?
>
> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <jonathan.esterhazy@gmail.com
> >:
>
>> I added this and still get the same exception. The same property is also
>> set in spark-defaults.conf.
>>
>> After that didn't work, I also tried adding --conf
>> spark.authenticate=true and --conf
>> spark.authenticate.enableSaslEncryption=true, to match the other related
>> settings in spark-defaults.conf. Still get the same classnotfoundexception.
>>
>>
>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hs...@nflabs.com>
>> wrote:
>>
>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
>>> zeppelin and retry your code.
>>>
>>>
>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterhazy@gmail.com>:
>>>
>>>> yes it does. i only see this problem in zeppelin.
>>>>
>>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hs...@nflabs.com>
>>>> wrote:
>>>>
>>>>> Hi Jonathan.
>>>>> It's not easy to build the test environments but I am working on this.
>>>>> I have question for you.
>>>>> Does your code working well on spark-shell in the spark.authenticate
>>>>> mode?
>>>>>
>>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>>>>> jonathan.esterhazy@gmail.com>:
>>>>>
>>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>>>>>> quite stumped.
>>>>>>
>>>>>> thx.
>>>>>>
>>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hs...@nflabs.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you.
>>>>>>> Let me try.
>>>>>>>
>>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>
>>>>>>>> Hyung,
>>>>>>>>
>>>>>>>> Yes, here they are.
>>>>>>>>
>>>>>>>> zeppelin-env.sh:
>>>>>>>>
>>>>>>>> export ZEPPELIN_PORT=8890
>>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>>>>> export MASTER=yarn-client
>>>>>>>> export SPARK_HOME=/usr/lib/spark
>>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>>> export
>>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>>>>> export
>>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>>>>>>
>>>>>>>> spark-defaults.conf:
>>>>>>>>
>>>>>>>> spark.master yarn
>>>>>>>> spark.driver.extraClassPath
>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>> spark.driver.extraLibraryPath
>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>> spark.executor.extraClassPath
>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>> spark.executor.extraLibraryPath
>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>> spark.eventLog.enabled true
>>>>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps
>>>>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps
>>>>>>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
>>>>>>>> spark.history.ui.port 18080
>>>>>>>> spark.shuffle.service.enabled true
>>>>>>>> spark.driver.extraJavaOptions
>>>>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>>>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>>>>>>> -XX:MaxHeapFreeRatio=70
>>>>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>>>>>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>> spark.dynamicAllocation.enabled true
>>>>>>>> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails
>>>>>>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>>>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>> spark.executor.memory 8640m
>>>>>>>> spark.executor.cores 7
>>>>>>>> spark.authenticate.enableSaslEncryption true
>>>>>>>> spark.driver.memory 1g
>>>>>>>> spark.network.sasl.serverAlwaysEncrypt true
>>>>>>>> spark.driver.cores 1
>>>>>>>> spark.ssl.protocol TLSv1.2
>>>>>>>> spark.ssl.keyStorePassword password
>>>>>>>> spark.yarn.maxAppAttempts 1
>>>>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks
>>>>>>>> spark.authenticate true
>>>>>>>> spark.ssl.keyPassword password
>>>>>>>> spark.ssl.enabled true
>>>>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA
>>>>>>>> spark.ssl.trustStore /etc/emr/security/truststore.jks
>>>>>>>> spark.authenticate.secret secret
>>>>>>>> spark.ssl.trustStorePassword password
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <hsshim@nflabs.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi.
>>>>>>>>> Could you share your conf/zeppelin-env.sh and spark-defaults.conf ?
>>>>>>>>>
>>>>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>> jonathan.esterhazy@gmail.com>:
>>>>>>>>>
>>>>>>>>>> I am having trouble using zeppelin in a spark cluster that has
>>>>>>>>>> spark node authentication turned on (e.g. with spark.authenticate=true,
>>>>>>>>>> spark.authenticate.secret=...)
>>>>>>>>>>
>>>>>>>>>> Notebook code that calls built-in spark functions (or other
>>>>>>>>>> things on executor classpath) work fine, but functions defined in the
>>>>>>>>>> notebook (anonymous or named) throw ClassNotFoundExceptions when called
>>>>>>>>>> from an executor.
>>>>>>>>>>
>>>>>>>>>> For example, this code works:
>>>>>>>>>>
>>>>>>>>>> val rdd = sc.textFile("hdfs://my-text-file")
>>>>>>>>>> rdd.take(1).foreach(println)
>>>>>>>>>>
>>>>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>>>>>>>>>>
>>>>>>>>>> but code like this...
>>>>>>>>>>
>>>>>>>>>> rdd.filter(_.contains("my data"))
>>>>>>>>>>
>>>>>>>>>> fails with
>>>>>>>>>>
>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I get the same kind of error if the filter function is defined as
>>>>>>>>>> a named function in the notebook, or as a member of singleton object
>>>>>>>>>> defined in the notebook.
>>>>>>>>>>
>>>>>>>>>> When I look at the executor's log output, I see this error:
>>>>>>>>>>
>>>>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to check
>>>>>>>>>> existence of class
>>>>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL
>>>>>>>>>> class server at https://172.30.54.30:34980
>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If I disable spark authentication, everything works as expected.
>>>>>>>>>> I am running zeppelin 0.5.6 on spark 1.6.1 with yarn.
>>>>>>>>>>
>>>>>>>>>> Has anyone been able to get zeppelin working with spark
>>>>>>>>>> authentication?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>