Bug: Connecting to Spark 1.6 from Jupyter

[Abhishek M <Ab...@swissre.com>]

Hi,
I am trying to connect to a Cloudera spark cluster 1.6 from Jupyter on Edge node. I have installed the Apache Toree 0.1.0 and 0.2.0 for this and I keep getting NoSuchMethodError. What could be the solution for this.

WARNING:root:kernel 1e6b66dd-a4f5-468e-aac3-4508c8d5f4f7 restarted
Starting Spark Kernel with SPARK_HOME=/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/spark
Exception in thread "main" java.lang.NoSuchMethodError: joptsimple/OptionParser.acceptsAll(Ljava/util/Collection;Ljava/lang/String;)Ljoptsimple/OptionSpecBuilder; (loaded from file:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/jars/jopt-simple-4.9.jar by sun.misc.Launcher$AppClassLoader@8d1ebeba) called from class org.apache.toree.boot.CommandLineOptions (loaded from file:/usr/local/share/jupyter/kernels/apache_toree_scala/lib/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar by org.apache.spark.util.MutableURLClassLoader@863c8f1f).
        at org.apache.toree.boot.CommandLineOptions.<init>(CommandLineOptions.scala:37)
        at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:25)
        at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24)
        at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
        at scala.App$$anonfun$main$1.apply(App.scala:71)
        at scala.App$$anonfun$main$1.apply(App.scala:71)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
        at scala.App$class.main(App.scala:71)
        at org.apache.toree.Main$.main(Main.scala:24)
        at org.apache.toree.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:508)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:730)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Thank you
Regards
Abhishek M



This e-mail, including attachments, is intended for the person(s) or company named and may contain confidential and/or legally privileged information.
Unauthorized disclosure, copying or use of this information may be unlawful and is prohibited. If you are not the intended recipient, please delete this message and notify the sender.
All incoming and outgoing e-mail messages are stored in the Swiss Re Electronic Message Repository.
If you do not wish the retention of potentially private e-mails by Swiss Re, we strongly advise you not to use the Swiss Re e-mail account for any private, non-business related communications.

Re: Bug: Connecting to Spark 1.6 from Jupyter

[Luciano Resende <lu...@gmail.com>]

For Spark 1.6, toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar will not
work. You will need to use the 0.1 release or build the 0.1 branch from
Toree.
​
Also, how are you submitting toree to Spark?  Yarn client or Yarn Cluster ?

If moving to Toree 0.1 and you still have these dependencies issues, you
might take a look at the following commit:
https://github.com/apache/incubator-toree/commit/14c80857a5f2a816cf0405624a32210018e85d53
but I usually only had this issue in Yarn cluster mode, although some other
folks have actually complained about it in CDH before.