You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@toree.apache.org by "Ming Yu (JIRA)" <ji...@apache.org> on 2016/11/20 07:41:58 UTC
[jira] [Created] (TOREE-354) Scala Error with Apache Spark when run
in Jupyter
Ming Yu created TOREE-354:
-----------------------------
Summary: Scala Error with Apache Spark when run in Jupyter
Key: TOREE-354
URL: https://issues.apache.org/jira/browse/TOREE-354
Project: TOREE
Issue Type: Bug
Affects Versions: 0.1.0
Environment: Apache Spark 2.0.2
Scala 2.11(Built with Apache Spark by default)
Reporter: Ming Yu
Fix For: 0.1.0
I'm having problems running Scala Spark on Jupyter. Below is my error message when I load Apache Toree - Scala notebook in jupyter.
{noformat}
root@ubuntu-2gb-sgp1-01:~# jupyter notebook --ip 0.0.0.0 --port 8888
[I 03:14:54.281 NotebookApp] Serving notebooks from local directory: /root
[I 03:14:54.281 NotebookApp] 0 active kernels
[I 03:14:54.281 NotebookApp] The Jupyter Notebook is running at: http://0.0.0.0:8888/
[I 03:14:54.281 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 03:14:54.282 NotebookApp] No web browser found: could not locate runnable browser.
[I 03:15:09.976 NotebookApp] 302 GET / (61.6.68.44) 1.21ms
[I 03:15:15.924 NotebookApp] Creating new notebook in
[W 03:15:16.592 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20161120031454 (61.6.68.44) 15.49ms referer=http://188.166.235.21:8888/notebooks/Untitled2.ipynb?kernel_name=apache_toree_scala
[I 03:15:16.677 NotebookApp] Kernel started: 94a63354-d294-4de7-a12c-2e05905e0c45
Starting Spark Kernel with SPARK_HOME=/usr/local/spark
16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - Kernel version: 0.1.0.dev8-incubating-SNAPSHOT
16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - Scala version: Some(2.10.4)
16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - ZeroMQ (JeroMQ) version: 3.2.2
16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - Initializing internal actor system
Exception in thread "main" java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;
at akka.actor.ActorCell$.<init>(ActorCell.scala:336)
at akka.actor.ActorCell$.<clinit>(ActorCell.scala)
at akka.actor.RootActorPath.$div(ActorPath.scala:185)
at akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:465)
at akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:453)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78)
at scala.util.Try$.apply(Try.scala:192)
at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73)
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
at scala.util.Success.flatMap(Try.scala:231)
at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)
at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:585)
at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:578)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
at akka.actor.ActorSystem$.apply(ActorSystem.scala:109)
at org.apache.toree.boot.layer.StandardBareInitialization$class.createActorSystem(BareInitialization.scala:71)
at org.apache.toree.Main$$anon$1.createActorSystem(Main.scala:35)
at org.apache.toree.boot.layer.StandardBareInitialization$class.initializeBare(BareInitialization.scala:60)
at org.apache.toree.Main$$anon$1.initializeBare(Main.scala:35)
at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:72)
at org.apache.toree.Main$delayedInit$body.apply(Main.scala:40)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at org.apache.toree.Main$.main(Main.scala:24)
at org.apache.toree.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[W 03:15:26.738 NotebookApp] Timeout waiting for kernel_info reply from 94a63354-d294-4de7-a12c-2e05905e0c45
{noformat}
When running Scala shell, this is my output logs
{noformat}
root@ubuntu-2gb-sgp1-01:~# spark-shell
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
16/11/20 03:17:11 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/11/20 03:17:12 WARN Utils: Your hostname, ubuntu-2gb-sgp1-01 resolves to a loopback address: 127.0.1.1; using 10.15.0.5 instead (on interface eth0)
16/11/20 03:17:12 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
16/11/20 03:17:13 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect.
Spark context Web UI available at http://10.15.0.5:4040
Spark context available as 'sc' (master = local[*], app id = local-1479611833426).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.0.2
/_/
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_111)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
{noformat}
This problem was highlighted before in jira https://issues.apache.org/jira/browse/TOREE-336 . However, I'm still unable to get it working for some reason.
I followed the instructions listed on their official site. https://toree.apache.org/documentation/user/quick-start
This is my path variables
{noformat}
scala> root@ubuntu-2gb-sgp1-01:~# echo $PATH
/root/bin:/root/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/spark:/usr/local/spark/bin
{noformat}
Thanks and much appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [jira] [Created] (TOREE-354) Scala Error with Apache Spark when
run in Jupyter
Posted by Marius van Niekerk <ma...@gmail.com>.
So for spark 2+ you currently need to install from master.
On Sun, Nov 20, 2016, 02:42 Ming Yu (JIRA) <ji...@apache.org> wrote:
> Ming Yu created TOREE-354:
> -----------------------------
>
> Summary: Scala Error with Apache Spark when run in Jupyter
> Key: TOREE-354
> URL: https://issues.apache.org/jira/browse/TOREE-354
> Project: TOREE
> Issue Type: Bug
> Affects Versions: 0.1.0
> Environment: Apache Spark 2.0.2
> Scala 2.11(Built with Apache Spark by default)
> Reporter: Ming Yu
> Fix For: 0.1.0
>
>
> I'm having problems running Scala Spark on Jupyter. Below is my error
> message when I load Apache Toree - Scala notebook in jupyter.
>
> {noformat}
> root@ubuntu-2gb-sgp1-01:~# jupyter notebook --ip 0.0.0.0 --port 8888
> [I 03:14:54.281 NotebookApp] Serving notebooks from local directory: /root
> [I 03:14:54.281 NotebookApp] 0 active kernels
> [I 03:14:54.281 NotebookApp] The Jupyter Notebook is running at:
> http://0.0.0.0:8888/
> [I 03:14:54.281 NotebookApp] Use Control-C to stop this server and shut
> down all kernels (twice to skip confirmation).
> [W 03:14:54.282 NotebookApp] No web browser found: could not locate
> runnable browser.
> [I 03:15:09.976 NotebookApp] 302 GET / (61.6.68.44) 1.21ms
> [I 03:15:15.924 NotebookApp] Creating new notebook in
> [W 03:15:16.592 NotebookApp] 404 GET
> /nbextensions/widgets/notebook/js/extension.js?v=20161120031454
> (61.6.68.44) 15.49ms referer=
> http://188.166.235.21:8888/notebooks/Untitled2.ipynb?kernel_name=apache_toree_scala
> [I 03:15:16.677 NotebookApp] Kernel started:
> 94a63354-d294-4de7-a12c-2e05905e0c45
> Starting Spark Kernel with SPARK_HOME=/usr/local/spark
> 16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - Kernel version:
> 0.1.0.dev8-incubating-SNAPSHOT
> 16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - Scala version: Some(2.10.4)
> 16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - ZeroMQ (JeroMQ) version:
> 3.2.2
> 16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - Initializing internal actor
> system
> Exception in thread "main" java.lang.NoSuchMethodError:
> scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;
> at akka.actor.ActorCell$.<init>(ActorCell.scala:336)
> at akka.actor.ActorCell$.<clinit>(ActorCell.scala)
> at akka.actor.RootActorPath.$div(ActorPath.scala:185)
> at
> akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:465)
> at
> akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:453)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78)
> at scala.util.Try$.apply(Try.scala:192)
> at
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73)
> at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
> at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
> at scala.util.Success.flatMap(Try.scala:231)
> at
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)
> at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:585)
> at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:578)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:109)
> at
> org.apache.toree.boot.layer.StandardBareInitialization$class.createActorSystem(BareInitialization.scala:71)
> at org.apache.toree.Main$$anon$1.createActorSystem(Main.scala:35)
> at
> org.apache.toree.boot.layer.StandardBareInitialization$class.initializeBare(BareInitialization.scala:60)
> at org.apache.toree.Main$$anon$1.initializeBare(Main.scala:35)
> at
> org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:72)
> at org.apache.toree.Main$delayedInit$body.apply(Main.scala:40)
> at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
> at
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
> at scala.App$$anonfun$main$1.apply(App.scala:76)
> at scala.App$$anonfun$main$1.apply(App.scala:76)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
> at scala.App$class.main(App.scala:76)
> at org.apache.toree.Main$.main(Main.scala:24)
> at org.apache.toree.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
> at
> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> [W 03:15:26.738 NotebookApp] Timeout waiting for kernel_info reply from
> 94a63354-d294-4de7-a12c-2e05905e0c45
> {noformat}
>
> When running Scala shell, this is my output logs
>
> {noformat}
> root@ubuntu-2gb-sgp1-01:~# spark-shell
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel).
> 16/11/20 03:17:11 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 16/11/20 03:17:12 WARN Utils: Your hostname, ubuntu-2gb-sgp1-01 resolves
> to a loopback address: 127.0.1.1; using 10.15.0.5 instead (on interface
> eth0)
> 16/11/20 03:17:12 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
> another address
> 16/11/20 03:17:13 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> Spark context Web UI available at http://10.15.0.5:4040
> Spark context available as 'sc' (master = local[*], app id =
> local-1479611833426).
> Spark session available as 'spark'.
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 2.0.2
> /_/
>
> Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_111)
> Type in expressions to have them evaluated.
> Type :help for more information.
>
> scala>
> {noformat}
>
> This problem was highlighted before in jira
> https://issues.apache.org/jira/browse/TOREE-336 . However, I'm still
> unable to get it working for some reason.
>
> I followed the instructions listed on their official site.
> https://toree.apache.org/documentation/user/quick-start
>
> This is my path variables
> {noformat}
> scala> root@ubuntu-2gb-sgp1-01:~# echo $PATH
>
> /root/bin:/root/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/spark:/usr/local/spark/bin
> {noformat}
>
> Thanks and much appreciated.
>
>
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>
--
regards
Marius van Niekerk