You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2014/11/20 10:37:33 UTC
[jira] [Commented] (SPARK-4490) Not found RandomGenerator through
spark-shell
[ https://issues.apache.org/jira/browse/SPARK-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219176#comment-14219176 ]
Sean Owen commented on SPARK-4490:
----------------------------------
So, in general, you have to supply your own dependencies and can't rely on Spark providing them for you. If you use Breeze, and it uses Commons Math, you need to put them both on the classpath with spark-shell.
That said, breeze happens to be on the classpath because it's built into Spark. And so should Commons Math. And we have heard a number of problems related to Commons Math not being found with Breeze, which I don't quite get. We are not shading Commons Math now.
So, I think this particular JIRA is NotAProblem, but there is something weird going on with the Commons Math dep. But, properly supplying the dependency seems to work for everyone, and that's the right thing to do.
> Not found RandomGenerator through spark-shell
> ---------------------------------------------
>
> Key: SPARK-4490
> URL: https://issues.apache.org/jira/browse/SPARK-4490
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.1.0
> Environment: spark-shell
> Reporter: Kai Sasaki
>
> In spark-1.1.0, exception is threw whenever RandomGenerator of commons-math3 is used. There is some workaround about this problem.
> http://find.searchhub.org/document/6df0c89201dfe386#10bd443c4da849a3
> ```
> scala> import breeze.linalg._
> import breeze.linalg._
> scala> Matrix.rand[Double](3, 3)
> java.lang.NoClassDefFoundError: org/apache/commons/math3/random/RandomGenerator
> at breeze.linalg.MatrixConstructors$class.rand$default$3(Matrix.scala:205)
> at breeze.linalg.Matrix$.rand$default$3(Matrix.scala:139)
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:14)
> at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:19)
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:21)
> at $iwC$$iwC$$iwC.<init>(<console>:23)
> at $iwC$$iwC.<init>(<console>:25)
> at $iwC.<init>(<console>:27)
> at <init>(<console>:29)
> at .<init>(<console>:33)
> at .<clinit>(<console>)
> at .<init>(<console>:7)
> at .<clinit>(<console>)
> at $print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
> at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:814)
> at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:859)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:771)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:616)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:624)
> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:629)
> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:954)
> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:902)
> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:902)
> at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:902)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:997)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> aused by: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomGenerator
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 44 more
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org