You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ritesh Kumar Singh <ri...@gmail.com> on 2014/11/17 09:24:24 UTC
RandomGenerator class not found exception
My sbt file for the project includes this:
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.1.0",
"org.apache.spark" %% "spark-mllib" % "1.1.0",
"org.apache.commons" % "commons-math3" % "3.3"
)
=============================================================
Still I am getting this error:
>java.lang.NoClassDefFoundError:
org/apache/commons/math3/random/RandomGenerator
=============================================================
The jar at location: ~/.m2/repository/org/apache/commons/commons-math3/3.3
contains the random generator class:
$ jar tvf commons-math3-3.3.jar | grep RandomGenerator
>org/apache/commons/math3/random/RandomGenerator.class
>org/apache/commons/math3/random/UniformRandomGenerator.class
>org/apache/commons/math3/random/SynchronizedRandomGenerator.class
>org/apache/commons/math3/random/AbstractRandomGenerator.class
>org/apache/commons/math3/random/RandomGeneratorFactory$1.class
>org/apache/commons/math3/random/RandomGeneratorFactory.class
>org/apache/commons/math3/random/StableRandomGenerator.class
>org/apache/commons/math3/random/NormalizedRandomGenerator.class
>org/apache/commons/math3/random/JDKRandomGenerator.class
>org/apache/commons/math3/random/GaussianRandomGenerator.class
Please help
Re: RandomGenerator class not found exception
Posted by Chitturi Padma <le...@gmail.com>.
As you are using sbt ..u need not put in ~/.m2/repositories for maven.
Include the jar explicitly using the option
--driver-class-path while submitting the jar to spark cluster
On Mon, Nov 17, 2014 at 7:41 PM, Ritesh Kumar Singh [via Apache Spark User
List] <ml...@n3.nabble.com> wrote:
> It's still not working. Keep getting the same error.
>
> I even deleted the commons-math3/* folder containing the jar. And then
> under the directory "org/apache/commons/" made a folder called 'math3'
> and put the commons-math3-3.3.jar in it.
> Still its not working.
>
> I also tried sc.addJar("/path/to/jar") within spark-shell and in my
> project sourcefile
> It still didn't import the jar at both locations.
> More
>
> Any fixes? Please help
>
> On Mon, Nov 17, 2014 at 2:14 PM, Chitturi Padma <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=19073&i=0>> wrote:
>
>> Include the commons-math3/3.3 in class path while submitting jar to spark
>> cluster. Like..
>> spark-submit --driver-class-path maths3.3jar --class MainClass --master
>> spark cluster url appjar
>>
>> On Mon, Nov 17, 2014 at 1:55 PM, Ritesh Kumar Singh [via Apache Spark
>> User List] <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=19057&i=0>> wrote:
>>
>>> My sbt file for the project includes this:
>>>
>>> libraryDependencies ++= Seq(
>>> "org.apache.spark" %% "spark-core" % "1.1.0",
>>> "org.apache.spark" %% "spark-mllib" % "1.1.0",
>>> "org.apache.commons" % "commons-math3" % "3.3"
>>> )
>>> =============================================================
>>>
>>> Still I am getting this error:
>>>
>>> >java.lang.NoClassDefFoundError:
>>> org/apache/commons/math3/random/RandomGenerator
>>>
>>> =============================================================
>>>
>>> The jar at location:
>>> ~/.m2/repository/org/apache/commons/commons-math3/3.3 contains the random
>>> generator class:
>>>
>>> $ jar tvf commons-math3-3.3.jar | grep RandomGenerator
>>> >org/apache/commons/math3/random/RandomGenerator.class
>>> >org/apache/commons/math3/random/UniformRandomGenerator.class
>>> >org/apache/commons/math3/random/SynchronizedRandomGenerator.class
>>> >org/apache/commons/math3/random/AbstractRandomGenerator.class
>>> >org/apache/commons/math3/random/RandomGeneratorFactory$1.class
>>> >org/apache/commons/math3/random/RandomGeneratorFactory.class
>>> >org/apache/commons/math3/random/StableRandomGenerator.class
>>> >org/apache/commons/math3/random/NormalizedRandomGenerator.class
>>> >org/apache/commons/math3/random/JDKRandomGenerator.class
>>> >org/apache/commons/math3/random/GaussianRandomGenerator.class
>>>
>>>
>>> Please help
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the
>>> discussion below:
>>>
>>> http://apache-spark-user-list.1001560.n3.nabble.com/RandomGenerator-class-not-found-exception-tp19055.html
>>> To start a new topic under Apache Spark User List, email [hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=19057&i=1>
>>> To unsubscribe from Apache Spark User List, click here.
>>> NAML
>>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>> ------------------------------
>> View this message in context: Re: RandomGenerator class not found
>> exception
>> <http://apache-spark-user-list.1001560.n3.nabble.com/RandomGenerator-class-not-found-exception-tp19055p19057.html>
>> Sent from the Apache Spark User List mailing list archive
>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/RandomGenerator-class-not-found-exception-tp19055p19073.html
> To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1h76@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=bGVhcm5pbmdzLmNoaXR0dXJpQGdtYWlsLmNvbXwxfC03NzExMjUwMg==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RandomGenerator-class-not-found-exception-tp19055p19086.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: RandomGenerator class not found exception
Posted by Ritesh Kumar Singh <ri...@gmail.com>.
It's still not working. Keep getting the same error.
I even deleted the commons-math3/* folder containing the jar. And then
under the directory "org/apache/commons/" made a folder called 'math3' and
put the commons-math3-3.3.jar in it.
Still its not working.
I also tried sc.addJar("/path/to/jar") within spark-shell and in my project
sourcefile
It still didn't import the jar at both locations.
More
Any fixes? Please help
On Mon, Nov 17, 2014 at 2:14 PM, Chitturi Padma <
learnings.chitturi@gmail.com> wrote:
> Include the commons-math3/3.3 in class path while submitting jar to spark
> cluster. Like..
> spark-submit --driver-class-path maths3.3jar --class MainClass --master
> spark cluster url appjar
>
> On Mon, Nov 17, 2014 at 1:55 PM, Ritesh Kumar Singh [via Apache Spark User
> List] <[hidden email] <http://user/SendEmail.jtp?type=node&node=19057&i=0>
> > wrote:
>
>> My sbt file for the project includes this:
>>
>> libraryDependencies ++= Seq(
>> "org.apache.spark" %% "spark-core" % "1.1.0",
>> "org.apache.spark" %% "spark-mllib" % "1.1.0",
>> "org.apache.commons" % "commons-math3" % "3.3"
>> )
>> =============================================================
>>
>> Still I am getting this error:
>>
>> >java.lang.NoClassDefFoundError:
>> org/apache/commons/math3/random/RandomGenerator
>>
>> =============================================================
>>
>> The jar at location:
>> ~/.m2/repository/org/apache/commons/commons-math3/3.3 contains the random
>> generator class:
>>
>> $ jar tvf commons-math3-3.3.jar | grep RandomGenerator
>> >org/apache/commons/math3/random/RandomGenerator.class
>> >org/apache/commons/math3/random/UniformRandomGenerator.class
>> >org/apache/commons/math3/random/SynchronizedRandomGenerator.class
>> >org/apache/commons/math3/random/AbstractRandomGenerator.class
>> >org/apache/commons/math3/random/RandomGeneratorFactory$1.class
>> >org/apache/commons/math3/random/RandomGeneratorFactory.class
>> >org/apache/commons/math3/random/StableRandomGenerator.class
>> >org/apache/commons/math3/random/NormalizedRandomGenerator.class
>> >org/apache/commons/math3/random/JDKRandomGenerator.class
>> >org/apache/commons/math3/random/GaussianRandomGenerator.class
>>
>>
>> Please help
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/RandomGenerator-class-not-found-exception-tp19055.html
>> To start a new topic under Apache Spark User List, email [hidden email]
>> <http://user/SendEmail.jtp?type=node&node=19057&i=1>
>> To unsubscribe from Apache Spark User List, click here.
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: RandomGenerator class not found
> exception
> <http://apache-spark-user-list.1001560.n3.nabble.com/RandomGenerator-class-not-found-exception-tp19055p19057.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>
Re: RandomGenerator class not found exception
Posted by Chitturi Padma <le...@gmail.com>.
Include the commons-math3/3.3 in class path while submitting jar to spark
cluster. Like..
spark-submit --driver-class-path maths3.3jar --class MainClass --master
spark cluster url appjar
On Mon, Nov 17, 2014 at 1:55 PM, Ritesh Kumar Singh [via Apache Spark User
List] <ml...@n3.nabble.com> wrote:
> My sbt file for the project includes this:
>
> libraryDependencies ++= Seq(
> "org.apache.spark" %% "spark-core" % "1.1.0",
> "org.apache.spark" %% "spark-mllib" % "1.1.0",
> "org.apache.commons" % "commons-math3" % "3.3"
> )
> =============================================================
>
> Still I am getting this error:
>
> >java.lang.NoClassDefFoundError:
> org/apache/commons/math3/random/RandomGenerator
>
> =============================================================
>
> The jar at location: ~/.m2/repository/org/apache/commons/commons-math3/3.3
> contains the random generator class:
>
> $ jar tvf commons-math3-3.3.jar | grep RandomGenerator
> >org/apache/commons/math3/random/RandomGenerator.class
> >org/apache/commons/math3/random/UniformRandomGenerator.class
> >org/apache/commons/math3/random/SynchronizedRandomGenerator.class
> >org/apache/commons/math3/random/AbstractRandomGenerator.class
> >org/apache/commons/math3/random/RandomGeneratorFactory$1.class
> >org/apache/commons/math3/random/RandomGeneratorFactory.class
> >org/apache/commons/math3/random/StableRandomGenerator.class
> >org/apache/commons/math3/random/NormalizedRandomGenerator.class
> >org/apache/commons/math3/random/JDKRandomGenerator.class
> >org/apache/commons/math3/random/GaussianRandomGenerator.class
>
>
> Please help
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/RandomGenerator-class-not-found-exception-tp19055.html
> To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1h76@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=bGVhcm5pbmdzLmNoaXR0dXJpQGdtYWlsLmNvbXwxfC03NzExMjUwMg==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RandomGenerator-class-not-found-exception-tp19055p19057.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: RandomGenerator class not found exception
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Add this jar
<http://mvnrepository.com/artifact/org.apache.commons/commons-math3/3.3>
while creating the sparkContext.
sc.addJar("/path/to/commons-math3-3.3.jar")
And make sure it is shipped and available in the environment tab (4040)
Thanks
Best Regards
On Mon, Nov 17, 2014 at 1:54 PM, Ritesh Kumar Singh <
riteshoneinamillion@gmail.com> wrote:
> My sbt file for the project includes this:
>
> libraryDependencies ++= Seq(
> "org.apache.spark" %% "spark-core" % "1.1.0",
> "org.apache.spark" %% "spark-mllib" % "1.1.0",
> "org.apache.commons" % "commons-math3" % "3.3"
> )
> =============================================================
>
> Still I am getting this error:
>
> >java.lang.NoClassDefFoundError:
> org/apache/commons/math3/random/RandomGenerator
>
> =============================================================
>
> The jar at location: ~/.m2/repository/org/apache/commons/commons-math3/3.3
> contains the random generator class:
>
> $ jar tvf commons-math3-3.3.jar | grep RandomGenerator
> >org/apache/commons/math3/random/RandomGenerator.class
> >org/apache/commons/math3/random/UniformRandomGenerator.class
> >org/apache/commons/math3/random/SynchronizedRandomGenerator.class
> >org/apache/commons/math3/random/AbstractRandomGenerator.class
> >org/apache/commons/math3/random/RandomGeneratorFactory$1.class
> >org/apache/commons/math3/random/RandomGeneratorFactory.class
> >org/apache/commons/math3/random/StableRandomGenerator.class
> >org/apache/commons/math3/random/NormalizedRandomGenerator.class
> >org/apache/commons/math3/random/JDKRandomGenerator.class
> >org/apache/commons/math3/random/GaussianRandomGenerator.class
>
>
> Please help
>