You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by wxhsdp <wx...@gmail.com> on 2014/05/04 13:07:37 UTC
NoSuchMethodError: breeze.linalg.DenseMatrix
Hi,
i'am trying to use breeze linalg library for matrix operation in my spark
code. i already add dependency
on breeze in my build.sbt, and package my code sucessfully.
when i run on local mode, sbt "run local...", everything is ok
but when turn to standalone mode, sbt "run spark://127.0.0.1:7077...",
error occurs
14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to
java.lang.NoSuchMethodError
java.lang.NoSuchMethodError:
breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
in my opinion, everything needed is packaged to the jar file, isn't it?
and does anyone used breeze before? is it good for matrix operation?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: NoSuchMethodError: breeze.linalg.DenseMatrix
Posted by Yadid Ayzenberg <ya...@media.mit.edu>.
An additional option 4) Use SparkContext.addJar() and have the
application ship your jar to all the nodes.
Yadid
On 5/4/14, 4:07 PM, DB Tsai wrote:
> If you add the breeze dependency in your build.sbt project, it will
> not be available to all the workers.
>
> There are couple options, 1) use sbt assembly to package breeze into
> your application jar. 2) manually copy breeze jar into all the nodes,
> and have them in the classpath. 3) spark 1.0 has breeze jar in the
> spark flat assembly jar, so you don't need to add breeze dependency
> yourself.
>
>
> Sincerely,
>
> DB Tsai
> -------------------------------------------------------
> My Blog: https://www.dbtsai.com
> LinkedIn: https://www.linkedin.com/in/dbtsai
>
>
> On Sun, May 4, 2014 at 4:07 AM, wxhsdp <wxhsdp@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi,
> i'am trying to use breeze linalg library for matrix operation in
> my spark
> code. i already add dependency
> on breeze in my build.sbt, and package my code sucessfully.
>
> when i run on local mode, sbt "run local...", everything is ok
>
> but when turn to standalone mode, sbt "run
> spark://127.0.0.1:7077...",
> error occurs
>
> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to
> java.lang.NoSuchMethodError
> java.lang.NoSuchMethodError:
> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
>
> in my opinion, everything needed is packaged to the jar file,
> isn't it?
> and does anyone used breeze before? is it good for matrix operation?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html
> Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
>
>
Re: NoSuchMethodError: breeze.linalg.DenseMatrix
Posted by wxhsdp <wx...@gmail.com>.
finally i fixed it. previous failure is caused by lack of some jars.
i pasted the classpath in local mode to workers by using "show
compile:dependencyClasspath"
and it works!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5732.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: NoSuchMethodError: breeze.linalg.DenseMatrix
Posted by Xiangrui Meng <me...@gmail.com>.
It doesn't work if you put the netlib-native jar inside an assembly
jar. Try to mark it "provided" in the dependencies, and use --jars to
include them with spark-submit. -Xiangrui
On Wed, May 14, 2014 at 6:12 PM, wxhsdp <wx...@gmail.com> wrote:
> Hi, DB
>
> i tried including breeze library by using spark 1.0, it works. but how can
> i call
> the native library in standalone cluster mode.
>
> in local mode
> 1. i include "org.scalanlp" % "breeze-natives_2.10" % "0.7" dependency in
> sbt build file
> 2. i install openblas
> it works
>
> in standalone mode
> 1. i include "org.scalanlp" % "breeze-natives_2.10" % "0.7" dependency in
> sbt build file
> 2. install openblas in workers
> 3. add sepatate jars using sc.addJar(). jars: breeze-natives_2.10-0.7.jar,
> netlib-native_ref-linux-
> x86_64-1.1-natives.jar,
> netlib-native_system-linux-x86_64-1.1-natives.jar
> 4. i also include classpath of the above jars
> but does not work:(
>
>
>
> DB Tsai-2 wrote
>> Hi Wxhsdp,
>>
>> I also have some difficulties witth "sc.addJar()". Since we include the
>> breeze library by using Spark 1.0, we don't have the problem you ran into.
>> However, when we add external jars via sc.addJar(), I found that the
>> executors actually fetch the jars but the classloader still doesn't honor
>> it. I'm trying to figure out the problem now.
>>
>>
>> Sincerely,
>>
>> DB Tsai
>> -------------------------------------------------------
>> My Blog: https://www.dbtsai.com
>> LinkedIn: https://www.linkedin.com/in/dbtsai
>>
>>
>> On Wed, May 14, 2014 at 5:46 AM, wxhsdp <
>
>> wxhsdp@
>
>> > wrote:
>>
>>> Hi, DB
>>> i've add breeze jars to workers using sc.addJar()
>>> breeze jars include :
>>> breeze-natives_2.10-0.7.jar
>>> breeze-macros_2.10-0.3.jar
>>> breeze-macros_2.10-0.3.1.jar
>>> breeze_2.10-0.8-SNAPSHOT.jar
>>> breeze_2.10-0.7.jar
>>>
>>> almost all the jars about breeze i can find, but still
>>> NoSuchMethodError:
>>> breeze.linalg.DenseMatrix
>>>
>>> from the executor stderr, you can see the executor successsully fetches
>>> these jars, what's wrong
>>> about my method? thank you!
>>>
>>> 14/05/14 20:36:02 INFO Executor: Fetching
>>> http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar with
>>> timestamp
>>> 1400070957376
>>> 14/05/14 20:36:02 INFO Utils: Fetching
>>> http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar to
>>> /tmp/fetchFileTemp7468892065227766972.tmp
>>> 14/05/14 20:36:02 INFO Executor: Adding
>>>
>>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-natives_2.10-0.7.jar
>>> to class loader
>>> 14/05/14 20:36:02 INFO Executor: Fetching
>>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar with timestamp
>>> 1400070957441
>>> 14/05/14 20:36:02 INFO Utils: Fetching
>>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar to
>>> /tmp/fetchFileTemp2324565598765584917.tmp
>>> 14/05/14 20:36:02 INFO Executor: Adding
>>>
>>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.jar
>>> to class loader
>>> 14/05/14 20:36:02 INFO Executor: Fetching
>>> http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar with
>>> timestamp
>>> 1400070957358
>>> 14/05/14 20:36:02 INFO Utils: Fetching
>>> http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar to
>>> /tmp/fetchFileTemp8730123100104850193.tmp
>>> 14/05/14 20:36:02 INFO Executor: Adding
>>>
>>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.8-SNAPSHOT.jar
>>> to class loader
>>> 14/05/14 20:36:02 INFO Executor: Fetching
>>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar with
>>> timestamp
>>> 1400070957414
>>> 14/05/14 20:36:02 INFO Utils: Fetching
>>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar to
>>> /tmp/fetchFileTemp3473404556989515218.tmp
>>> 14/05/14 20:36:02 INFO Executor: Adding
>>>
>>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.1.jar
>>> to class loader
>>> 14/05/14 20:36:02 INFO Executor: Fetching
>>> http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar with timestamp
>>> 1400070956753
>>> 14/05/14 20:36:02 INFO Utils: Fetching
>>> http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar to
>>> /tmp/fetchFileTemp1289055585501269156.tmp
>>> 14/05/14 20:36:02 INFO Executor: Adding
>>>
>>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./build-project_2.10-1.0.jar
>>> to class loader
>>> 14/05/14 20:36:02 INFO Executor: Fetching
>>> http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar with timestamp
>>> 1400070957228
>>> 14/05/14 20:36:02 INFO Utils: Fetching
>>> http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar to
>>> /tmp/fetchFileTemp1287317286108432726.tmp
>>> 14/05/14 20:36:02 INFO Executor: Adding
>>>
>>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.7.jar
>>> to class loader
>>>
>>>
>>> DB Tsai-2 wrote
>>> > Since the breeze jar is brought into spark by mllib package, you may
>>> want
>>> > to add mllib as your dependency in spark 1.0. For bring it from your
>>> > application yourself, you can either use sbt assembly in ur build
>>> project
>>> > to generate a flat myApp-assembly.jar which contains breeze jar, or use
>>> > spark add jar api like Yadid said.
>>> >
>>> >
>>> > Sincerely,
>>> >
>>> > DB Tsai
>>> > -------------------------------------------------------
>>> > My Blog: https://www.dbtsai.com
>>> > LinkedIn: https://www.linkedin.com/in/dbtsai
>>> >
>>> >
>>> > On Sun, May 4, 2014 at 10:24 PM, wxhsdp <
>>>
>>> > wxhsdp@
>>>
>>> > > wrote:
>>> >
>>> >> Hi, DB, i think it's something related to "sbt publishLocal"
>>> >>
>>> >> if i remove the breeze dependency in my sbt file, breeze can not be
>>> found
>>> >>
>>> >> [error] /home/wxhsdp/spark/example/test/src/main/scala/test.scala:5:
>>> not
>>> >> found: object breeze
>>> >> [error] import breeze.linalg._
>>> >> [error] ^
>>> >>
>>> >> here's my sbt file:
>>> >>
>>> >> name := "Build Project"
>>> >>
>>> >> version := "1.0"
>>> >>
>>> >> scalaVersion := "2.10.4"
>>> >>
>>> >> libraryDependencies += "org.apache.spark" %% "spark-core" %
>>> >> "1.0.0-SNAPSHOT"
>>> >>
>>> >> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
>>> >>
>>> >> i run "sbt publishLocal" on the Spark tree.
>>> >>
>>> >> but if i manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in
>>> >> /lib
>>> >> directory, sbt package is
>>> >> ok, i can run my app in workers without addJar
>>> >>
>>> >> what's the difference between add dependency in sbt after "sbt
>>> >> publishLocal"
>>> >> and manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib
>>> >> directory?
>>> >>
>>> >> why can i run my app in worker without addJar this time?
>>> >>
>>> >>
>>> >> DB Tsai-2 wrote
>>> >> > If you add the breeze dependency in your build.sbt project, it will
>>> not
>>> >> be
>>> >> > available to all the workers.
>>> >> >
>>> >> > There are couple options, 1) use sbt assembly to package breeze into
>>> >> your
>>> >> > application jar. 2) manually copy breeze jar into all the nodes, and
>>> >> have
>>> >> > them in the classpath. 3) spark 1.0 has breeze jar in the spark flat
>>> >> > assembly jar, so you don't need to add breeze dependency yourself.
>>> >> >
>>> >> >
>>> >> > Sincerely,
>>> >> >
>>> >> > DB Tsai
>>> >> > -------------------------------------------------------
>>> >> > My Blog: https://www.dbtsai.com
>>> >> > LinkedIn: https://www.linkedin.com/in/dbtsai
>>> >> >
>>> >> >
>>> >> > On Sun, May 4, 2014 at 4:07 AM, wxhsdp <
>>> >>
>>> >> > wxhsdp@
>>> >>
>>> >> > > wrote:
>>> >> >
>>> >> >> Hi,
>>> >> >> i'am trying to use breeze linalg library for matrix operation in
>>> my
>>> >> >> spark
>>> >> >> code. i already add dependency
>>> >> >> on breeze in my build.sbt, and package my code sucessfully.
>>> >> >>
>>> >> >> when i run on local mode, sbt "run local...", everything is ok
>>> >> >>
>>> >> >> but when turn to standalone mode, sbt "run spark://127.0.0.1:7077
>>> >> ...",
>>> >> >> error occurs
>>> >> >>
>>> >> >> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to
>>> >> >> java.lang.NoSuchMethodError
>>> >> >> java.lang.NoSuchMethodError:
>>> >> >>
>>> >> >>
>>> >>
>>> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
>>> >> >>
>>> >> >> in my opinion, everything needed is packaged to the jar file,
>>> isn't
>>> >> it?
>>> >> >> and does anyone used breeze before? is it good for matrix
>>> operation?
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> View this message in context:
>>> >> >>
>>> >>
>>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html
>>> >> >> Sent from the Apache Spark User List mailing list archive at
>>> >> Nabble.com.
>>> >> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> View this message in context:
>>> >>
>>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5355.html
>>> >> Sent from the Apache Spark User List mailing list archive at
>>> Nabble.com.
>>> >>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5704.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5730.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: NoSuchMethodError: breeze.linalg.DenseMatrix
Posted by wxhsdp <wx...@gmail.com>.
Hi, DB
i tried including breeze library by using spark 1.0, it works. but how can
i call
the native library in standalone cluster mode.
in local mode
1. i include "org.scalanlp" % "breeze-natives_2.10" % "0.7" dependency in
sbt build file
2. i install openblas
it works
in standalone mode
1. i include "org.scalanlp" % "breeze-natives_2.10" % "0.7" dependency in
sbt build file
2. install openblas in workers
3. add sepatate jars using sc.addJar(). jars: breeze-natives_2.10-0.7.jar,
netlib-native_ref-linux-
x86_64-1.1-natives.jar,
netlib-native_system-linux-x86_64-1.1-natives.jar
4. i also include classpath of the above jars
but does not work:(
DB Tsai-2 wrote
> Hi Wxhsdp,
>
> I also have some difficulties witth "sc.addJar()". Since we include the
> breeze library by using Spark 1.0, we don't have the problem you ran into.
> However, when we add external jars via sc.addJar(), I found that the
> executors actually fetch the jars but the classloader still doesn't honor
> it. I'm trying to figure out the problem now.
>
>
> Sincerely,
>
> DB Tsai
> -------------------------------------------------------
> My Blog: https://www.dbtsai.com
> LinkedIn: https://www.linkedin.com/in/dbtsai
>
>
> On Wed, May 14, 2014 at 5:46 AM, wxhsdp <
> wxhsdp@
> > wrote:
>
>> Hi, DB
>> i've add breeze jars to workers using sc.addJar()
>> breeze jars include :
>> breeze-natives_2.10-0.7.jar
>> breeze-macros_2.10-0.3.jar
>> breeze-macros_2.10-0.3.1.jar
>> breeze_2.10-0.8-SNAPSHOT.jar
>> breeze_2.10-0.7.jar
>>
>> almost all the jars about breeze i can find, but still
>> NoSuchMethodError:
>> breeze.linalg.DenseMatrix
>>
>> from the executor stderr, you can see the executor successsully fetches
>> these jars, what's wrong
>> about my method? thank you!
>>
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar with
>> timestamp
>> 1400070957376
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar to
>> /tmp/fetchFileTemp7468892065227766972.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-natives_2.10-0.7.jar
>> to class loader
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar with timestamp
>> 1400070957441
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar to
>> /tmp/fetchFileTemp2324565598765584917.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.jar
>> to class loader
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar with
>> timestamp
>> 1400070957358
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar to
>> /tmp/fetchFileTemp8730123100104850193.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.8-SNAPSHOT.jar
>> to class loader
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar with
>> timestamp
>> 1400070957414
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar to
>> /tmp/fetchFileTemp3473404556989515218.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.1.jar
>> to class loader
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar with timestamp
>> 1400070956753
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar to
>> /tmp/fetchFileTemp1289055585501269156.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./build-project_2.10-1.0.jar
>> to class loader
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar with timestamp
>> 1400070957228
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar to
>> /tmp/fetchFileTemp1287317286108432726.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.7.jar
>> to class loader
>>
>>
>> DB Tsai-2 wrote
>> > Since the breeze jar is brought into spark by mllib package, you may
>> want
>> > to add mllib as your dependency in spark 1.0. For bring it from your
>> > application yourself, you can either use sbt assembly in ur build
>> project
>> > to generate a flat myApp-assembly.jar which contains breeze jar, or use
>> > spark add jar api like Yadid said.
>> >
>> >
>> > Sincerely,
>> >
>> > DB Tsai
>> > -------------------------------------------------------
>> > My Blog: https://www.dbtsai.com
>> > LinkedIn: https://www.linkedin.com/in/dbtsai
>> >
>> >
>> > On Sun, May 4, 2014 at 10:24 PM, wxhsdp <
>>
>> > wxhsdp@
>>
>> > > wrote:
>> >
>> >> Hi, DB, i think it's something related to "sbt publishLocal"
>> >>
>> >> if i remove the breeze dependency in my sbt file, breeze can not be
>> found
>> >>
>> >> [error] /home/wxhsdp/spark/example/test/src/main/scala/test.scala:5:
>> not
>> >> found: object breeze
>> >> [error] import breeze.linalg._
>> >> [error] ^
>> >>
>> >> here's my sbt file:
>> >>
>> >> name := "Build Project"
>> >>
>> >> version := "1.0"
>> >>
>> >> scalaVersion := "2.10.4"
>> >>
>> >> libraryDependencies += "org.apache.spark" %% "spark-core" %
>> >> "1.0.0-SNAPSHOT"
>> >>
>> >> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
>> >>
>> >> i run "sbt publishLocal" on the Spark tree.
>> >>
>> >> but if i manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in
>> >> /lib
>> >> directory, sbt package is
>> >> ok, i can run my app in workers without addJar
>> >>
>> >> what's the difference between add dependency in sbt after "sbt
>> >> publishLocal"
>> >> and manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib
>> >> directory?
>> >>
>> >> why can i run my app in worker without addJar this time?
>> >>
>> >>
>> >> DB Tsai-2 wrote
>> >> > If you add the breeze dependency in your build.sbt project, it will
>> not
>> >> be
>> >> > available to all the workers.
>> >> >
>> >> > There are couple options, 1) use sbt assembly to package breeze into
>> >> your
>> >> > application jar. 2) manually copy breeze jar into all the nodes, and
>> >> have
>> >> > them in the classpath. 3) spark 1.0 has breeze jar in the spark flat
>> >> > assembly jar, so you don't need to add breeze dependency yourself.
>> >> >
>> >> >
>> >> > Sincerely,
>> >> >
>> >> > DB Tsai
>> >> > -------------------------------------------------------
>> >> > My Blog: https://www.dbtsai.com
>> >> > LinkedIn: https://www.linkedin.com/in/dbtsai
>> >> >
>> >> >
>> >> > On Sun, May 4, 2014 at 4:07 AM, wxhsdp <
>> >>
>> >> > wxhsdp@
>> >>
>> >> > > wrote:
>> >> >
>> >> >> Hi,
>> >> >> i'am trying to use breeze linalg library for matrix operation in
>> my
>> >> >> spark
>> >> >> code. i already add dependency
>> >> >> on breeze in my build.sbt, and package my code sucessfully.
>> >> >>
>> >> >> when i run on local mode, sbt "run local...", everything is ok
>> >> >>
>> >> >> but when turn to standalone mode, sbt "run spark://127.0.0.1:7077
>> >> ...",
>> >> >> error occurs
>> >> >>
>> >> >> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to
>> >> >> java.lang.NoSuchMethodError
>> >> >> java.lang.NoSuchMethodError:
>> >> >>
>> >> >>
>> >>
>> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
>> >> >>
>> >> >> in my opinion, everything needed is packaged to the jar file,
>> isn't
>> >> it?
>> >> >> and does anyone used breeze before? is it good for matrix
>> operation?
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> View this message in context:
>> >> >>
>> >>
>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html
>> >> >> Sent from the Apache Spark User List mailing list archive at
>> >> Nabble.com.
>> >> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5355.html
>> >> Sent from the Apache Spark User List mailing list archive at
>> Nabble.com.
>> >>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5704.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5730.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
unsubscribe
Posted by eric perler <er...@hotmail.com>.
Re: NoSuchMethodError: breeze.linalg.DenseMatrix
Posted by DB Tsai <db...@stanford.edu>.
Hi Wxhsdp,
I also have some difficulties witth "sc.addJar()". Since we include the
breeze library by using Spark 1.0, we don't have the problem you ran into.
However, when we add external jars via sc.addJar(), I found that the
executors actually fetch the jars but the classloader still doesn't honor
it. I'm trying to figure out the problem now.
Sincerely,
DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai
On Wed, May 14, 2014 at 5:46 AM, wxhsdp <wx...@gmail.com> wrote:
> Hi, DB
> i've add breeze jars to workers using sc.addJar()
> breeze jars include :
> breeze-natives_2.10-0.7.jar
> breeze-macros_2.10-0.3.jar
> breeze-macros_2.10-0.3.1.jar
> breeze_2.10-0.8-SNAPSHOT.jar
> breeze_2.10-0.7.jar
>
> almost all the jars about breeze i can find, but still NoSuchMethodError:
> breeze.linalg.DenseMatrix
>
> from the executor stderr, you can see the executor successsully fetches
> these jars, what's wrong
> about my method? thank you!
>
> 14/05/14 20:36:02 INFO Executor: Fetching
> http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar with timestamp
> 1400070957376
> 14/05/14 20:36:02 INFO Utils: Fetching
> http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar to
> /tmp/fetchFileTemp7468892065227766972.tmp
> 14/05/14 20:36:02 INFO Executor: Adding
>
> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-natives_2.10-0.7.jar
> to class loader
> 14/05/14 20:36:02 INFO Executor: Fetching
> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar with timestamp
> 1400070957441
> 14/05/14 20:36:02 INFO Utils: Fetching
> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar to
> /tmp/fetchFileTemp2324565598765584917.tmp
> 14/05/14 20:36:02 INFO Executor: Adding
>
> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.jar
> to class loader
> 14/05/14 20:36:02 INFO Executor: Fetching
> http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar with
> timestamp
> 1400070957358
> 14/05/14 20:36:02 INFO Utils: Fetching
> http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar to
> /tmp/fetchFileTemp8730123100104850193.tmp
> 14/05/14 20:36:02 INFO Executor: Adding
>
> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.8-SNAPSHOT.jar
> to class loader
> 14/05/14 20:36:02 INFO Executor: Fetching
> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar with
> timestamp
> 1400070957414
> 14/05/14 20:36:02 INFO Utils: Fetching
> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar to
> /tmp/fetchFileTemp3473404556989515218.tmp
> 14/05/14 20:36:02 INFO Executor: Adding
>
> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.1.jar
> to class loader
> 14/05/14 20:36:02 INFO Executor: Fetching
> http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar with timestamp
> 1400070956753
> 14/05/14 20:36:02 INFO Utils: Fetching
> http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar to
> /tmp/fetchFileTemp1289055585501269156.tmp
> 14/05/14 20:36:02 INFO Executor: Adding
>
> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./build-project_2.10-1.0.jar
> to class loader
> 14/05/14 20:36:02 INFO Executor: Fetching
> http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar with timestamp
> 1400070957228
> 14/05/14 20:36:02 INFO Utils: Fetching
> http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar to
> /tmp/fetchFileTemp1287317286108432726.tmp
> 14/05/14 20:36:02 INFO Executor: Adding
>
> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.7.jar
> to class loader
>
>
> DB Tsai-2 wrote
> > Since the breeze jar is brought into spark by mllib package, you may want
> > to add mllib as your dependency in spark 1.0. For bring it from your
> > application yourself, you can either use sbt assembly in ur build project
> > to generate a flat myApp-assembly.jar which contains breeze jar, or use
> > spark add jar api like Yadid said.
> >
> >
> > Sincerely,
> >
> > DB Tsai
> > -------------------------------------------------------
> > My Blog: https://www.dbtsai.com
> > LinkedIn: https://www.linkedin.com/in/dbtsai
> >
> >
> > On Sun, May 4, 2014 at 10:24 PM, wxhsdp <
>
> > wxhsdp@
>
> > > wrote:
> >
> >> Hi, DB, i think it's something related to "sbt publishLocal"
> >>
> >> if i remove the breeze dependency in my sbt file, breeze can not be
> found
> >>
> >> [error] /home/wxhsdp/spark/example/test/src/main/scala/test.scala:5: not
> >> found: object breeze
> >> [error] import breeze.linalg._
> >> [error] ^
> >>
> >> here's my sbt file:
> >>
> >> name := "Build Project"
> >>
> >> version := "1.0"
> >>
> >> scalaVersion := "2.10.4"
> >>
> >> libraryDependencies += "org.apache.spark" %% "spark-core" %
> >> "1.0.0-SNAPSHOT"
> >>
> >> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
> >>
> >> i run "sbt publishLocal" on the Spark tree.
> >>
> >> but if i manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in
> >> /lib
> >> directory, sbt package is
> >> ok, i can run my app in workers without addJar
> >>
> >> what's the difference between add dependency in sbt after "sbt
> >> publishLocal"
> >> and manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib
> >> directory?
> >>
> >> why can i run my app in worker without addJar this time?
> >>
> >>
> >> DB Tsai-2 wrote
> >> > If you add the breeze dependency in your build.sbt project, it will
> not
> >> be
> >> > available to all the workers.
> >> >
> >> > There are couple options, 1) use sbt assembly to package breeze into
> >> your
> >> > application jar. 2) manually copy breeze jar into all the nodes, and
> >> have
> >> > them in the classpath. 3) spark 1.0 has breeze jar in the spark flat
> >> > assembly jar, so you don't need to add breeze dependency yourself.
> >> >
> >> >
> >> > Sincerely,
> >> >
> >> > DB Tsai
> >> > -------------------------------------------------------
> >> > My Blog: https://www.dbtsai.com
> >> > LinkedIn: https://www.linkedin.com/in/dbtsai
> >> >
> >> >
> >> > On Sun, May 4, 2014 at 4:07 AM, wxhsdp <
> >>
> >> > wxhsdp@
> >>
> >> > > wrote:
> >> >
> >> >> Hi,
> >> >> i'am trying to use breeze linalg library for matrix operation in my
> >> >> spark
> >> >> code. i already add dependency
> >> >> on breeze in my build.sbt, and package my code sucessfully.
> >> >>
> >> >> when i run on local mode, sbt "run local...", everything is ok
> >> >>
> >> >> but when turn to standalone mode, sbt "run spark://127.0.0.1:7077
> >> ...",
> >> >> error occurs
> >> >>
> >> >> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to
> >> >> java.lang.NoSuchMethodError
> >> >> java.lang.NoSuchMethodError:
> >> >>
> >> >>
> >>
> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
> >> >>
> >> >> in my opinion, everything needed is packaged to the jar file, isn't
> >> it?
> >> >> and does anyone used breeze before? is it good for matrix
> operation?
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html
> >> >> Sent from the Apache Spark User List mailing list archive at
> >> Nabble.com.
> >> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5355.html
> >> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5704.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: NoSuchMethodError: breeze.linalg.DenseMatrix
Posted by wxhsdp <wx...@gmail.com>.
Hi, DB
i've add breeze jars to workers using sc.addJar()
breeze jars include :
breeze-natives_2.10-0.7.jar
breeze-macros_2.10-0.3.jar
breeze-macros_2.10-0.3.1.jar
breeze_2.10-0.8-SNAPSHOT.jar
breeze_2.10-0.7.jar
almost all the jars about breeze i can find, but still NoSuchMethodError:
breeze.linalg.DenseMatrix
from the executor stderr, you can see the executor successsully fetches
these jars, what's wrong
about my method? thank you!
14/05/14 20:36:02 INFO Executor: Fetching
http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar with timestamp
1400070957376
14/05/14 20:36:02 INFO Utils: Fetching
http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar to
/tmp/fetchFileTemp7468892065227766972.tmp
14/05/14 20:36:02 INFO Executor: Adding
file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-natives_2.10-0.7.jar
to class loader
14/05/14 20:36:02 INFO Executor: Fetching
http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar with timestamp
1400070957441
14/05/14 20:36:02 INFO Utils: Fetching
http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar to
/tmp/fetchFileTemp2324565598765584917.tmp
14/05/14 20:36:02 INFO Executor: Adding
file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.jar
to class loader
14/05/14 20:36:02 INFO Executor: Fetching
http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar with timestamp
1400070957358
14/05/14 20:36:02 INFO Utils: Fetching
http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar to
/tmp/fetchFileTemp8730123100104850193.tmp
14/05/14 20:36:02 INFO Executor: Adding
file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.8-SNAPSHOT.jar
to class loader
14/05/14 20:36:02 INFO Executor: Fetching
http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar with timestamp
1400070957414
14/05/14 20:36:02 INFO Utils: Fetching
http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar to
/tmp/fetchFileTemp3473404556989515218.tmp
14/05/14 20:36:02 INFO Executor: Adding
file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.1.jar
to class loader
14/05/14 20:36:02 INFO Executor: Fetching
http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar with timestamp
1400070956753
14/05/14 20:36:02 INFO Utils: Fetching
http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar to
/tmp/fetchFileTemp1289055585501269156.tmp
14/05/14 20:36:02 INFO Executor: Adding
file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./build-project_2.10-1.0.jar
to class loader
14/05/14 20:36:02 INFO Executor: Fetching
http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar with timestamp
1400070957228
14/05/14 20:36:02 INFO Utils: Fetching
http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar to
/tmp/fetchFileTemp1287317286108432726.tmp
14/05/14 20:36:02 INFO Executor: Adding
file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.7.jar
to class loader
DB Tsai-2 wrote
> Since the breeze jar is brought into spark by mllib package, you may want
> to add mllib as your dependency in spark 1.0. For bring it from your
> application yourself, you can either use sbt assembly in ur build project
> to generate a flat myApp-assembly.jar which contains breeze jar, or use
> spark add jar api like Yadid said.
>
>
> Sincerely,
>
> DB Tsai
> -------------------------------------------------------
> My Blog: https://www.dbtsai.com
> LinkedIn: https://www.linkedin.com/in/dbtsai
>
>
> On Sun, May 4, 2014 at 10:24 PM, wxhsdp <
> wxhsdp@
> > wrote:
>
>> Hi, DB, i think it's something related to "sbt publishLocal"
>>
>> if i remove the breeze dependency in my sbt file, breeze can not be found
>>
>> [error] /home/wxhsdp/spark/example/test/src/main/scala/test.scala:5: not
>> found: object breeze
>> [error] import breeze.linalg._
>> [error] ^
>>
>> here's my sbt file:
>>
>> name := "Build Project"
>>
>> version := "1.0"
>>
>> scalaVersion := "2.10.4"
>>
>> libraryDependencies += "org.apache.spark" %% "spark-core" %
>> "1.0.0-SNAPSHOT"
>>
>> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
>>
>> i run "sbt publishLocal" on the Spark tree.
>>
>> but if i manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in
>> /lib
>> directory, sbt package is
>> ok, i can run my app in workers without addJar
>>
>> what's the difference between add dependency in sbt after "sbt
>> publishLocal"
>> and manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib
>> directory?
>>
>> why can i run my app in worker without addJar this time?
>>
>>
>> DB Tsai-2 wrote
>> > If you add the breeze dependency in your build.sbt project, it will not
>> be
>> > available to all the workers.
>> >
>> > There are couple options, 1) use sbt assembly to package breeze into
>> your
>> > application jar. 2) manually copy breeze jar into all the nodes, and
>> have
>> > them in the classpath. 3) spark 1.0 has breeze jar in the spark flat
>> > assembly jar, so you don't need to add breeze dependency yourself.
>> >
>> >
>> > Sincerely,
>> >
>> > DB Tsai
>> > -------------------------------------------------------
>> > My Blog: https://www.dbtsai.com
>> > LinkedIn: https://www.linkedin.com/in/dbtsai
>> >
>> >
>> > On Sun, May 4, 2014 at 4:07 AM, wxhsdp <
>>
>> > wxhsdp@
>>
>> > > wrote:
>> >
>> >> Hi,
>> >> i'am trying to use breeze linalg library for matrix operation in my
>> >> spark
>> >> code. i already add dependency
>> >> on breeze in my build.sbt, and package my code sucessfully.
>> >>
>> >> when i run on local mode, sbt "run local...", everything is ok
>> >>
>> >> but when turn to standalone mode, sbt "run spark://127.0.0.1:7077
>> ...",
>> >> error occurs
>> >>
>> >> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to
>> >> java.lang.NoSuchMethodError
>> >> java.lang.NoSuchMethodError:
>> >>
>> >>
>> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
>> >>
>> >> in my opinion, everything needed is packaged to the jar file, isn't
>> it?
>> >> and does anyone used breeze before? is it good for matrix operation?
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html
>> >> Sent from the Apache Spark User List mailing list archive at
>> Nabble.com.
>> >>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5355.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5704.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: NoSuchMethodError: breeze.linalg.DenseMatrix
Posted by DB Tsai <db...@stanford.edu>.
Since the breeze jar is brought into spark by mllib package, you may want
to add mllib as your dependency in spark 1.0. For bring it from your
application yourself, you can either use sbt assembly in ur build project
to generate a flat myApp-assembly.jar which contains breeze jar, or use
spark add jar api like Yadid said.
Sincerely,
DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai
On Sun, May 4, 2014 at 10:24 PM, wxhsdp <wx...@gmail.com> wrote:
> Hi, DB, i think it's something related to "sbt publishLocal"
>
> if i remove the breeze dependency in my sbt file, breeze can not be found
>
> [error] /home/wxhsdp/spark/example/test/src/main/scala/test.scala:5: not
> found: object breeze
> [error] import breeze.linalg._
> [error] ^
>
> here's my sbt file:
>
> name := "Build Project"
>
> version := "1.0"
>
> scalaVersion := "2.10.4"
>
> libraryDependencies += "org.apache.spark" %% "spark-core" %
> "1.0.0-SNAPSHOT"
>
> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
>
> i run "sbt publishLocal" on the Spark tree.
>
> but if i manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib
> directory, sbt package is
> ok, i can run my app in workers without addJar
>
> what's the difference between add dependency in sbt after "sbt
> publishLocal"
> and manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib
> directory?
>
> why can i run my app in worker without addJar this time?
>
>
> DB Tsai-2 wrote
> > If you add the breeze dependency in your build.sbt project, it will not
> be
> > available to all the workers.
> >
> > There are couple options, 1) use sbt assembly to package breeze into your
> > application jar. 2) manually copy breeze jar into all the nodes, and have
> > them in the classpath. 3) spark 1.0 has breeze jar in the spark flat
> > assembly jar, so you don't need to add breeze dependency yourself.
> >
> >
> > Sincerely,
> >
> > DB Tsai
> > -------------------------------------------------------
> > My Blog: https://www.dbtsai.com
> > LinkedIn: https://www.linkedin.com/in/dbtsai
> >
> >
> > On Sun, May 4, 2014 at 4:07 AM, wxhsdp <
>
> > wxhsdp@
>
> > > wrote:
> >
> >> Hi,
> >> i'am trying to use breeze linalg library for matrix operation in my
> >> spark
> >> code. i already add dependency
> >> on breeze in my build.sbt, and package my code sucessfully.
> >>
> >> when i run on local mode, sbt "run local...", everything is ok
> >>
> >> but when turn to standalone mode, sbt "run spark://127.0.0.1:7077
> ...",
> >> error occurs
> >>
> >> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to
> >> java.lang.NoSuchMethodError
> >> java.lang.NoSuchMethodError:
> >>
> >>
> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
> >>
> >> in my opinion, everything needed is packaged to the jar file, isn't
> it?
> >> and does anyone used breeze before? is it good for matrix operation?
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html
> >> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5355.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: NoSuchMethodError: breeze.linalg.DenseMatrix
Posted by wxhsdp <wx...@gmail.com>.
Hi, DB, i think it's something related to "sbt publishLocal"
if i remove the breeze dependency in my sbt file, breeze can not be found
[error] /home/wxhsdp/spark/example/test/src/main/scala/test.scala:5: not
found: object breeze
[error] import breeze.linalg._
[error] ^
here's my sbt file:
name := "Build Project"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.0.0-SNAPSHOT"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
i run "sbt publishLocal" on the Spark tree.
but if i manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib
directory, sbt package is
ok, i can run my app in workers without addJar
what's the difference between add dependency in sbt after "sbt publishLocal"
and manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib
directory?
why can i run my app in worker without addJar this time?
DB Tsai-2 wrote
> If you add the breeze dependency in your build.sbt project, it will not be
> available to all the workers.
>
> There are couple options, 1) use sbt assembly to package breeze into your
> application jar. 2) manually copy breeze jar into all the nodes, and have
> them in the classpath. 3) spark 1.0 has breeze jar in the spark flat
> assembly jar, so you don't need to add breeze dependency yourself.
>
>
> Sincerely,
>
> DB Tsai
> -------------------------------------------------------
> My Blog: https://www.dbtsai.com
> LinkedIn: https://www.linkedin.com/in/dbtsai
>
>
> On Sun, May 4, 2014 at 4:07 AM, wxhsdp <
> wxhsdp@
> > wrote:
>
>> Hi,
>> i'am trying to use breeze linalg library for matrix operation in my
>> spark
>> code. i already add dependency
>> on breeze in my build.sbt, and package my code sucessfully.
>>
>> when i run on local mode, sbt "run local...", everything is ok
>>
>> but when turn to standalone mode, sbt "run spark://127.0.0.1:7077...",
>> error occurs
>>
>> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to
>> java.lang.NoSuchMethodError
>> java.lang.NoSuchMethodError:
>>
>> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
>>
>> in my opinion, everything needed is packaged to the jar file, isn't it?
>> and does anyone used breeze before? is it good for matrix operation?
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5355.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: NoSuchMethodError: breeze.linalg.DenseMatrix
Posted by DB Tsai <db...@stanford.edu>.
If you add the breeze dependency in your build.sbt project, it will not be
available to all the workers.
There are couple options, 1) use sbt assembly to package breeze into your
application jar. 2) manually copy breeze jar into all the nodes, and have
them in the classpath. 3) spark 1.0 has breeze jar in the spark flat
assembly jar, so you don't need to add breeze dependency yourself.
Sincerely,
DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai
On Sun, May 4, 2014 at 4:07 AM, wxhsdp <wx...@gmail.com> wrote:
> Hi,
> i'am trying to use breeze linalg library for matrix operation in my spark
> code. i already add dependency
> on breeze in my build.sbt, and package my code sucessfully.
>
> when i run on local mode, sbt "run local...", everything is ok
>
> but when turn to standalone mode, sbt "run spark://127.0.0.1:7077...",
> error occurs
>
> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to
> java.lang.NoSuchMethodError
> java.lang.NoSuchMethodError:
>
> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
>
> in my opinion, everything needed is packaged to the jar file, isn't it?
> and does anyone used breeze before? is it good for matrix operation?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>