You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jaonary Rabarisoa <ja...@gmail.com> on 2014/09/26 17:42:13 UTC
Build error when using spark with breeze
Hi all,
I'm using some functions from Breeze in a spark job but I get the following
build error :
*Error:scalac: bad symbolic reference. A signature in RandBasis.class
refers to term math3*
*in package org.apache.commons which is not available.*
*It may be completely missing from the current classpath, or the version on*
*the classpath might be incompatible with the version used when compiling
RandBasis.class.*
In my case, I just declare a new Gaussian distribution
*val g = new Gaussian(0d,1d)*
I'm using spark 1.1
Any ideas to fix this ?
Best regards,
Jao
Re: Build error when using spark with breeze
Posted by Xiangrui Meng <me...@gmail.com>.
We removed commons-math3 from dependencies to avoid version conflict
with hadoop-common. hadoop-common-2.3+ depends on commons-math3-3.1.1,
while breeze depends on commons-math3-3.3. 3.3 is not backward
compatible with 3.1.1. So we removed it because the breeze functions
we use do not touch commons-math3 code. As Sean suggested, please
include breeze in the dependency set of your project. Do not rely on
transitive dependencies. -Xiangrui
On Fri, Sep 26, 2014 at 9:08 AM, Jaonary Rabarisoa <ja...@gmail.com> wrote:
> I solve the problem by including the commons-math3 package in my sbt
> dependencies as Sean suggested. Thanks.
>
> On Fri, Sep 26, 2014 at 6:05 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> You can use scope of runtime.
>>
>> See
>> http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope
>>
>> Cheers
>>
>> On Fri, Sep 26, 2014 at 8:57 AM, Jaonary Rabarisoa <ja...@gmail.com>
>> wrote:
>>>
>>> Thank Ted. Can you tell me how to adjust the scope ?
>>>
>>> On Fri, Sep 26, 2014 at 5:47 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>> spark-core's dependency on commons-math3 is @ test scope (core/pom.xml):
>>>> <dependency>
>>>> <groupId>org.apache.commons</groupId>
>>>> <artifactId>commons-math3</artifactId>
>>>> <version>3.3</version>
>>>> <scope>test</scope>
>>>> </dependency>
>>>>
>>>> Adjusting the scope should solve the problem below.
>>>>
>>>> On Fri, Sep 26, 2014 at 8:42 AM, Jaonary Rabarisoa <ja...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm using some functions from Breeze in a spark job but I get the
>>>>> following build error :
>>>>>
>>>>> Error:scalac: bad symbolic reference. A signature in RandBasis.class
>>>>> refers to term math3
>>>>> in package org.apache.commons which is not available.
>>>>> It may be completely missing from the current classpath, or the version
>>>>> on
>>>>> the classpath might be incompatible with the version used when
>>>>> compiling RandBasis.class.
>>>>>
>>>>> In my case, I just declare a new Gaussian distribution
>>>>>
>>>>> val g = new Gaussian(0d,1d)
>>>>>
>>>>> I'm using spark 1.1
>>>>>
>>>>>
>>>>> Any ideas to fix this ?
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>>
>>>>> Jao
>>>>
>>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Build error when using spark with breeze
Posted by Jaonary Rabarisoa <ja...@gmail.com>.
I solve the problem by including the commons-math3 package in my sbt
dependencies as Sean suggested. Thanks.
On Fri, Sep 26, 2014 at 6:05 PM, Ted Yu <yu...@gmail.com> wrote:
> You can use scope of runtime.
>
> See
> http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope
>
> Cheers
>
> On Fri, Sep 26, 2014 at 8:57 AM, Jaonary Rabarisoa <ja...@gmail.com>
> wrote:
>
>> Thank Ted. Can you tell me how to adjust the scope ?
>>
>> On Fri, Sep 26, 2014 at 5:47 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> spark-core's dependency on commons-math3 is @ test scope (core/pom.xml):
>>> <dependency>
>>> <groupId>org.apache.commons</groupId>
>>> <artifactId>commons-math3</artifactId>
>>> <version>3.3</version>
>>> <scope>test</scope>
>>> </dependency>
>>>
>>> Adjusting the scope should solve the problem below.
>>>
>>> On Fri, Sep 26, 2014 at 8:42 AM, Jaonary Rabarisoa <ja...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm using some functions from Breeze in a spark job but I get the
>>>> following build error :
>>>>
>>>> *Error:scalac: bad symbolic reference. A signature in RandBasis.class
>>>> refers to term math3*
>>>> *in package org.apache.commons which is not available.*
>>>> *It may be completely missing from the current classpath, or the
>>>> version on*
>>>> *the classpath might be incompatible with the version used when
>>>> compiling RandBasis.class.*
>>>>
>>>> In my case, I just declare a new Gaussian distribution
>>>>
>>>> *val g = new Gaussian(0d,1d)*
>>>>
>>>> I'm using spark 1.1
>>>>
>>>>
>>>> Any ideas to fix this ?
>>>>
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Jao
>>>>
>>>
>>>
>>
>
Re: Build error when using spark with breeze
Posted by Ted Yu <yu...@gmail.com>.
You can use scope of runtime.
See
http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope
Cheers
On Fri, Sep 26, 2014 at 8:57 AM, Jaonary Rabarisoa <ja...@gmail.com>
wrote:
> Thank Ted. Can you tell me how to adjust the scope ?
>
> On Fri, Sep 26, 2014 at 5:47 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> spark-core's dependency on commons-math3 is @ test scope (core/pom.xml):
>> <dependency>
>> <groupId>org.apache.commons</groupId>
>> <artifactId>commons-math3</artifactId>
>> <version>3.3</version>
>> <scope>test</scope>
>> </dependency>
>>
>> Adjusting the scope should solve the problem below.
>>
>> On Fri, Sep 26, 2014 at 8:42 AM, Jaonary Rabarisoa <ja...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I'm using some functions from Breeze in a spark job but I get the
>>> following build error :
>>>
>>> *Error:scalac: bad symbolic reference. A signature in RandBasis.class
>>> refers to term math3*
>>> *in package org.apache.commons which is not available.*
>>> *It may be completely missing from the current classpath, or the version
>>> on*
>>> *the classpath might be incompatible with the version used when
>>> compiling RandBasis.class.*
>>>
>>> In my case, I just declare a new Gaussian distribution
>>>
>>> *val g = new Gaussian(0d,1d)*
>>>
>>> I'm using spark 1.1
>>>
>>>
>>> Any ideas to fix this ?
>>>
>>>
>>> Best regards,
>>>
>>>
>>> Jao
>>>
>>
>>
>
Re: Build error when using spark with breeze
Posted by Jaonary Rabarisoa <ja...@gmail.com>.
Thank Ted. Can you tell me how to adjust the scope ?
On Fri, Sep 26, 2014 at 5:47 PM, Ted Yu <yu...@gmail.com> wrote:
> spark-core's dependency on commons-math3 is @ test scope (core/pom.xml):
> <dependency>
> <groupId>org.apache.commons</groupId>
> <artifactId>commons-math3</artifactId>
> <version>3.3</version>
> <scope>test</scope>
> </dependency>
>
> Adjusting the scope should solve the problem below.
>
> On Fri, Sep 26, 2014 at 8:42 AM, Jaonary Rabarisoa <ja...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I'm using some functions from Breeze in a spark job but I get the
>> following build error :
>>
>> *Error:scalac: bad symbolic reference. A signature in RandBasis.class
>> refers to term math3*
>> *in package org.apache.commons which is not available.*
>> *It may be completely missing from the current classpath, or the version
>> on*
>> *the classpath might be incompatible with the version used when compiling
>> RandBasis.class.*
>>
>> In my case, I just declare a new Gaussian distribution
>>
>> *val g = new Gaussian(0d,1d)*
>>
>> I'm using spark 1.1
>>
>>
>> Any ideas to fix this ?
>>
>>
>> Best regards,
>>
>>
>> Jao
>>
>
>
Re: Build error when using spark with breeze
Posted by Sean Owen <so...@cloudera.com>.
Shouldn't the user's application depend on commons-math3 if it uses
it? it shouldn't require a Spark change. Maybe I misunderstand.
On Fri, Sep 26, 2014 at 4:47 PM, Ted Yu <yu...@gmail.com> wrote:
> spark-core's dependency on commons-math3 is @ test scope (core/pom.xml):
> <dependency>
> <groupId>org.apache.commons</groupId>
> <artifactId>commons-math3</artifactId>
> <version>3.3</version>
> <scope>test</scope>
> </dependency>
>
> Adjusting the scope should solve the problem below.
>
> On Fri, Sep 26, 2014 at 8:42 AM, Jaonary Rabarisoa <ja...@gmail.com>
> wrote:
>>
>> Hi all,
>>
>> I'm using some functions from Breeze in a spark job but I get the
>> following build error :
>>
>> Error:scalac: bad symbolic reference. A signature in RandBasis.class
>> refers to term math3
>> in package org.apache.commons which is not available.
>> It may be completely missing from the current classpath, or the version on
>> the classpath might be incompatible with the version used when compiling
>> RandBasis.class.
>>
>> In my case, I just declare a new Gaussian distribution
>>
>> val g = new Gaussian(0d,1d)
>>
>> I'm using spark 1.1
>>
>>
>> Any ideas to fix this ?
>>
>>
>> Best regards,
>>
>>
>> Jao
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Build error when using spark with breeze
Posted by Ted Yu <yu...@gmail.com>.
spark-core's dependency on commons-math3 is @ test scope (core/pom.xml):
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
<version>3.3</version>
<scope>test</scope>
</dependency>
Adjusting the scope should solve the problem below.
On Fri, Sep 26, 2014 at 8:42 AM, Jaonary Rabarisoa <ja...@gmail.com>
wrote:
> Hi all,
>
> I'm using some functions from Breeze in a spark job but I get the
> following build error :
>
> *Error:scalac: bad symbolic reference. A signature in RandBasis.class
> refers to term math3*
> *in package org.apache.commons which is not available.*
> *It may be completely missing from the current classpath, or the version
> on*
> *the classpath might be incompatible with the version used when compiling
> RandBasis.class.*
>
> In my case, I just declare a new Gaussian distribution
>
> *val g = new Gaussian(0d,1d)*
>
> I'm using spark 1.1
>
>
> Any ideas to fix this ?
>
>
> Best regards,
>
>
> Jao
>