You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Yu Ishikawa <yu...@gmail.com> on 2015/03/15 08:45:22 UTC

[mllib] Is there any bugs to divide a Breeze sparse vectors at Spark v1.3.0-rc3?

Hi all,

Is there any bugs to divide a Breeze sparse vector at Spark v1.3.0-rc3? When
I tried to divide a sparse vector at Spark v1.3.0-rc3, I got a wrong result
if the target vector has any zero values.

Spark v1.3.0-rc3 depends on Breeze v0.11.1. And Breeze v0.11.1 seems to have
any bugs to divide a sparse vector by a scalar value. When dividing a breeze
sparse vector which has any zero values, the result seems to be a zero
vector. However, we can run the same code on Spark v1.2.x.

However, there is no problem to multiply a breeze sparse vector. I asked the
breeze community this problem on the below issue.
https://github.com/scalanlp/breeze/issues/382

For example,
```
test("dividing a breeze spark vector") {
    val vec = Vectors.sparse(6, Array(0, 4), Array(0.0, 10.0)).toBreeze
    val n = 60.0
    val answer1 = vec :/ n
    val answer2 = vec.toDenseVector :/ n
    println(vec)
    println(answer1)
    println(answer2)
    assert(answer1.toDenseVector === answer2)
}

SparseVector((0,0.0), (4,10.0))
SparseVector()
DenseVector(0.0, 0.0, 0.0, 0.0, 0.16666666666666666, 0.0)

DenseVector(0.0, 0.0, 0.0, 0.0, 0.0, 0.0) did not equal DenseVector(0.0,
0.0, 0.0, 0.0, 0.16666666666666666, 0.0)
org.scalatest.exceptions.TestFailedException: DenseVector(0.0, 0.0, 0.0,
0.0, 0.0, 0.0) did not equal DenseVector(0.0, 0.0, 0.0, 0.0,
0.16666666666666666, 0.0)
```

Thanks,
Yu Ishikawa



-----
-- Yu Ishikawa
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Is-there-any-bugs-to-divide-a-Breeze-sparse-vectors-at-Spark-v1-3-0-rc3-tp11056.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: [mllib] Is there any bugs to divide a Breeze sparse vectors at Spark v1.3.0-rc3?

Posted by DB Tsai <db...@dbtsai.com>.
It's a bug in breeze's side. Once David fixes it and publishes it to
maven, we can upgrade to breeze 0.11.2. Please file a jira ticket for
this issue. thanks.

Sincerely,

DB Tsai
-------------------------------------------------------
Blog: https://www.dbtsai.com


On Sun, Mar 15, 2015 at 12:45 AM, Yu Ishikawa
<yu...@gmail.com> wrote:
> Hi all,
>
> Is there any bugs to divide a Breeze sparse vector at Spark v1.3.0-rc3? When
> I tried to divide a sparse vector at Spark v1.3.0-rc3, I got a wrong result
> if the target vector has any zero values.
>
> Spark v1.3.0-rc3 depends on Breeze v0.11.1. And Breeze v0.11.1 seems to have
> any bugs to divide a sparse vector by a scalar value. When dividing a breeze
> sparse vector which has any zero values, the result seems to be a zero
> vector. However, we can run the same code on Spark v1.2.x.
>
> However, there is no problem to multiply a breeze sparse vector. I asked the
> breeze community this problem on the below issue.
> https://github.com/scalanlp/breeze/issues/382
>
> For example,
> ```
> test("dividing a breeze spark vector") {
>     val vec = Vectors.sparse(6, Array(0, 4), Array(0.0, 10.0)).toBreeze
>     val n = 60.0
>     val answer1 = vec :/ n
>     val answer2 = vec.toDenseVector :/ n
>     println(vec)
>     println(answer1)
>     println(answer2)
>     assert(answer1.toDenseVector === answer2)
> }
>
> SparseVector((0,0.0), (4,10.0))
> SparseVector()
> DenseVector(0.0, 0.0, 0.0, 0.0, 0.16666666666666666, 0.0)
>
> DenseVector(0.0, 0.0, 0.0, 0.0, 0.0, 0.0) did not equal DenseVector(0.0,
> 0.0, 0.0, 0.0, 0.16666666666666666, 0.0)
> org.scalatest.exceptions.TestFailedException: DenseVector(0.0, 0.0, 0.0,
> 0.0, 0.0, 0.0) did not equal DenseVector(0.0, 0.0, 0.0, 0.0,
> 0.16666666666666666, 0.0)
> ```
>
> Thanks,
> Yu Ishikawa
>
>
>
> -----
> -- Yu Ishikawa
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Is-there-any-bugs-to-divide-a-Breeze-sparse-vectors-at-Spark-v1-3-0-rc3-tp11056.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: [mllib] Is there any bugs to divide a Breeze sparse vectors at Spark v1.3.0-rc3?

Posted by David Hall <da...@gmail.com>.
sure.

On Wed, Mar 18, 2015 at 12:19 AM, Debasish Das <de...@gmail.com>
wrote:

> Hi David,
>
> We are stress testing breeze.optimize.proximal and nnls...if you are
> cutting a release now, we will need another release soon once we get the
> runtime optimizations in place and merged to breeze.
>
> Thanks.
> Deb
>  On Mar 15, 2015 9:39 PM, "David Hall" <da...@gmail.com> wrote:
>
>> snapshot is pushed. If you verify I'll publish the new artifacts.
>>
>> On Sun, Mar 15, 2015 at 1:14 AM, Yu Ishikawa <
>> yuu.ishikawa+spark@gmail.com>
>> wrote:
>>
>> > David Hall who is a breeze creator told me that it's a bug. So, I made a
>> > jira
>> > ticket about this issue. We need to upgrade breeze from 0.11.1 to
>> 0.11.2 or
>> > later in order to fix the bug, when the new version of breeze will be
>> > released.
>> >
>> > [SPARK-6341] Upgrade breeze from 0.11.1 to 0.11.2 or later - ASF JIRA
>> > https://issues.apache.org/jira/browse/SPARK-6341
>> >
>> > Thanks,
>> > Yu Ishikawa
>> >
>> >
>> >
>> > -----
>> > -- Yu Ishikawa
>> > --
>> > View this message in context:
>> >
>> http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Is-there-any-bugs-to-divide-a-Breeze-sparse-vectors-at-Spark-v1-3-0-rc3-tp11056p11058.html
>> > Sent from the Apache Spark Developers List mailing list archive at
>> > Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: dev-help@spark.apache.org
>> >
>> >
>>
>

Re: [mllib] Is there any bugs to divide a Breeze sparse vectors at Spark v1.3.0-rc3?

Posted by Debasish Das <de...@gmail.com>.
Hi David,

We are stress testing breeze.optimize.proximal and nnls...if you are
cutting a release now, we will need another release soon once we get the
runtime optimizations in place and merged to breeze.

Thanks.
Deb
 On Mar 15, 2015 9:39 PM, "David Hall" <da...@gmail.com> wrote:

> snapshot is pushed. If you verify I'll publish the new artifacts.
>
> On Sun, Mar 15, 2015 at 1:14 AM, Yu Ishikawa <yuu.ishikawa+spark@gmail.com
> >
> wrote:
>
> > David Hall who is a breeze creator told me that it's a bug. So, I made a
> > jira
> > ticket about this issue. We need to upgrade breeze from 0.11.1 to 0.11.2
> or
> > later in order to fix the bug, when the new version of breeze will be
> > released.
> >
> > [SPARK-6341] Upgrade breeze from 0.11.1 to 0.11.2 or later - ASF JIRA
> > https://issues.apache.org/jira/browse/SPARK-6341
> >
> > Thanks,
> > Yu Ishikawa
> >
> >
> >
> > -----
> > -- Yu Ishikawa
> > --
> > View this message in context:
> >
> http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Is-there-any-bugs-to-divide-a-Breeze-sparse-vectors-at-Spark-v1-3-0-rc3-tp11056p11058.html
> > Sent from the Apache Spark Developers List mailing list archive at
> > Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > For additional commands, e-mail: dev-help@spark.apache.org
> >
> >
>

Re: [mllib] Is there any bugs to divide a Breeze sparse vectors at Spark v1.3.0-rc3?

Posted by Yu Ishikawa <yu...@gmail.com>.
Sorry for the delay in replying. I moved from Tokyo to New York in order to
attend Spark Summit East.
I verified the snapshot and the difference.
https://github.com/scalanlp/breeze/commit/f61d2f61137807651fc860404a244640e213f6d3

Thank you for your great work!
Yu Ishikawa



-----
-- Yu Ishikawa
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Is-there-any-bugs-to-divide-a-Breeze-sparse-vectors-at-Spark-v1-3-0-rc3-tp11056p11107.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org

Re: [mllib] Is there any bugs to divide a Breeze sparse vectors at Spark v1.3.0-rc3?

Posted by David Hall <da...@gmail.com>.
ping?

On Sun, Mar 15, 2015 at 9:38 PM, David Hall <da...@gmail.com> wrote:

> snapshot is pushed. If you verify I'll publish the new artifacts.
>
> On Sun, Mar 15, 2015 at 1:14 AM, Yu Ishikawa <yuu.ishikawa+spark@gmail.com
> > wrote:
>
>> David Hall who is a breeze creator told me that it's a bug. So, I made a
>> jira
>> ticket about this issue. We need to upgrade breeze from 0.11.1 to 0.11.2
>> or
>> later in order to fix the bug, when the new version of breeze will be
>> released.
>>
>> [SPARK-6341] Upgrade breeze from 0.11.1 to 0.11.2 or later - ASF JIRA
>> https://issues.apache.org/jira/browse/SPARK-6341
>>
>> Thanks,
>> Yu Ishikawa
>>
>>
>>
>> -----
>> -- Yu Ishikawa
>> --
>> View this message in context:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Is-there-any-bugs-to-divide-a-Breeze-sparse-vectors-at-Spark-v1-3-0-rc3-tp11056p11058.html
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>

Re: [mllib] Is there any bugs to divide a Breeze sparse vectors at Spark v1.3.0-rc3?

Posted by David Hall <da...@gmail.com>.
snapshot is pushed. If you verify I'll publish the new artifacts.

On Sun, Mar 15, 2015 at 1:14 AM, Yu Ishikawa <yu...@gmail.com>
wrote:

> David Hall who is a breeze creator told me that it's a bug. So, I made a
> jira
> ticket about this issue. We need to upgrade breeze from 0.11.1 to 0.11.2 or
> later in order to fix the bug, when the new version of breeze will be
> released.
>
> [SPARK-6341] Upgrade breeze from 0.11.1 to 0.11.2 or later - ASF JIRA
> https://issues.apache.org/jira/browse/SPARK-6341
>
> Thanks,
> Yu Ishikawa
>
>
>
> -----
> -- Yu Ishikawa
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Is-there-any-bugs-to-divide-a-Breeze-sparse-vectors-at-Spark-v1-3-0-rc3-tp11056p11058.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Re: [mllib] Is there any bugs to divide a Breeze sparse vectors at Spark v1.3.0-rc3?

Posted by Yu Ishikawa <yu...@gmail.com>.
David Hall who is a breeze creator told me that it's a bug. So, I made a jira
ticket about this issue. We need to upgrade breeze from 0.11.1 to 0.11.2 or
later in order to fix the bug, when the new version of breeze will be
released.

[SPARK-6341] Upgrade breeze from 0.11.1 to 0.11.2 or later - ASF JIRA
https://issues.apache.org/jira/browse/SPARK-6341

Thanks,
Yu Ishikawa



-----
-- Yu Ishikawa
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Is-there-any-bugs-to-divide-a-Breeze-sparse-vectors-at-Spark-v1-3-0-rc3-tp11056p11058.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org