You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sean Owen <so...@cloudera.com> on 2014/06/09 18:20:49 UTC
Re: implementing the VectorAccumulatorParam
(The user@ list might be a bit better but I can see why it might look
like a dev@ question.)
Did you import org.apache.spark.mllib.linalg.Vector ? I think you are
picking up Scala's Vector class instead.
On Mon, Jun 9, 2014 at 11:57 AM, dataginjaninja
<ri...@gmail.com> wrote:
> The programming-guide
> <http://spark.apache.org/docs/latest/programming-guide.html> has the
> following:
>
> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
> def zero(initialValue: Vector): Vector = {
> Vector.zeros(initialValue.size)
> }
> def addInPlace(v1: Vector, v2: Vector): Vector = {
> v1 += v2
> }
> }
>
>
> // Then, create an Accumulator of this type:
> val vecAccum = sc.accumulator(new Vector(...))(VectorAccumulatorParam)
>
> However, when I try to use this I get an error:
>
> scala> import org.apache.spark.AccumulatorParam
> import org.apache.spark.AccumulatorParam
>
> scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
> | def zero(initialValue: Vector): Vector = {
> | Vector.zeros(initialValue.size)
> | }
> | def addInPlace(v1: Vector, v2: Vector): Vector = {
> | v1 += v2
> | }
> | }
> <console>:12: error: type Vector takes type parameters
> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
> ^
>
>
> Last thing, am I posting on the wrong list?
>
>
>
> -----
> Cheers,
>
> Stephanie
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: implementing the VectorAccumulatorParam
Posted by dataginjaninja <ri...@gmail.com>.
New error :-(
scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
| def zero(initialValue: Vector): Vector = {
| Vector.zeros(initialValue.size)
| }
| def addInPlace(v1: Vector, v2: Vector): Vector = {
| v1 += v2
| }
| }
<console>:12: error: not found: type AccumulatorParam
object VectorAccumulatorParam extends AccumulatorParam[Vector] {
^
<console>:14: error: value zeros is not a member of object
scala.collection.immutable.Vector
Vector.zeros(initialValue.size)
^
<console>:17: error: value += is not a member of
org.apache.spark.mllib.linalg.Vector
v1 += v2
^
-----
Cheers,
Stephanie
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6976.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: implementing the VectorAccumulatorParam
Posted by dataginjaninja <ri...@gmail.com>.
You are right. I was using the wrong vector class. Thanks.
-----
Cheers,
Stephanie
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6975.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: implementing the VectorAccumulatorParam
Posted by Sean Owen <so...@cloudera.com>.
(BCC dev@)
The example is out of date with respect to current Vector class. The
zeros() method is on "Vectors". There is not currently a += operation
for Vector anymore.
To be fair the example doesn't claim this illustrates use of the Spark
Vector class but it did work with the now-deprecated Vector.
Make sure you still have AccumulableParam imported.
You could make a PR to adjust the example to something that works with
the newer class once you have it working.
On Mon, Jun 9, 2014 at 12:27 PM, dataginjaninja
<ri...@gmail.com> wrote:
> New error :-(
>
> scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
> | def zero(initialValue: Vector): Vector = {
> | Vector.zeros(initialValue.size)
> | }
> | def addInPlace(v1: Vector, v2: Vector): Vector = {
> | v1 += v2
> | }
> | }
> <console>:14: error: value zeros is not a member of object
> scala.collection.immutable.Vector
> Vector.zeros(initialValue.size)
> ^
> <console>:17: error: value += is not a member of
> org.apache.spark.mllib.linalg.Vector
> v1 += v2
>
>
>
> -----
> Cheers,
>
> Stephanie
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6978.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: implementing the VectorAccumulatorParam
Posted by Sean Owen <so...@cloudera.com>.
(BCC dev@)
The example is out of date with respect to current Vector class. The
zeros() method is on "Vectors". There is not currently a += operation
for Vector anymore.
To be fair the example doesn't claim this illustrates use of the Spark
Vector class but it did work with the now-deprecated Vector.
Make sure you still have AccumulableParam imported.
You could make a PR to adjust the example to something that works with
the newer class once you have it working.
On Mon, Jun 9, 2014 at 12:27 PM, dataginjaninja
<ri...@gmail.com> wrote:
> New error :-(
>
> scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
> | def zero(initialValue: Vector): Vector = {
> | Vector.zeros(initialValue.size)
> | }
> | def addInPlace(v1: Vector, v2: Vector): Vector = {
> | v1 += v2
> | }
> | }
> <console>:14: error: value zeros is not a member of object
> scala.collection.immutable.Vector
> Vector.zeros(initialValue.size)
> ^
> <console>:17: error: value += is not a member of
> org.apache.spark.mllib.linalg.Vector
> v1 += v2
>
>
>
> -----
> Cheers,
>
> Stephanie
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6978.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: implementing the VectorAccumulatorParam
Posted by dataginjaninja <ri...@gmail.com>.
New error :-(
scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
| def zero(initialValue: Vector): Vector = {
| Vector.zeros(initialValue.size)
| }
| def addInPlace(v1: Vector, v2: Vector): Vector = {
| v1 += v2
| }
| }
<console>:14: error: value zeros is not a member of object
scala.collection.immutable.Vector
Vector.zeros(initialValue.size)
^
<console>:17: error: value += is not a member of
org.apache.spark.mllib.linalg.Vector
v1 += v2
-----
Cheers,
Stephanie
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6978.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.