You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sa...@wellsfargo.com on 2015/07/23 14:37:55 UTC

[MLLIB] Anyone tried correlation with RDD[Vector] ?

I tried with a RDD[DenseVector] but RDDs are not transformable, so T+ RDD[DenseVector] not >: RDD[Vector] and can't get to use the RDD input method of correlation.

Thanks,
Saif

RE: [MLLIB] Anyone tried correlation with RDD[Vector] ?

Posted by Sa...@wellsfargo.com.
Thank you very much,

working fine so far
Saif

From: Robin East [mailto:robin.east@xense.co.uk]
Sent: Thursday, July 23, 2015 12:26 PM
To: Rishi Yadav
Cc: Ellafi, Saif A.; user@spark.apache.org; Liu, Weicheng
Subject: Re: [MLLIB] Anyone tried correlation with RDD[Vector] ?

The OP’s problem is he gets this:

<console>:47: error: type mismatch;
 found   : org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.DenseVector]
 required: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector]
Note: org.apache.spark.mllib.linalg.DenseVector <: org.apache.spark.mllib.linalg.Vector, but class RDD is invariant in type T.
You may wish to define T as +T instead. (SLS 4.5)

The solution is to ensure you have a RDD[Vector] not RDD[DenseVector]

On 23 Jul 2015, at 15:30, Rishi Yadav <ri...@infoobjects.com>> wrote:

can you explain what transformation is failing. Here's a simple example.

http://www.infoobjects.com/spark-calculating-correlation-using-rdd-of-vectors/

On Thu, Jul 23, 2015 at 5:37 AM, <Sa...@wellsfargo.com>> wrote:
I tried with a RDD[DenseVector] but RDDs are not transformable, so T+ RDD[DenseVector] not >: RDD[Vector] and can’t get to use the RDD input method of correlation.

Thanks,
Saif

Re: [MLLIB] Anyone tried correlation with RDD[Vector] ?

Posted by Robin East <ro...@xense.co.uk>.
The OP’s problem is he gets this:

<console>:47: error: type mismatch;
 found   : org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.DenseVector]
 required: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector]
Note: org.apache.spark.mllib.linalg.DenseVector <: org.apache.spark.mllib.linalg.Vector, but class RDD is invariant in type T.
You may wish to define T as +T instead. (SLS 4.5)

The solution is to ensure you have a RDD[Vector] not RDD[DenseVector]

> On 23 Jul 2015, at 15:30, Rishi Yadav <ri...@infoobjects.com> wrote:
> 
> can you explain what transformation is failing. Here's a simple example.
> 
> http://www.infoobjects.com/spark-calculating-correlation-using-rdd-of-vectors/ <http://www.infoobjects.com/spark-calculating-correlation-using-rdd-of-vectors/>
> 
> On Thu, Jul 23, 2015 at 5:37 AM, <Saif.A.Ellafi@wellsfargo.com <ma...@wellsfargo.com>> wrote:
> I tried with a RDD[DenseVector] but RDDs are not transformable, so T+ RDD[DenseVector] not >: RDD[Vector] and can’t get to use the RDD input method of correlation.
>  
> Thanks,
> Saif
>  
>

Re: [MLLIB] Anyone tried correlation with RDD[Vector] ?

Posted by Rishi Yadav <ri...@infoobjects.com>.
can you explain what transformation is failing. Here's a simple example.

http://www.infoobjects.com/spark-calculating-correlation-using-rdd-of-vectors/

On Thu, Jul 23, 2015 at 5:37 AM, <Sa...@wellsfargo.com> wrote:

>  I tried with a RDD[DenseVector] but RDDs are not transformable, so T+
> RDD[DenseVector] not >: RDD[Vector] and can’t get to use the RDD input
> method of correlation.
>
> Thanks,
> Saif
>
>