You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Fabian Böhnlein <fa...@gmail.com> on 2015/08/11 14:43:29 UTC

mllib on (key, Iterable[Vector])

Hi everyone,

I am trying to use mllib.clustering.GaussianMixture, but am blocked by the
fact that the API only accepts RDD[Vector].

Broadly speaking I need to run the clustering on an RDD[(key,
Iterable[Vector]), e.g. (fabricated):

val WebsiteUserAgeRDD : RDD[url, userAgeVector]

val ageClusterByUrl =
WebsiteUserAgeRDD.groupby(_.url).mapValues(GaussianMixture.setK(x).run)

This obviously does not work, as the mapValues function is called on
Iterable[Vector] but requires RDD[Vector]
As I see it, parallelizing this Iterable is not possible, would result in
an RDD of RDDs?

Anyone has an idea how to cluster an RDD of (key, Iterable[Vector]) like in
above groupBy result?

Many thanks,
Fabian

Re: mllib on (key, Iterable[Vector])

Posted by Feynman Liang <fl...@databricks.com>.
You could try flatMapping i.e. if you have data : RDD[(key,
Iterable[Vector])] then  data.flatMap(_._2) : RDD[Vector], which can be
GMMed.

If you want to first partition by url, I would first create multiple RDDs
using `filter`, then running GMM on each of the filtered rdds.

On Tue, Aug 11, 2015 at 5:43 AM, Fabian Böhnlein <fabian.boehnlein@gmail.com
> wrote:

> Hi everyone,
>
> I am trying to use mllib.clustering.GaussianMixture, but am blocked by the
> fact that the API only accepts RDD[Vector].
>
> Broadly speaking I need to run the clustering on an RDD[(key,
> Iterable[Vector]), e.g. (fabricated):
>
> val WebsiteUserAgeRDD : RDD[url, userAgeVector]
>
> val ageClusterByUrl =
> WebsiteUserAgeRDD.groupby(_.url).mapValues(GaussianMixture.setK(x).run)
>
> This obviously does not work, as the mapValues function is called on
> Iterable[Vector] but requires RDD[Vector]
> As I see it, parallelizing this Iterable is not possible, would result in
> an RDD of RDDs?
>
> Anyone has an idea how to cluster an RDD of (key, Iterable[Vector]) like
> in above groupBy result?
>
> Many thanks,
> Fabian
>