You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Derrick Burns (JIRA)" <ji...@apache.org> on 2015/02/25 09:06:05 UTC
[jira] [Comment Edited] (SPARK-4039) KMeans support sparse cluster
centers
[ https://issues.apache.org/jira/browse/SPARK-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336177#comment-14336177 ]
Derrick Burns edited comment on SPARK-4039 at 2/25/15 8:05 AM:
---------------------------------------------------------------
Supporting sparse data in the clusterer is a bad idea because centroid calculation on sparse data can be very inefficient.
But so is converting sparse data directly to a dense representation for any data of significant dimension.
So, what do you do?
The HashingTF is one approach to reduce the dimension of the data, but it is not a good one. Collisions can lead to dramatic overestimates.
Instead, randoming indexing should be used.
Randoming Indexing (http://en.wikipedia.org/wiki/Random_indexing), per the Johnson-Lindenstrauss Lemma, guarantees that the embedding in a lower dimension space preserves certain distance measures.
See https://github.com/derrickburns/generalized-kmeans-clustering for an implementation.
was (Author: derrickburns):
Support sparse data in the clusterer is a bad idea.
But so is converting sparse data directly to a dense representation for any data of significant dimension.
So, what do you do?
The HashingTF is one approach to reduce the dimension of the data, but it is not a good one. Collisions can lead to dramatic overestimates.
Instead, randoming indexing should be used.
Randoming Indexing (http://en.wikipedia.org/wiki/Random_indexing), per the Johnson-Lindenstrauss Lemma, guarantees that the embedding in a lower dimension space preserves certain distance measures.
See https://github.com/derrickburns/generalized-kmeans-clustering for an implementation.
> KMeans support sparse cluster centers
> -------------------------------------
>
> Key: SPARK-4039
> URL: https://issues.apache.org/jira/browse/SPARK-4039
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 1.1.0
> Reporter: Antoine Amend
> Labels: clustering
>
> When the number of features is not known, it might be quite helpful to create sparse vectors using HashingTF.transform. KMeans transforms centers vectors to dense vectors (https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala#L307), therefore leading to OutOfMemory (even with small k).
> Any way to keep vectors sparse ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org