[jira] [Commented] (SPARK-2429) Hierarchical Implementation of KMeans

["Yu Ishikawa (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136601#comment-14136601 ] 

Yu Ishikawa commented on SPARK-2429:
------------------------------------

Hi [~rnowling],

I have read the papers which you referred.
The complexity of Divisive Hievarchical K-Means by Lamrous, et al. seems to be at least greater than O(n^2). Because it uses Silhouette score. In my opinion, Its first part is difficult to be scalable.
[http://en.wikipedia.org/wiki/Silhouette_(clustering)]

So I try to implement a hierarchical clustering with a bisecting algorithm as Freeman did. And then benchmark it and share the result with you. Please give me some days.

thanks

> Hierarchical Implementation of KMeans
> -------------------------------------
>
>                 Key: SPARK-2429
>                 URL: https://issues.apache.org/jira/browse/SPARK-2429
>             Project: Spark
>          Issue Type: New Feature
>          Components: MLlib
>            Reporter: RJ Nowling
>            Priority: Minor
>
> Hierarchical clustering algorithms are widely used and would make a nice addition to MLlib.  Clustering algorithms are useful for determining relationships between clusters as well as offering faster assignment. Discussion on the dev list suggested the following possible approaches:
> * Top down, recursive application of KMeans
> * Reuse DecisionTree implementation with different objective function
> * Hierarchical SVD
> It was also suggested that support for distance metrics other than Euclidean such as negative dot or cosine are necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org