You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yu Ishikawa (JIRA)" <ji...@apache.org> on 2014/11/07 13:35:33 UTC
[jira] [Comment Edited] (SPARK-2429) Hierarchical Implementation of
KMeans
[ https://issues.apache.org/jira/browse/SPARK-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201990#comment-14201990 ]
Yu Ishikawa edited comment on SPARK-2429 at 11/7/14 12:34 PM:
--------------------------------------------------------------
Hi [~rnowling],
I have a suggestion to you about new function. I think it is difficult for this algorithm to have an advantage in computational complexity. So I implemented a function to cut a cluster tree as a result of clustering by height. This function restructures a cluster tree, not changing the original cluster tree. We can control the number of clusters in a cluster tree by height without recomputation. This is an advantage against KMeans and other clustering algorighms.
You can see a test code and a example code at below URLs.
- [https://github.com/yu-iskw/spark/blob/8355f959f02ca67454c9cb070912480db0a44671/mllib/src/test/scala/org/apache/spark/mllib/clustering/HierarchicalClusteringModelSuite.scala#L116]
- [https://github.com/yu-iskw/spark/blob/8355f959f02ca67454c9cb070912480db0a44671/examples/src/main/python/mllib/hierarchical_clustering.py]
was (Author: yuu.ishikawa@gmail.com):
Hi [~rnowling],
I have a suggestion to you about new function. I think it is difficult for this algorithm to have an advantage in computational complexity. So I implemented a function to cut a cluster tree as a result of clustering by height. This function restructures a cluster tree, not changing the original cluster tree. We can control the number of clusters in a cluster tree by height without recomputation. This is an advantage against KMeans and other clustering algorighms.
You can see a test code at below URL.
[https://github.com/yu-iskw/spark/blob/8355f959f02ca67454c9cb070912480db0a44671/mllib/src/test/scala/org/apache/spark/mllib/clustering/HierarchicalClusteringModelSuite.scala#L116]
> Hierarchical Implementation of KMeans
> -------------------------------------
>
> Key: SPARK-2429
> URL: https://issues.apache.org/jira/browse/SPARK-2429
> Project: Spark
> Issue Type: New Feature
> Components: MLlib
> Reporter: RJ Nowling
> Assignee: Yu Ishikawa
> Priority: Minor
> Attachments: 2014-10-20_divisive-hierarchical-clustering.pdf, The Result of Benchmarking a Hierarchical Clustering.pdf, benchmark-result.2014-10-29.html, benchmark2.html
>
>
> Hierarchical clustering algorithms are widely used and would make a nice addition to MLlib. Clustering algorithms are useful for determining relationships between clusters as well as offering faster assignment. Discussion on the dev list suggested the following possible approaches:
> * Top down, recursive application of KMeans
> * Reuse DecisionTree implementation with different objective function
> * Hierarchical SVD
> It was also suggested that support for distance metrics other than Euclidean such as negative dot or cosine are necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org